- what resolution reads
- provider, model, dialect, and the capabilities in the request
- what an alias resolves to
- whatever the provider that serves it says, per candidate
- what a profile narrows
allowed_models,require_caps,pool_subset, its own rules- the answer without the call
POST /v1/preflight
The request names an alias
A client asks for fast. The provider that serves it decides what fast means, and the alias is advertised in GET /v1/models so a client can use it without knowing what it currently resolves to. That is the point of an alias, and it is also the gap. On a failover the candidate resolves the alias with its own table, which may be a different model than the primary would have sent. The request did not change. What answered it did.
A profile closes the gap. The capabilities your eval depended on are the ones the request is served with, or the request is refused and the refusal names what was missing.
Capabilities move the same way. An eval that passed did so on one endpoint, with a particular set of things honoured: strict tools, structured output, prompt caching, extended thinking, the cache and reasoning token splits a budget is metered against. None of that is in the request either. Route blind and the deployment that ships is not the deployment that was measured, and the difference arrives as a 200.
Providers host each other's dialects
Gemini serves an OpenAI-compatible endpoint. Azure serves one. Anthropic serves one and says in its own documentation that it is not production-grade, because it drops strict tools, response formats, prompt caching and extended thinking, and returns empty usage details. Bedrock's support varies per model. The provider name alone stopped being enough to say what a request would get.
A channel is one way a provider is reachable: a dialect, a tier, an optional model gate, and, for a compatibility channel, an explicit list of what it preserves. The tiers are native and provider_compat, tried in that order. A native channel is forbidden from declaring capabilities at all, because a list there would imply it was exhaustive. Translation is neither tier and comes last: it rides a native channel when no channel matches the inbound dialect. Channels and fidelity.
Write the finding down
require_caps on a profile is where the eval's result becomes configuration. It names the capabilities the validated behaviour rested on, and every channel missing one of them is out of the running for that profile's traffic, whatever else happens to be down. It is policy: on_fidelity_loss: allow never overrides it, and a probe cannot see what a profile requires, so anything a profile required stays in your configuration where you own it.
require_caps: [usage_details] is the standing example. It pins a budget-metered profile off every channel that hides the cache and reasoning token splits, which is every translated path and several compatibility endpoints.
For what you did not pin, on_fidelity_loss is the deployment's default. Before a request is forwarded, pistra reads which capabilities it uses. If every channel that could serve it would drop one, reject answers with a structured error naming every capability the channel costs and citing the page the claim was read from, so the request is fixed in one pass. allow forwards on the best candidate and counts the downgrade instead, which is the setting for traffic where the loss is known and acceptable. The read is a bytes.Contains prefilter, then a structural confirm, memoized, and never run on native traffic, where there is nothing to decide. Configuration reference.
A profile narrows what can serve it
A key names a profile, and nothing on a profile is per-key, so the envelope is edited once for everyone holding it. A key that needs its own is given its own profile rather than an override, because an override is a second place the answer can live. A profile's keys take models, channels and pool members out of the running before anything is dialled, and its access rules take the rest.
| on the profile | what it takes out of the running | counted as |
|---|---|---|
allowed_models | models outside the list, by exact name or trailing-* prefix | model_denied |
require_caps | channels missing any listed capability | fidelity_loss |
pool_subset | members outside one named subset of whichever pool serves it | pool_subset_missing |
access_rules | whatever the profile's own rules deny, after the deployment-wide ones | the rule's own code |
The right-hand column is the same envelope seen from the other side. A failover candidate has to enter it too, so a candidate the profile excludes is passed over untried rather than tried and refused, and the reason it would have given is what pistra_failover_skipped_total carries. A candidate that is always skipped is a failover policy that does not do what it reads.
Two teams on one deployment can therefore hold different routing envelopes over the same providers: one pinned to models it has evaluated and channels that report the usage split, another free to take whatever answers. The providers are the deployment's. What may serve a given caller is the profile's. Give a team its own policy.
What a translated route drops
Two dialect pairs are wired, and each one is the case where nobody else could do it: a client that speaks one dialect in front of a backend that speaks the other, with no provider hosting a compatibility endpoint for it. Where a provider-side option exists, taking it is strictly better, because the translation then happens in their infrastructure, where they own the drift.
| inbound dialect | upstream dialect | what it does not preserve |
|---|---|---|
openai | anthropic | audio_input, multi_choice, usage_details |
anthropic | openai | strict_tools, prompt_caching, structured_output, usage_details |
Dispatch is decided by the inbound route rather than by content: a request on /v1/messages is an Anthropic-dialect client, one on /v1/chat/completions is an OpenAI-dialect client. Same-dialect traffic never enters a translator at all. usage_details is in both rows because translated usage collapses the cache and reasoning token splits, in either direction. Translation.
Failover is an admission, not a redirect
A provider's retries run out before its failover begins. Each attempt is admitted under the same key, budget and guardrail decisions as the first, and the candidate is walked from that provider's own failover.to list in order. A candidate's own list applies when that candidate is the primary, never here.
The failures that move a request on are connect, 5xx, 429 and timeout. A 4xx other than 429 never fails over, because the same request would fail everywhere. timeout is opt-in and has to be named, because the abandoned request may still run to completion upstream, GPU time the gateway cannot recall. Naming it obliges every candidate to carry a bound of its own: an attempt that times out at the first provider and then hangs at the second still waits, once at each.
A candidate outside the caller's profile is passed over untried, on the envelope above and for the reason it names. A suspended provider is not silently rerouted to its failover list either. Taking a provider out because it is mishandling data is not the same as asking for the same prompts to go to a different third party, and moving that traffic is a decision that belongs in the configuration where it can be read. On a failover that does happen, the model and the backend move as a pair: the candidate that served, and that candidate's own resolution of the alias, which may be a different model than the primary would have sent. Both are metered and both are what the reconcile audit reads, so the two ledgers never disagree about what one request was, and the model that actually answered is a fact you can look up. Suspend a key, provider or rule.
A pool is asked, and the answer is checked
A pool is a set of self-hosted model servers reached under one name. Which server takes a request is worth real latency, because each one holds a different prefix cache and a different queue. pistra does not make that choice. Either an inference gateway already in the path makes it, and pistra sees a provider whose base_url is that gateway's, or pistra asks the pool's endpoint picker over ext_proc and dials the server it names. A pool is a pool because the document says pool:, not because a controller read a Kubernetes resource: the data plane reads no cluster API and holds no credential for one.
A remote scheduler naming a destination is telling the gateway where to send a request, so every endpoint the picker names is checked against the set the pool's name resolves to at that moment. An address outside it is refused, logged and counted, and the request is spread over the members in turn instead. The check holds however well regarded the scheduler is, because what it bounds is arbitrary-destination forwarding rather than any particular picker's judgement. Inference pools.
Every change of route is counted
Retries and failovers are successes from the client's point of view. A provider that is steadily failing is invisible in the error rate and obvious here.
| counter | labels | what a value says |
|---|---|---|
pistra_channel_downgrade_total | provider, channel, cap, action | A capability was lost, and whether the request was skipped over that candidate, allowed lossy by policy, or rejected. |
pistra_failover_skipped_total | provider, reason | A candidate could not be admitted and was passed over untried. Always skipped is a failover policy that does not do what it reads. |
pistra_failover_total | from, to, reason | An attempt was abandoned for another provider. Steady here is a provider failing steadily, hidden from clients rather than fixed. |
pistra_retry_total | provider, reason | An attempt was re-sent to the same provider after a wait. Rising under load is the early sign of a retry storm. |
pistra_pool_picks_total | provider, outcome | How a pooled request was sent. A rising fallback is a pool that declares a scheduler and has quietly stopped being scheduled. |
The decision is also available before the call. POST /v1/preflight answers the routing and fidelity question without performing the request, and refuses to pretend it answers more than that. POST /admin/v1/preflight asks the same resolver on a named key's behalf, so reproducing a tenant's refusal needs neither their credential nor a temporary edit to the configuration. Each routing decision records which overlay's claim it acted on as pistra.channel_origin, and GET /admin/v1/catalog/report lists every routable channel with its provenance and the date it was verified. Silent degradation.
Threat model
The asset is the behaviour you validated before you shipped, and the capabilities it rested on. Nobody attacks it. A provider adds a compatibility endpoint, a candidate takes the request while the primary is down, a pool member answers instead of the one the scheduler named, and the call is served with less: strict tools gone, structured output gone, the cache and reasoning token splits a budget meters collapsed into a total. It arrives as a successful response, which is what makes it hard to see. The control is resolution per channel, a profile that pins the capabilities the validated behaviour needed, and a counter on every route that changed. The limit is that a capability claim is a fact about somebody else's service. It is read off their documentation on a day, and the day it stops being true is not a day anybody announces.
The entry this section expands, and the ten beside it: threats and limits.
What this does not do
- It checks capabilities, not behaviour. A channel is admitted on what it preserves. Two endpoints that both honour every capability the request uses can still answer differently, and nothing here compares their output or re-runs your eval. Pinning says the validated behaviour can run; it does not say it did.
- It does not rank model servers. With no picker configured a pool is spread over its members in turn, which reads nothing about them. Prefix-cache affinity and prefill/decode placement live in the picker, and pistra does not ship a worse version underneath one.
- It is not a Gateway API implementation. No
GatewayClass, noGatewayobject, nostatus.parentswritten back, and no conformance claim. Where you run anInferencePool, something else has to write its address into pistra's configuration. - A capability claim is dated, not checked. It is a fact about somebody else's service, read off their documentation or observed live on one day. The claim carries
verified_atandverified_byso you can see how old it is. Nothing tells you the day it stopped being true. - Two translated pairs, and no more. A dialect that is not carried is cut on the way in rather than left dormant, so there is no translator to switch on. Reviving one is a change to the wired roots and a release, not a configuration key.
- The member set is as fresh as DNS. The picker watches the Kubernetes API and pistra watches DNS, so a pod the picker already knows about can be one TTL away from being dialable here. That shows up as a fallback, and the request is still served, without the picker's help.
- There is no route variable in a cost expression. Cost is per-token and models carry the prices, so the discrimination is already in
model. Route-dependent policy belongs in an access rule, which gets a realroute. An expression referencing one in a cost is refused at the config write rather than left to evaluate empty.
The argument
- Proxies and the unified API. One API for every provider is a good trade until somebody asks what the provider actually received. A proxy that parses your request into its own shape and writes it back cannot answer, because what survives is whatever that shape knew about the week it was written.
- Response caching. A response cache in front of the model saves the whole call, including the part where something decided about it. Caching belongs where the model's own cache is, and a gateway earns the provider's discount by leaving the bytes alone.