Proxies and the unified API

The first proxy we ran in front of a model was forty lines of Python and it saved us a week. One key per team and a spend counter in Postgres. LiteLLM and Portkey solved the problem those forty lines solved, for teams that could not keep the forty lines alive, and then kept going: a hundred providers behind one API, virtual keys, budgets, retries, a dashboard. For a developer, or an ML platform team of five, that is the right product.

Before launch, somebody asks for confirmation that the tool definitions the application sends are the tool definitions the model sees. The proxy’s dashboard shows what the application sent it, the provider’s dashboard shows tokens, and nothing shows the bytes in between, so nobody can confirm it. The answer will be the same in three months, when the question comes back because a response was wrong or a bill was high.

The rewrite in the middle

To offer one API for every provider, a proxy has to parse your request into its own structure and write it back out in the provider’s dialect. That is the product, not a flaw in it. The consequence is that whatever the structure does not know about is gone, and it is gone without an error. No status code changes, nothing retries, the request succeeds and quietly does less than the application asked for.

The part of the request a structure has to know about holds the tool schemas, which get normalised, reordered or have unknown keys dropped. It holds the provider-specific fields that arrived last month: a cache marker, a reasoning-effort setting, a structured-output mode, a header that turns on a beta. And it holds the order of keys inside objects, which a language’s JSON library is free to change and which at least one provider hashes.

That last one shows where the line falls. Providers do not match on your HTTP bytes. They match on the rendered prompt, the sequence of tokens the model sees after the request has been laid out. Whitespace and the order of top-level fields do not render, so a proxy that only normalises those has a correct objection here. The content renders, and so do the tool definitions in their order, the settings that change the layout, and the markers that say where a cached prefix ends. A proxy that rebuilds any of those has changed what the model saw, and the provider’s cache, its pricing and its behaviour follow the rendered prompt.

In February 2026 a LiteLLM user found that Anthropic’s top-level cache marker, the one that turns on automatic caching, was being dropped because it was not on the proxy’s list of known parameters. It was fixed, in a numbered pull request, a couple of weeks later. That is the mechanism working as designed, and it is why the question cannot be answered from the proxy’s side. Every provider feature launch is a release you wait for, and between the launch and the release the request succeeds.

So a confirmation has to come from something that did not rebuild the request in the first place. The provider saw what the client sent, and the record attests to that request.

Who carries the list

A gateway that forwards the body unchanged cannot have dropped anything. That property comes for free instead of being maintained, and it is one way to be correct rather than the only one. A proxy that parsed and re-serialised with perfect knowledge of every field would render identically, and a provider would not know the difference. The difference is who carries the list. With a structure in the middle, somebody maintains the list of fields the structure knows, and the provider adds to the other list every month. Without one, there is no list.

There is a cost, and it is normalisation. A gateway that does not rebuild the request cannot present an OpenAI-shaped request to an Anthropic model without doing the rewrite this post is about. pistra does that in one case only, when no channel speaks the client’s dialect, and it comes last in the routing walk. The box at the end says what the two translators drop and what the gateway does when a request uses one of those fields. The performance page in the docs covers what that rewrite costs, and why it costs more on a stream than on a unary body.

Who should use a proxy anyway

Five people and three providers. The value is the uniformity: one SDK shape in the codebase, one place to swap a model, a spend counter that is right enough. Nobody there has asked what the provider received. Run LiteLLM. Keep it upgraded, because that is where the list lives.

The line falls in the same place as last week. The first time somebody has to show what the model was sent, the proxy can offer a version number.

What pistra does at this box

pistra’s answer is dull on purpose. On a request whose dialect a provider speaks natively, the model field is read, and patched in place if an alias maps it to a provider’s name. The credential is swapped in the headers. On a metered stream to an OpenAI-dialect provider, stream_options.include_usage is set when the client left it out, so the provider reports the tokens the budget needs, and the usage chunk that produces is dropped before it reaches the client. The rest of the body reaches the provider as the client sent it, and the record attests to that request.

We measured what that costs: 76.1 µs per request against 68.0 µs for the Go standard library’s reverse proxy doing nothing, and 30.6 µs with no gateway at all, on an M4 over loopback with the upstream answering immediately. Most of what a gateway costs is the extra hop, and about eight microseconds of it is pistra’s own work. One row of that table reads as though translation were free, because on a unary body it very nearly is. The page it comes from says so, and says where the cost lands.

All posts