Metering

Pistra meters the usage count reported by the provider. It does not run its own tokenizer or publish a second billing estimate.

Why the gateway does not tokenize

Every provider tokenizes differently, and so does every model behind the same provider. The vocabularies differ, the chat template that wraps messages before tokenization differs, images become a tile count nobody publishes a formula for, and tool schemas are serialized into the prompt in a shape that is not the shape you sent. Two of those change without notice, on the provider’s release schedule rather than yours.

A local count would be an estimate. It can differ from the provider’s count for templates, images, tools, or model changes. The budget would then enforce a number that disagrees with the invoice.

Reading the provider’s figure avoids that version skew. Pistra maintains no tokenizer vocabulary or per-model token table.

Where the numbers come from

On the passthrough path nothing is unmarshaled, so usage is read with targeted scans of the response body. There are three wire shapes, not two:

Shape	Routes	Input	Output
OpenAI chat	chat_completions, completions, embeddings	prompt_tokens	completion_tokens
OpenAI Responses	responses	input_tokens	output_tokens
Anthropic	messages	input_tokens	output_tokens

The route determines the wire shape, not only the dialect. /v1/responses and /v1/chat/completions are the same provider speaking the same dialect, and they share no usage field name at all. The detail objects are renamed too, and a streamed Responses run nests its usage inside the response object rather than putting it at the top of the chunk. The dialect answers “what does this upstream speak”, which an operator declares and the catalog probe reads. Only the route label separates OpenAI’s two, so the route picks between them.

Translated traffic does not use any of this. When dialects differ the vendored translators have already parsed the response, and the usage comes from them.

Gemini is metered through its OpenAI-compatible endpoint, which is where the catalog preset points and which reports prompt_tokens and completion_tokens like any other OpenAI-compatible server. There is no native generateContent dialect, so a provider pointed at the raw v1beta surface would report no usage and settle against the estimate below on every request.

Streaming has to be asked

An OpenAI-dialect stream reports no usage at all unless the request asked for it. A metered stream that did not ask has nothing to settle against.

So the gateway asks on the client’s behalf. It sets stream_options.include_usage on streamed chat requests that lack it, and removes the usage-only chunk that produces on the way out. The client’s stream is the one it would have received either way. The chunk test is deliberately narrow, because dropping a frame that carried text would lose output rather than accounting. Usage alongside real content is content, and some servers do that.

Only the chat shape is asked. /v1/responses accepts a stream_options object but not that member, and rejects the request rather than ignoring it, so asking there would turn a metering gap into a 400. Nothing is lost. A Responses stream reports usage in its terminal event unasked, including when it ends early on response.incomplete or response.failed. An Anthropic stream restates cumulative counts on every message_delta, and that is also what makes a run whose server tools drove several inference turns come out right.

See stream_usage for turning it off, which is worth doing for an OpenAI-compatible server that rejects unknown fields.

The reservation, and the estimate that is not a bill

The budget hold as an interval: opened at step 5 on an estimate, closed at step 10 on the token counts the provider reported, with the difference returned

Cost is reserved before the request is forwarded and settled against real usage when the response comes back. Envoy AI Gateway, whose cost vocabulary pistra borrows, debits after the response instead. A limit that is only checked on the way out cannot stop concurrent long streams from overrunning it together, and long streams are the traffic most worth limiting. The divergence is intentional.

Reserving before means guessing, and this is the one place a token count is ever invented:

• input, the request body length divided by four
• output, max_tokens or max_completion_tokens if the request states one, otherwise 1024

Both are crude on purpose. A high guess costs transient headroom that is given back within milliseconds. A low guess admits traffic that is over budget. The number is a hold, not a bill, and settlement replaces it with what the provider said.

The basis is not a guess, though. The hold is priced with the same model and provider settlement will use, so a cost expression that weights one model above another holds at that weight too. Otherwise a budget could admit on a cheap reading and charge on an expensive one.

Three outcomes at settlement:

• Usage reported, the hold is replaced by the real cost.
• Nothing reported, upstream answered 2xx, the hold settles at the estimate. Charging a guess is wrong. Letting spend that really happened go free is worse, and it is the failure that compounds.
• The request never completed upstream, released in full.

A budget is denominated in tokens, not money

There is no price table in pistra and no currency anywhere in the config. A budget is a number of cost units per window, and the cost block says what one request draws:

budgets:
  - name: team-daily
    limit: 2000000
    window: day
    cost: {type: TotalToken}

That is two million tokens a day, not two million of anything else. The types are InputToken, OutputToken, TotalToken (the default), CachedInputToken, CacheCreationInputToken, ReasoningToken, or CEL. Declaring one and attaching it to a team is Cap what a team spends. The rest of this page is why it works the way it does.

CEL is for per-model and per-provider price weighting, and it is the only way to express it. The gateway will not do it for you, because it does not know what you pay:

budgets:
  - name: team-daily
    limit: 5000000
    window: day
    cost:
      type: CEL
      cel: 'model.startsWith("gpt-4") ? input_tokens + output_tokens * uint(4) : input_tokens + output_tokens'

The expression sees exactly these:

Variable	Type	Is
model	string	the model actually sent upstream, the alias target where one applies
backend	string	the provider that served it
input_tokens	uint	prompt tokens, cache reads included where the provider counts them there
cached_input_tokens	uint	the cached subset of the input
cache_creation_input_tokens	uint	tokens written to the cache (Anthropic; zero elsewhere)
output_tokens	uint	completion tokens
total_tokens	uint	the provider’s total, or input plus output when it reported none
reasoning_tokens	uint	the reasoning subset of the output

model and backend are resolved together, and always describe the same real request. A key that sends the alias fast is priced on gpt-4o-mini, because that is the name that went upstream and the name the bill will carry. On failover the pair moves as a pair: the candidate that served, and that candidate’s own resolution of the alias, which may be a different model than the primary would have sent. It is also the name the reconcile audit meters, so the two ledgers never disagree about what one request was.

There is no route variable. Cost is per-token and models carry the prices, so the discrimination a cost expression needs is already in model. Route-dependent policy belongs in an access rule, which gets a real route. Envoy AI Gateway’s vocabulary has a route_name naming an AIGatewayRoute. pistra has no such object, so the variable is not declared rather than left to compile and evaluate empty forever. An expression referencing it is refused at the config write.

Expressions compile at config build, never at request time. A typo fails the build, the write is refused, and the snapshot that is already serving keeps serving. A compiled program over six integers runs on the hot path.

What is not token-metered

Three surfaces have no token usage to read, and all three charge flat:

• MCP calls draw cost_per_call from the bound budget, default 1. A key whose budget also meters LLM tokens is mixing units. Give MCP-heavy keys their own budget.
• A2A calls draw cost_per_call from the same budget on the same terms, configured per agent. One call is one message method or one task read, whatever the peer then does about it: a task that runs for an hour and a task that fails immediately cost the same, because the gateway is charging for the door rather than for the work behind it.
• WebSocket sessions reserve cost_per_session at the upgrade, default 1. On top of that, usage the upstream reports inside the session is spent incrementally as it streams past, OpenAI Realtime’s response.done, Gemini Live’s usageMetadata. A session that breaks its budget mid-stream is closed on both sides with WebSocket close code 1008. The frame decoder is passive. If it ever desyncs it turns itself off rather than touch the session.

What it does not promise

A limit is a cluster limit, but not a hard one. Every node meters only the traffic it served, so a bucket holds three quantities: the durable cluster-wide total it last read, this node’s settled spend that has not reached the store yet, and its live reservations. The flush tick, every 500 ms, writes the second away and re-reads the first, so a node learns what its siblings spent within one interval.

The other nodes’ unflushed spend remains, bounded by that same interval. So a limit can be exceeded slightly, by roughly what the rest of the cluster can spend in half a second, and it cannot be exceeded by a multiple of itself. Making it exact would mean leasing slices of the allowance through raft, which is a much larger mechanism than a spend cap needs.

A crash loses up to one flush interval of settled spend. Durability is asynchronous in batches, which is a bounded overspend chosen over an fsync per request inside the overhead budget. The interval and the bargain are the same ones as above.

Missing usage settles at a guess. See above. It is the estimate, not zero, but it is still a guess, and a provider that never reports usage will produce a ledger made only of them.

Watching a budget

Three signals, and they answer different questions.

pistra_budget_used_ratio{budget} is the one to alert on. It is the fraction drawn by whichever of the budget’s buckets is closest to its limit, republished on every flush tick. A threshold below 1 is the difference between finding out at 80% and finding out from a 429. It is the worst bucket rather than the budget averaged out, because a budget with per-key allowances has no single fill level.

pistra_budget_buckets{budget, state} counts the open buckets on either side of the limit, and exists because the ratio alone cannot tell one exhausted key from every exhausted key. Both report 1.0.

pistra_auth_failures_total{reason="budget_exceeded"} counts refusals that already happened. The name says auth because the counter is shared by every pre-provider refusal, and budget_error is a separate reason. A ledger that could not answer is an outage, not a spending pattern. Do not reach for pistra_request_duration_seconds{code="429"} instead. Three different paths produce a 429, two of them a provider’s own rate limit passed back.

All three are one node’s view, accurate to one flush interval. Exact per-bucket figures, including buckets left draining by a budget the configuration no longer declares, are at GET /admin/v1/budgets.

Checking the arithmetic

None of those signals can show spend the gateway never metered, because a ledger cannot report what it did not see. Everything above trusts the wire scan. reconcile is the check on it: a second ledger built from the provider’s own admin usage API, compared against what this cluster metered, judged per token class once the day is old enough to have stopped moving. It turns “we believe the scanner sees everything” into something with a verdict on it.

See audit your metering against the provider.