Inference is usually the largest line on the platform bill, and a response cache is the cheapest-looking way to cut it. On some traffic it is.
A hit is a request nothing on the path decided about. No rule ran on it and no span in it was redacted, because there was no call to run anything on. The provider never saw it either, so its usage and its own audit have no entry for it.
The second effect shows up later. A support conversation goes wrong, the transcript is in the log, and the engineer replaying the prompt gets a different answer from the model. The log has the prompt the application sent and the answer the application received. It does not say whether a model produced that answer for that prompt, or for a prompt that resembled it a week earlier.
Two kinds of cache sit in a proxy, and they reach those two effects differently.
Exact and similar
An exact-match cache hashes the whole request body, and every parameter, and returns the stored response when the hash repeats. On a conversation it almost never repeats. Each turn is the previous request plus what was just said, so by construction the hash changes every turn, and the cache holds a growing set of entries that will never be read. It hits on the traffic that does repeat byte for byte: an evaluation suite, a CI job, a developer pressing enter again, a retry, an FAQ bot answering the same question the same way. That is real money.
A similarity cache embeds the prompt and returns a stored response when the embedding is close enough to a stored one. It was built for the traffic the exact cache cannot serve, and it fails on the traffic that dominates production for the same reason the exact cache misses it. Consecutive turns of a conversation are the previous turn plus a little, so their embeddings are almost identical, and at any threshold that catches paraphrases it also catches the previous turn. The vendors say this themselves now, in their own warning boxes, and the box at the end quotes one. The symptom is an agent that repeats the same tool call until somebody notices.
So a response cache does cut the bill. On any hit, nothing decided anything, because there was no call to decide about. On a similarity hit, the prompt in your log is not the prompt that produced the answer.
The record on a hit
On any hit the provider’s usage has no entry, the provider’s logs have no entry, and whatever record sits behind the cache has no entry, because no request reached it. A cache in front of the thing keeping your record is traffic the record omits silently. The redaction rule that would have rewritten a card number before it left never ran either, because nothing left.
This is an argument about where a cache goes, not against caches.
The provider’s own cache
The model providers cache too, and theirs is a different thing. When the same prefix of a prompt arrives again, the provider skips recomputing the model’s internal state for that prefix, and charges a fraction of the input price for the tokens it skipped. It is matched on the rendered prompt. The markers that say where a cached prefix ends, the order of the tool definitions, and the settings that change the layout all have to arrive as the application sent them. A gateway earns the read discount by leaving the bytes alone.
It is priced as a trade. Writing a prefix into the cache costs more than sending it plain, reading it back costs a tenth, entries live for minutes to an hour unless you pay to keep them, and a prefix has to be long enough to be worth it. Output tokens are full price either way. The split is visible: the usage the provider reports says how many tokens came from cache, and pistra’s meter spends that figure instead of an estimate.
A gateway that rebuilds requests breaks this by accident.
Who should run a response cache anyway
If your traffic is single-shot and repeats, run an exact-match cache and keep it. Evaluation runs, CI, a development loop, retries, an FAQ bot: the same request hours or days apart gets nothing from the provider’s cache, which has long expired, and everything from yours, which saves the input, the output and the latency. That is a bigger saving than the provider’s discount, on traffic where a stale answer costs nothing.
Hits on a cache in front of pistra are outside pistra’s record and its redaction. The cache is also a store of prompt bodies, so it answers to the same data-handling policy as your log store, which is a decision to take before deciding where to put it. For anything compliance covers, put the cache behind pistra, so the decision and the record happen before the lookup, or do not run one.
Similarity caches we would keep away from anything with a second turn, and the vendors’ own documentation now agrees.
What pistra does at this box
There is no similarity cache in the decision path, and no cache that changes what the provider sees. The request goes to the provider as the application sent it, cache markers included. The provider’s usage comes back with the cached and uncached counts separated, the meter spends what the provider reported, and the trace carries the split. A route that would drop prompt caching on the way to a provider is refused by default, because a cache that silently stops working shows up as a bill and never as an error.
As of 28 August 2026
LiteLLM, in its proxy caching docs: "Semantic caching is designed for single-shot prompts. On multi-turn or agentic traffic it will replay stale responses." The explanation given: consecutive turns' embeddings are "~0.99 similar", so "every turn matches the previous turn's cached entry and the client replays a stale response", which "typically shows up as an agent repeating the same tool call over and over." docs.litellm.ai/docs/proxy/caching. Portkey's simple and semantic cache, Kong's AI Semantic Cache plugin and Cloudflare AI Gateway's exact-match cache are documented by the vendors at portkey.ai, developer.konghq.com and developers.cloudflare.com; the thresholds, lifetimes and the exact-versus-similar behaviour differ between them and are on those pages.
Provider prompt caching, from the providers' pricing and guide pages. Anthropic: up to four cache_control breakpoints; writes 1.25× base input for a five-minute lifetime or 2× for one hour, reads 0.1×; minimum cacheable prefix 512 to 4,096 tokens by model, and a shorter prompt "cannot be cached, even if marked" with no error. platform.claude.com. OpenAI: on GPT-5.6 and later, writes 1.25× and reads 0.1×, thirty-minute lifetime refreshed on use, minimum 1,024 tokens; earlier models charge no write premium, discount reads by model, and hold entries five to ten minutes of inactivity. developers.openai.com. Gemini: implicit caching on by default for 2.5 and newer with a 2,048 or 4,096 token minimum; explicit caching bills a read discount plus storage per hour. ai.google.dev/gemini-api/docs/caching.
pistra, per the docs of the same date: the meter reads cached_input and cache_creation_input token counts as the provider reports them; the fidelity guard refuses a channel that drops prompt caching unless the operator sets on_fidelity_loss: allow, and require_caps on a profile pins metered traffic off channels that hide the split.