Skip to content
pistra

Inference pools

A pool is a set of self-hosted model servers reached under one name. Which server takes a request is worth real latency, because each one holds a different prefix cache and a different queue, and under disaggregated serving each holds a different part of the job. pistra does not make that choice. Something that watches the servers makes it, and pistra reaches that something in one of two ways.

Two ways a pool is reached: fronted, where an inference gateway holds the picker, and direct, where pistra asks the picker itself and checks the answer against the pool's members before dialing

The two shapes

Section titled “The two shapes”

In the fronted shape, pistra sees a provider whose base_url is an inference gateway’s. That gateway does model-to-pool routing and endpoint picking behind it. pistra does keys, budgets, guardrails, metering and failover to a different provider. The cost is one hop inside the cluster, and pistra tracks no scheduling protocol.

In the direct shape, pistra dials the model servers itself and consults the picker over ext_proc to decide which one. There is no extra hop, and there is a protocol dependency, which the rest of this page is mostly about.

What decides between them is whether a gateway that already speaks the picker protocol is in the path. Where one is, use it: the fronted shape asks less of pistra and less of you, and llm-d, GKE and Envoy AI Gateway each install one. The direct shape is for the deployment that would otherwise install a second gateway purely to be scheduled, and for the fleet with no Kubernetes around it.

Who owns what

Section titled “Who owns what”

The Gateway API Inference Extension and llm-d split in August 2026, and the pieces landed in two repositories. Reading either one alone gives a misleading picture of the other.

Where What lives there
sigs.k8s.io/gateway-api-inference-extension The InferencePool API, the Endpoint Picker Protocol specification, LWEPP as a conformance picker, the conformance tests.
github.com/llm-d/llm-d-router The production endpoint picker, InferenceObjective, InferenceModelRewrite, body-based routing.

llm-d Router therefore implements a protocol upstream still specifies. The split moved the implementation, and left the API and the specification where they were.

pistra is a client of that protocol and of nothing else in either repository. It imports neither module. In the direct shape it speaks the picker protocol over plain ext_proc, and in the fronted shape it speaks HTTP to a gateway and knows none of this exists.

Declared, never detected

Section titled “Declared, never detected”

Every other consumer of an InferencePool is a controller. It watches the Kubernetes API, sees the custom resource, and reconciles. pistra reads no Kubernetes API. A provider is a pool because the document says it is:

providers:
  - name: llama-fleet
    dialect: openai
    pool:
      service: http://vllm.gpu.internal:8000
      picker:
        service: http://llama-epp.llm-d.svc.cluster.local:9002

The presence of pool: is the whole discriminator. There is no controller reading InferencePool resources, no status.parents written anywhere, and the data plane holds no Kubernetes credential to read one with. This follows from two decisions the rest of the system already rests on.

The first is that a configuration document travels. It moves through the admin API, a CRD, a Terraform resource and the raft log, and every node has to read it identically, so it cannot carry facts about one cluster’s API server. The second is the threat model. A data plane that watches the Kubernetes API is a data plane holding a credential for it, inside the process on the request path, and the deployment shapes are built so that the operator is the only component with a cluster credential.

The consequence: pistra supports the picker protocol, and does not support the InferencePool resource. Where you run one, something has to write its address into pistra’s configuration. The operator does that for its own kinds, and reads no InferencePool either.

The answer is checked

Section titled “The answer is checked”

A remote scheduler naming a destination is telling the gateway where to send a request. Every endpoint the picker names is therefore checked against the set the pool’s own name resolves to at that moment.

One pick in four steps: ask the picker, take its ip:port answer, check that address against what the pool's name resolves to, then dial it or refuse and spread instead

An address outside that set is refused, logged and counted, and the request falls back to being spread over the members in turn. The check holds however well regarded the scheduler that gave the answer is, because what it bounds is arbitrary-destination forwarding rather than any particular picker’s judgement.

It costs currency in the member set. The picker watches the Kubernetes API and pistra watches DNS, so a pod the picker already knows about can be one DNS TTL away from being dialable here. That shows up as a fallback, and the request is served without the picker’s help rather than failing.

What this does not make pistra

Section titled “What this does not make pistra”

It is not a Gateway API implementation. There is no GatewayClass, no Gateway object, no status.parents written back, and no conformance claim. Consulting the picker is a client behaviour, and the architecture page has the longer version of why pistra is not a proxy with LLM features bolted on.

It is not a scheduler. With no picker configured, a pool is spread over its members in turn, which reads nothing about them and cannot rank them. That floor exists because leaving requests on the pool’s own address pins one server per connection for that connection’s life. Prefix-cache affinity and prefill/decode placement live in the picker, and pistra does not ship a worse version underneath one.

What it costs

Section titled “What it costs”

The direct shape tracks a protocol somebody else owns, which is the reason to prefer the fronted shape wherever a gateway is already in the path.

The surface pistra uses is small: ask, read one answer, report what was served. The end-to-end test runs against upstream’s own LWEPP at a pinned version on a real cluster, so a protocol change arrives as a failing test. And every request records how it was scheduled, in pistra_pool_picks_total. The fallback outcome on that counter is the one to alert on, since a pool that has quietly stopped being scheduled keeps serving and nothing else reports it.

Section titled “Related”