Schedule a pool with an endpoint picker

Use pool: for self-hosted model servers behind one DNS name. pistra does not rank them itself. It asks the pool’s endpoint picker which server should take each request, checks the answer against the pool’s own membership, and dials it.

The picker is anything speaking the Gateway API Inference Extension’s picker protocol, such as llm-d’s Router or upstream’s LWEPP. That is where prefix-cache affinity and prefill/decode scheduling live, and pistra does not ship a worse version underneath one.

Declare the pool

A pool is one provider whose address is a name rather than an endpoint:

providers:
  - name: llama-fleet
    dialect: openai
    models: ["llama-*"]
    pool:
      service: http://vllm.gpu.internal:8000

service is the name that resolves to every server, a headless Service in Kubernetes or a round-robin A record anywhere else, and the port is the one the servers listen on.

That is the floor. Nothing is scheduled, but requests still go round the servers in turn rather than to the pool’s own address, because leaving them on the address pins one server per connection for that connection’s life. Spreading in turn reads nothing about the servers and cannot rank them. Go on if you want ranking.

Add the picker

Name the scheduler inside the pool block:

    pool:
      service: http://vllm.gpu.internal:8000
      picker:
        service: http://llama-epp.llm-d.svc.cluster.local:9002

The scheme chooses the transport. http:// dials in the clear, https:// with TLS. The port is required, because a picker serves gRPC on a port of its own rather than at a web root.

Per request, after your keys, budgets and guardrails have had their say, pistra opens one ext_proc exchange to that address and sends the request headers and the body that is about to go upstream. The picker answers with one or more ip:port endpoints in preference order. pistra dials the first one the pool contains, and tells the picker how the request ended so its view of the fleet’s load stays true.

Two things are deliberately not sent. The caller’s Authorization, Cookie and X-API-Key are withheld: a scheduler decides on the model and the body, and forwarding the credential your client authenticated with would put it in a third party’s logs for nothing.

Trust the picker’s certificate

An https:// picker is verified against the pod’s trust store. An ordinary certificate needs nothing configured here.

A picker whose certificate comes from a mesh CA, or from your own cert-manager issuer, needs that CA in the store. There is no field for it on the pool. A bundle is a path on one machine, and providers is the half of the configuration that travels to every other one, through the admin API, a CRD and the raft log. Deliver it the way the deployment delivers any other trust. For the chart that is one value:

trustBundle:
  configMapName: internal-cas
  key: ca-certificates.crt

The ConfigMap replaces the roots the image ships rather than adding to them, so it has to carry the public roots as well. cert-manager’s trust-manager writes that shape from a Bundle with useDefaultCAs: true. One bundle then covers every outbound connection the gateway makes, which is usually what a private CA is for.

A trust store cannot answer for the reference pickers. Upstream turns TLS on by default and serves a certificate it generates in memory at startup: nothing signs it, it is written nowhere, and it changes on every restart, so no bundle can hold it. Say that you are not verifying:

      picker:
        service: https://llama-epp.llm-d.svc.cluster.local:9002
        insecure_skip_verify: true

Setting it on an http:// picker is refused. There is no certificate there to verify. The gateway logs a warning on every connection it makes without verification.

The exposure is bounded by the fence, which sits on pistra’s side and is not part of the trust decision. Anything that can occupy the picker’s address can answer as the picker, and the endpoints its answer may contain are still only the ones the pool’s own name resolves to. An unauthenticated picker can therefore influence which model server serves a request, and cannot send it anywhere else. Bounded is still not safe. Prefer a signed certificate where the deployment can issue one, and prefer running the picker with TLS off over claiming a verification that is not happening.

The answer is checked

An endpoint the picker names is dialed only if it is in the set the pool’s own name currently resolves to. An address outside it is refused, logged, and the request falls back.

The check bounds arbitrary-destination forwarding rather than any particular picker’s judgement. A destination chosen by another process and dialed unchecked is a forwarder for whoever holds that process.

It costs currency in the member set. The picker watches the Kubernetes API and pistra watches DNS, so a pod the picker already knows about can be one pistra has not resolved yet. Membership is re-resolved every 5 seconds, which bounds that window. A pick inside it is refused, and the request is served without the picker’s help.

Confine a key to part of the pool

A profile’s pool_subset still applies, and the picker is told about it rather than worked around:

    pool:
      service: http://vllm.gpu.internal:8000
      picker:
        service: http://llama-epp.llm-d.svc.cluster.local:9002
      subsets:
        - name: premium
          service: http://vllm-premium.gpu.internal:8000

A confined request carries the subset’s resolved addresses to the picker as the set it must choose from, so the scheduler never has to have heard of your subset names. The fence then checks the answer against the subset as well as the pool, because a confinement enforced only by asking politely is not enforced.

When the picker does not answer

Two settings say what a failure to answer costs:

      picker:
        service: http://llama-epp.llm-d.svc.cluster.local:9002
        on_error: fallback   # or refuse
        timeout: 250ms

fallback is the default. The request is spread over the pool’s members in turn, exactly as a pool with no picker is served, so the degraded path is the one every unscheduled deployment already exercises. A scheduler that stops answering is a degradation and not a policy bypass, because the fence is on pistra’s side and nothing reaches an endpoint outside the pool either way.

refuse fails the request instead. Use it where being scheduled is the point, such as disaggregated serving, or a pool whose servers are not interchangeable and being spread over them arbitrarily is worse than a 503.

A picker refusing a request is not this case. Its own 503 for nothing eligible, or 429 for shedding load, is that request’s answer and reaches the caller whichever value on_error holds.

timeout defaults to 100ms. It is spent before a byte goes upstream, so it is a floor under every pooled request.

Pair it with retries and failover

Scheduling chooses the server before the send. It does not make the send succeed, and these three settings are worth setting together on a self-hosted fleet:

providers:
  - name: llama-fleet
    dialect: openai
    models: ["llama-*"]
    pool:
      service: http://vllm.gpu.internal:8000
      picker:
        service: http://llama-epp.llm-d.svc.cluster.local:9002
    first_byte_timeout: 20s
    retry:
      attempts: 2
      when: [connect, 5xx, timeout]
    failover:
      to: [together]
      when: [connect, 5xx, timeout]

  - preset: together
    api_key: ${TOGETHER_API_KEY}
    first_byte_timeout: 20s

Each attempt asks again, so a retry lands wherever the picker sends it next rather than back on the server that just failed. first_byte_timeout catches a server that accepted the request and is queuing it behind a long backlog. The attempt is abandoned at 20 seconds and the next one goes elsewhere. It is opt-in because the abandoned request may still run to completion on the server that took it, GPU time the gateway cannot recall, and timeout has to appear in when to fire. Naming it obliges every candidate to carry a bound as well, so together sets one. An attempt that can time out at the fleet and then hang at the fallback would have moved the wait, not removed it.

Retries run out before failover begins. There are two attempts at the fleet, then the hosted provider, each admitted under the same key, budget and guardrail decisions as the first. The failover candidate does not need to claim the model. It needs to be able to serve it.

Check it

One series says whether the pool is being scheduled at all:

$ curl -s localhost:9464/metrics | grep 'pistra_pool_'
pistra_pool_picks_total{provider="llama-fleet",outcome="epp"} 1883
pistra_pool_picks_total{provider="llama-fleet",outcome="fallback"} 0

Outcome	Means
epp	The picker chose an endpoint and pistra dialed it.
refused	The picker decided against the request. Its 503 or 429 reached the caller.
fallback	The picker did not answer. The request was spread over the members in turn instead.
failed	The picker did not answer and on_error: refuse turned that into a refusal.
round_robin	The pool declares no picker. Spread in turn.
unresolved	The pool’s name answered with nothing, so the request went to the pool address.

fallback is the one to alert on. The pool is still serving, so nothing else reports that it has quietly stopped being scheduled.

When there is no pool: at all

A pool already fronted by a gateway is not a pool to pistra. It is a provider whose address is that gateway’s:

providers:
  - name: llm-d
    dialect: openai
    models: ["llama-*"]
    base_url: http://llm-d-gateway.llm-d.svc.cluster.local:80

The gateway does model-to-pool routing and scheduling inside the pool. pistra does keys, budgets, guardrails and failover to another provider when that one is unhealthy. Use this when a gateway is already in the path. Use pool: with a picker: when you would rather pistra reached the model servers directly and asked the scheduler itself.

Related

• Inference pools, the two shapes drawn out, who owns the picker protocol, and why a pool is declared rather than detected.
• Configuration reference, pool, and the retry and failover keys it pairs with.
• Channels and fidelity, what a failover candidate has to preserve to be admitted.