Watch the gateway

Every signal the gateway produces follows one convention, and none of it is configured in the document. Metrics are a Prometheus registry on a listener of their own. Traces and logs are OpenTelemetry, switched on by the standard OTEL_* environment, and the audit trail rides the same switch on its own scope. This page is where each one comes out, and what on the dashboard is worth an alert.

Metrics: scrape it, or have it pushed

/metrics is served on metrics_listen, :9464 by default, and nowhere else. It is not on the data plane, because the data plane is the front door and a scrape is a list of every route, provider and model you use with counts beside them. The listener is plain HTTP for the pod’s own perimeter. Reach it from your Prometheus and from nothing else.

$ curl -s localhost:9464/metrics | grep '^pistra_tokens_total'
pistra_tokens_total{model="gpt-4o",provider="openai",route="chat_completions",token_type="input"} 18342
pistra_tokens_total{model="gpt-4o",provider="openai",route="chat_completions",token_type="output"} 2911

On Kubernetes the chart puts it on the metrics container port and, with metrics.podMonitor.enabled, renders a PodMonitor over every pod, because which one leads and what each one holds are per-pod facts. With networkPolicy.enabled the policy admits the namespaces in networkPolicy.metricsFrom to that port and nobody else. The chart refuses a PodMonitor with an empty metricsFrom, because a scrape that installs cleanly and never succeeds is the failure nobody notices.

If your collector takes OTLP instead, the same registry is pushed:

$ OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
  OTEL_METRICS_EXPORTER=otlp OTEL_METRIC_EXPORT_INTERVAL=30000 pistra -config pistra.yaml

The collector receives exactly the series a scrape would, under the same names, on the resource every other signal shares. There is one set of instruments and two ways out. A deployment that pushes can set metrics_listen: none and serve nothing.

Neither is a reason to leave OTEL_METRICS_PRODUCERS set. Autoexport’s own prometheus producer reads Go’s default registry, and nothing of the gateway’s lives there.

What a scrape costs, and what it discloses

Every label in the registry is a bounded set, and two of the bounds are deliberate rather than incidental. The route label collapses the request path to eight values, chat_completions, messages, completions, embeddings, responses, realtime, models, other, so a client cannot mint series by requesting paths. And model is only ever set from a provider’s own response, so a model that was never served never reports a token.

Your configuration grows the series, not your traffic:

Label	Grows with
provider	providers you configured
model	models a provider actually served
budget, rule, detector	names in your configuration
entity	entity types your detectors can produce, the coverage page is the ceiling
everything else	nothing: fixed vocabularies, mostly two to four values

The series that multiplies is pistra_tokens_total, at route × provider × model × token_type. Five providers serving four models each over two routes, at the five token types, is 200 series, small, and worth knowing the shape of before you add a hundred models.

No label carries a caller, an end user or content. There is no key label. A per-key cut is the ledger’s job, a per-caller one is the audit trail’s, and a label a client can mint is unbounded cardinality served to whoever can scrape. Guardrail labels name entity types and the detectors that found them, never the matched text.

That is the disclosure to reason about when deciding who may scrape. A scrape is not content, but it is a list of every route, provider and model you use with counts beside them. /metrics is on its own listener and never the front door.

The dashboard

deploy/grafana/pistra-gateway.json is a Grafana dashboard over the registry: overhead as the receipt, tokens as the bill, guardrail decisions, the cluster, and the metering ledger against the provider’s. Import it into any Grafana with a Prometheus data source. With the sidecar that kube-prometheus-stack ships, either chart installs it for you:

metrics:
  podMonitor:
    enabled: true
  grafanaDashboard:
    enabled: true
    annotations:
      grafana_folder: pistra

The series it names are checked against the registry in internal/metrics, so a rename on one side fails a test rather than leaving a panel blank.

What to alert on

These are shipped, not just described. deploy/prometheus/pistra-rules.yaml is the table below as fourteen alerting rules, load it with rule_files:, or have either chart render it as a PrometheusRule:

metrics:
  prometheusRule:
    enabled: true

Four thresholds in it are deployment-specific and marked TUNE: the error budget, the overhead percentile, the metering drift tolerance and the judge’s token rate. They ship with defaults that are defensible rather than correct for you, and an alert that fires constantly is one people mute.

The table is the same shortlist in prose. Every series the gateway exports, with its type, its labels, its histogram buckets and the help text it publishes, is in the metrics reference.

Series	Condition	What it means
pistra_cluster_leader_known	0 on any node for more than an election	That node is partitioned, or the cluster has lost quorum, see recovering a cluster
pistra_cluster_members{suffrage="voter"}	above the replica count	A departed node is still counted toward quorum
pistra_audit_export_failures_total	any value above zero	Audit records the collector never received; the node’s text log is their only copy
pistra_overhead_seconds	p99 above what the performance page measured for your shape	The gateway itself got slower, the one latency that is its fault
pistra_request_duration_seconds_count{code=~"5.."}	ratio above your error budget	Whichever side failed, the client saw it
pistra_budget_used_ratio	above 0.8	A budget is about to start refusing; pistra_budget_buckets{state="exhausted"} says when it has
pistra_guardrail_detector_errors_total	any steady rate	A fail_open detector is failing every time, and nothing is being inspected by it
pistra_guardrail_async_total{disposition="dropped"}	any	async.max_in_flight was reached; those requests were never scanned
pistra_guardrail_model_tokens_total	a rate the delta cache should not allow	An llm judge is being asked about text it has seen, a cache too small for the traffic, or single-shot traffic where every segment is new; either way this is the bill for judging
pistra_reconcile_drift_ratio	above the tolerance you set	The wire and the provider disagree about what you were billed, see auditing your metering
pistra_config_revision	differs between nodes	A node has not applied the latest revision

Traces and logs

Tracing is off until asked for, and the environment is how you ask:

$ OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 pistra -config pistra.yaml

That one variable turns on traces, logs and metrics together. OTEL_TRACES_EXPORTER, OTEL_LOGS_EXPORTER and OTEL_METRICS_EXPORTER select them one at a time, and OTEL_SERVICE_NAME renames the resource. Every request is one server span in the GenAI semantic conventions, gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens. The upstream call is its child, and the client’s traceparent is its parent when it sends one. Those conventions are on the spans only. The metrics keep the gateway’s own names, so a dashboard built for the GenAI metric conventions finds none of them.

Logs are never redirected. The text log stays on stderr for the host, and when OTLP logs are on each record is also sent through the bridge, stamped with the trace and span of the request it was logged under. The bridge exists for that correlation.

The audit trail is a separate logger on the same switch, its own scope, synchronous delivery, and a local copy that is always written. The audit trail is why it is separate. The audit trail reference is every event it records, every field of a record and the attributes a collector can route on without parsing a body.

The environment in full

Everything below is the standard OpenTelemetry environment, honoured because the gateway configures no exporter of its own. There is no telemetry: block in the configuration document, and that is deliberate. Where a signal goes is a property of the host, not of the deployment every node shares.

Variable
OTEL_EXPORTER_OTLP_ENDPOINT	turns on all three signals at once. Per-signal: OTEL_EXPORTER_OTLP_{TRACES,LOGS,METRICS}_ENDPOINT
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf (the default) or grpc. The default is why the examples say :4318; pointing at :4317 without setting this sends HTTP to a gRPC port
OTEL_{TRACES,LOGS,METRICS}_EXPORTER	otlp (default), console, or none, select one signal without turning on the others
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES	the resource every signal shares
OTEL_EXPORTER_OTLP_HEADERS	what a hosted collector’s API key goes in. Also _TIMEOUT, _COMPRESSION, _CERTIFICATE, _CLIENT_CERTIFICATE, _CLIENT_KEY, _INSECURE, each with a per-signal form
OTEL_TRACES_SAMPLER	parentbased_always_on (default), always_on, always_off, traceidratio, parentbased_traceidratio, parentbased_always_off; the ratio goes in OTEL_TRACES_SAMPLER_ARG
OTEL_BSP_*	the span batch processor: SCHEDULE_DELAY, MAX_QUEUE_SIZE, MAX_EXPORT_BATCH_SIZE, EXPORT_TIMEOUT
OTEL_BLRP_*	the same four for the operational log batch processor
OTEL_METRIC_EXPORT_INTERVAL	how often the registry is pushed
OTEL_METRICS_PRODUCERS	leave unset. See above

OTEL_BLRP_* does not reach the audit trail. Those settings tune batching, and the audit provider does not batch. It exports on the emitting goroutine so that a record has either left the process or been counted as lost. Tuning the log queue changes the operational log and nothing about the trail.

Sampling is worth one thought before you set it. A sampled-out request is still metered, still inspected and still audited. Sampling drops the span, not the enforcement, so traceidratio costs you the trace of an incident, never the record of one.

On Kubernetes, the environment goes in the chart’s env:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector.monitoring:4318