What a cluster survives

Three replicas is not by itself a deployment that survives anything. What it survives depends on where the three are placed, and raft decides the rest. This page names each failure domain in turn, says what the chart’s defaults survive in it, and says what the loss of a quorum actually costs.

The arithmetic is the same in every shape. Replicas are raft voters, and a raft cluster tolerates floor((n-1)/2) failures. One of three, none of two, one of four. Four voters tolerate what three do, so the fourth buys nothing, and the chart refuses an even count unless cluster.allowEvenVoters permits it. Scale by two.

What a lost quorum costs

Traffic does not stop. Each node serves from the state it already holds, so requests keep flowing under whatever configuration was last applied. Keys already minted still authenticate. Rules already loaded still refuse. Detectors already configured still inspect.

What stops is every write. No key can be minted, no budget delta can land, no configuration source can be written, and no provider can be suspended. Every door out of that state is itself a write, which is what makes it different from the other failures in this documentation. Recover a cluster that has lost quorum is the way back.

Read that as two separate outages. Enforcement continues on the last policy the cluster agreed on. Governance stops. A team that needs a key cannot get one, and spend recorded during the outage is bounded by what the nodes can hold unflushed. Alert on pistra_cluster_leader_known, which reads 0 on a node that is partitioned or in a cluster with no quorum.

One node

A cluster of one commits, because it is its own quorum. It survives nothing. Its restart is the gateway’s outage, and the loss of its volume is the loss of every key, budget ledger and certificate in it.

This is the right shape for development, and for a deployment where an outage is acceptable and a backup covers the volume. It is not a degraded three. It is a different promise.

Across nodes

The chart spreads replicas over kubernetes.io/hostname, and pdb.maxUnavailable: 1 keeps a voluntary disruption to one pod at a time. Two of three voters drained at once is a cluster with no leader.

The constraint is whenUnsatisfiable: ScheduleAnyway, so a cluster with fewer nodes than replicas still schedules. That is deliberate: a single-node development cluster comes up rather than hanging with pods pending. It also means the spread is a preference, and a cluster that cannot honour it gives you three voters on one host without saying so. Check the placement rather than assuming it.

Run this command to see where the voters actually landed:

$ kubectl get pods -n pistra-system -l app.kubernetes.io/name=pistra \
    -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName

Across zones

The chart also spreads over topology.kubernetes.io/zone, with the same ScheduleAnyway. Three voters in three zones survive the loss of a zone.

The same concession applies, and it bites harder here. In a two-zone cluster the constraint is satisfied as well as it can be, and two of the three voters share a zone. Losing that zone is losing the quorum. A cluster with three zones and a node pool in only two is the same case. Spread a cluster across zones covers what to set and what to check.

A commit now crosses a zone, so it is worth naming what waits on one. Minting a key, revoking one, changing the membership, writing a configuration source and issuing an ACME certificate are raft writes. Each node also offers its signed audit head to the cluster about once a minute. Revocation takes effect in microseconds on the node that serves it and about 30 ms across a cluster, and a zone hop is added to the second figure rather than the first.

The request path is not a raft write. Budgets in particular do not lease their limit through the log. Each node meters only the traffic it served, and a bucket holds three quantities: the durable cluster total it last read, this node’s settled spend that has not reached the store yet, and its live reservations. A flush tick writes the second away and re-reads the first.

The concession is stated rather than hidden. The residual error is the other nodes’ unflushed spend, bounded by the same interval that bounds the crash window, so a cluster limit is a bounded overspend and not a limit that could never be exceeded. Leasing slices of a limit through raft would make it one, at the price of an fsync per request inside the overhead budget. Spreading a cluster across zones therefore costs control-plane latency and does not cost request latency.

Across regions

One raft cluster does not span regions. Every commit would pay the round trip between them, and a partition between two regions leaves at most one side able to commit anything.

Run one deployment per region. Each is a cluster of its own, with its own keys, its own budget ledgers, its own certificates, and its own signing key per node. That is not a workaround. The audit trail is already per node, and pistra audit verify reads trails from any number of nodes in any order, so evidence from every region verifies together without the regions sharing a log.

What does not carry across is anything the log holds. A key minted in one region does not authenticate in another. A budget is a cluster limit, so two regions under one policy enforce two limits and not one shared one. Decide that deliberately rather than discovering it.

What none of this covers

Availability here is the gateway’s. A provider that is down is a provider that is down, and pools and retries are the answer to that. A collector that is unreachable does not stop the local copy of the audit trail, which is written by the node either way. The architecture page has the four planes this splits along.