Rotate the cluster secret

The cluster secret does three jobs. It authenticates members to each other, it derives the certificate authority under which they speak TLS to each other, and, unless a cluster.kek_file takes the job, it is the key-encryption key (KEK) that wraps the data key. Every sealed value depends on that data key: stored secrets and ACME certificates. The two are kept apart by the keyring, so changing the secret is three small steps and never a re-encryption of anything.

The keyring holds each term of the data key wrapped under every KEK the cluster has been told about. A sealed value names its term. GET /admin/v1/secrets/keyring shows it, ids only, never a key:

$ pistra admin /admin/v1/secrets/keyring
{"version":1,"current":1,"terms":[{"id":1,"created_at":"…","keks":["cluster:5f1e…"]}],"held":["cluster:5f1e…"],"opens":[1]}

held is what this node derives from its own secret and key file. opens is which terms that lets it open.

1. Wrap under the new secret, while the old still works everywhere

Run this command:

$ printf '{"op":"add","kek":{"kind":"cluster-secret","secret":"%s"}}' "$NEW_SECRET" \
    | pistra admin POST /admin/v1/secrets/keyring -d @-

Every term is now wrapped under both ids. The material is used once to derive a key and is not stored. Send it through the local socket or over TLS, and from stdin rather than the command line. This is refused, not partially applied, if the node cannot open every term.

2. Roll the nodes to the new secret

Update the Secret the chart’s envFrom delivers PISTRA_CLUSTER_SECRET from and roll the StatefulSet. Each node comes up deriving the new id, finds a wrapping for it, and opens everything. Nodes not yet rolled keep opening under the old id. Nothing sealed is touched.

The roll is a partition while it lasts. A node on the new secret and a node on the old one hold different certificate authorities and refuse each other’s connections, on the raft transport as on the forward listener, so a rolled node has no leader until a majority has rolled, and serves the state it had. Roll one node at a time, which is what the StatefulSet does, and expect writes through the rolled minority to fail with no leader until the majority is over. A three-node cluster is over it after the second pod.

3. Drop the old wrapping

Remove the old wrapping only after every node can open the new one:

Run this command:

$ pistra admin POST /admin/v1/secrets/keyring -d '{"op":"drop","kek_id":"cluster:5f1e…"}'

Caution

The last wrapping of any term cannot be dropped. A mistyped id leaves the keyring readable, but dropping a valid old wrapping too early can leave an unrolled node unable to open its data.

Rotating the data key itself

Run this command after a suspected compromise of the key material, not of the secret:

$ pistra admin POST /admin/v1/secrets/keyring -d '{"op":"rotate"}'

A new term seals every write from now on. Values sealed earlier stay readable under their term. Rewriting them under the new term is a PUT of each secret and a certificate renewal. That is not automated, because only you know whether the old ciphertext ever left the cluster. The node performing the rotation must hold every KEK the current term is wrapped under. Drop the wrapping it cannot honour first.

Moving the key off the cluster secret

Deliver a file of at least 32 bytes of key material to every node and name it as cluster.kek_file. Each node then holds two KEKs. Add a wrapping under the file’s key from any node ({"op":"add","kek": {"kind":"file","path":"/etc/pistra/kek"}}, the path is read on the node that takes the request), and drop the cluster secret’s. From then on the join credential and the storage key rotate apart, and a leaked cluster secret opens no sealed value.

This does not give you a key that never rests on a node. The file is on every node’s disk. That is the job of a KMS-backed KEK, which the keyring is shaped to take next and does not have yet.

Policy

Both operations are instance keyring in class secrets:

@id("platform-rotates-keys")
permit(principal in Role::"platform", action in [Action::"getKeyring", Action::"rewrapKeyring"], resource == Resource::"secrets/keyring");