Identity signing-key and authority-fence provisioning

Source of truth: docs/runbooks/identity-signing-key-provisioning.md (synced at build time — edit the original, not this copy). This runbook covers provisioning the identity signing key ring and the authority fence for the Veltro identity authority — the ES256 key(s) the gateway-authorized request-assertion surface signs with, and the single-row fence that names which deployment may mint sessions at the current generation. Both belong to standing up a fresh identity database. Key rotation and fence advances (signing-tool fence-advance, break-glass) are separate, deliberate operator actions and are not covered here as boot steps.

Why this exists

Every gateway-authorized product route (/sigmatic/…, /warden/…) is gated by an internal nginx auth_request to /api/authority/browser-assertion, which mints a browser request assertion through the signing key ring. The ring is persisted encrypted in veltro.identity_signing_keys (owned migration 007), protected by an out-of-band passphrase. Migration 007 creates the table but no row: a fresh identity database starts with an empty ring, and buildSigningKeyRing fails closed (signkey_ring_empty) rather than mint unsigned. An unprovisioned composed boot therefore 500s every authorized route while looking otherwise healthy — the identity readiness probe does not exercise minting.

Boot-time provisioning: identity-init (automatic, idempotent)

In Compose, one identity-init service does all of identity’s bootstrap (#1690). It runs the same pinned identity image (node apps/identity/signing-tool/boot-init.cjs) once per up and exits. In order, it:

1. applies the owned migrations;
2. mints the first active signing key when the ring is empty;
3. claims the deployment issuer and writes generation 1 of the authority fence when there’s no fence;
4. creates identity’s own veltro-identity workload credential;
5. creates or re-emits one workload credential per installed product (VELTRO_PRODUCTS) and writes its secret to /run/suite-workload-secrets/<product>;
6. creates the TOTP keyring in the identity-keyring volume when there isn’t one (see below).

It’s the only service that gets the owner migration DSN (VELTRO_IDENTITY_MIGRATION_DATABASE_URL). It also gets the signing passphrase, both from the environment, never on a command line. The veltro identity service and every product backend wait for condition: service_completed_successfully on it, so nothing is served before the ring holds a key and the fence is claimable. Its output is one line of booleans and counts per step, never a key, DSN or issuer id.

Every step is idempotent. A re-run with an active key keeps it and exits 0: it never adds, rotates or replaces a key, so a second key in the ring on its own is a tamper or operator signal.

scripts/validate_identity_credentials.py (run by make pull / make up before Docker) fails closed when VELTRO_IDENTITY_SIGNING_PASSPHRASE is present but shorter than the 12-character minimum the sealing envelope enforces (assertPassphrase), so a boot that can’t provision fails at its source, naming only the variable.

Helm still runs these steps as separate Jobs (job-migrate, job-provision-signing-key, job-provision-authority-fence, job-provision-identity-workload), from the boot-*.cjs bundles in the same image.

The TOTP keyring

The keyring seals TOTP factors, the upstream OIDC client secret, the SCIM token and notification destination secrets. identity-init keeps it at /var/lib/veltro-identity/keyring/destination-totp-keyring.json in the identity-keyring volume (veltro-identity-keyring), owned by uid 1000 at mode 0600. Identity mounts the same volume read-only.

• No keyring and nothing sealed in the database: identity-init creates one.
• A keyring there: it’s checked with identity’s own loader and never rewritten. It also has to hold every key id the database stores beside sealed data. If one is missing (the wrong backup, the wrong host file), identity-init stops and names the missing ids.
• No keyring but the database holds sealed data: identity-init stops with an error and creates nothing. A new key couldn’t open that data. Restore the keyring (from a backup, or from the host file below) and run up again.

Backups seal the volume’s archive under VELTRO_BACKUP_PASSPHRASE, and restore puts it back (see backup-restore).

Moving the TOTP keyring into its volume

Before #1690 the keyring was a host file: VELTRO_IDENTITY_TOTP_KEYRING_FILE_SOURCE in .env, relative paths under compose/, usually compose/secrets/veltro-identity/destination-totp-keyring.json. That file has to land in the volume before identity-init first runs.

On a CLI install you don’t do anything. veltro upgrade, veltro up, veltro install, restore and rollback check for that host file before they start the stack. When it’s there, a short-lived identity-init container (docker compose run --rm --no-deps, the pinned image, the host file mounted read-only) copies it into the volume if the volume has no keyring yet, hands it to uid 1000 at 0600, and leaves the host file where it is. Once the copy is there, later starts see the same bytes and move on.

If the volume already holds a different keyring, the CLI stops and changes nothing. Usually that means identity-init made a fresh keyring because a plain docker compose up ran before the copy. Two ways out:

• The volume’s keyring is the right one (say you just restored a backup): move the host file aside and remove VELTRO_IDENTITY_TOTP_KEYRING_FILE_SOURCE from .env.

• The host file is the one your data is sealed with and nothing new was enrolled since: stop the stack, remove the volume’s keyring, and start again so the copy runs:

  docker run --rm -v veltro-identity-keyring:/k alpine:3.20 \
    rm /k/destination-totp-keyring.json
  veltro up

Homelab promotion can’t do this step. It only ever triggers DeployStack on the Komodo stack, and a homelab overlay can’t add a service. Do it by hand on the homelab Docker host, once, before you promote the first release with identity-init. KEYRING is the absolute path the stack’s VELTRO_IDENTITY_TOTP_KEYRING_FILE_SOURCE points at, and the labels match the Compose project Komodo runs the stack as (the stack name, veltro):

KEYRING=/path/to/destination-totp-keyring.json
docker volume create \
  --label com.docker.compose.project=veltro \
  --label com.docker.compose.volume=identity-keyring \
  veltro-identity-keyring
docker run --rm -v veltro-identity-keyring:/k -v "$KEYRING":/src:ro alpine:3.20 sh -euc '
  dst=/k/destination-totp-keyring.json
  if [ -e "$dst" ]; then
    cmp -s /src "$dst" && echo present && exit 0
    echo "the volume holds a different keyring; nothing changed" >&2; exit 3
  fi
  umask 077
  cp /src /k/.seed.tmp && chown 1000:1000 /k/.seed.tmp && chmod 0600 /k/.seed.tmp
  ln /k/.seed.tmp "$dst" && rm /k/.seed.tmp && echo seeded'

It prints seeded the first time and present after. Then promote. Also check homelab-overlay.yml in komodo/veltro before that promotion: an entry for identity-migrate, any identity-provision-* service, or a veltro.volumes list that still binds the keyring file has to go, or the render refuses the stale entry (a volumes list replaces the release’s whole list, which would drop the identity-keyring mount). The stack’s VELTRO_IDENTITY_TOTP_KEYRING_FILE_SOURCE and VELTRO_IDENTITY_TOTP_KEYRING_FILE variables are unused afterwards.

When the old host file can go

Keep the host keyring file until both of these are true:

• a backup taken on the new release exists. Its envelope holds the volume’s keyring, so the host file is no longer the only copy outside the volume;
• you won’t roll back to a release from before identity-init. Those releases bind the host file into identity and never read the volume, so rolling back without it leaves identity without a keyring.

Then delete the file and remove VELTRO_IDENTITY_TOTP_KEYRING_FILE_SOURCE (and VELTRO_IDENTITY_TOTP_KEYRING_FILE) from .env, or from the homelab stack’s environment. Delete it before the next time you’d rotate the keyring, too: once the volume’s keyring changes, the old file no longer matches it and the next start stops on the difference.

Authority fence boot-time provisioning (automatic, idempotent)

identity-init also provisions the deployment issuer and authority fence (issue #752). Without these rows, credentials verify and then prepareSession answers 503 SECURITY_CONFIGURATION_UNAVAILABLE on every successful login — forever — because nothing else writes veltro.deployment_issuer or veltro.identity_authority_fence.

Sequencing, if you are changing a bootstrap step. Compose runs boot-init.cjs out of the pinned identity image, and that file exists only because the protected publication job bundles it. A step identity waits on therefore can’t ship before the pinned image contains it: the container exits MODULE_NOT_FOUND, service_completed_successfully never holds, and the whole deployment stops at boot instead of merely refusing logins. Since ADR-016 a release builds the identity image from its own commit and writes that version into versions.env, so a release always carries the bundle its compose file runs. The hand-set VELTRO_VERSION on main can predate a step, so a stack started from a main checkout needs a released pin (or a sha-<commit> image) that has it.

Sequencing, the other direction: a receiver and its material move together. The gateway mints a browser request assertion on every product route, and a correct receiver fails closed on a presented assertion it cannot verify — falling back to the session cookie after a presented credential fails verification would be a bypass. So the coupling has two failure modes, not one:

• Restoring the gateway’s Cookie strip before the release carrying the receiver is pinned refuses every browser request (veltro#758, #772).
• Pinning a release that contains a receiver before that product’s compose block carries its SUITE_AUTHORITY_V2_* settings refuses every browser request too.

The second cost a release train: WARDEN 0.11.1 carried warden#138’s receiver before Warden got its keys, and /warden/api/auth/me went 200 → 401 on the version bump by itself. Give the product’s compose block the issuer, generation, audience and identity’s JWKS URL (http://veltro:3000/.well-known/jwks.json) in the same change that moves the pin.

Each product’s SUITE_AUTHORITY_V2_AUDIENCE is a compose literal. The suite .env is shared, so one scalar there resolves every product to the first audience and each receiver then refuses for “wrong audience” (veltro#775).

Idempotency and refusal

• Matching fence (restart): if a fence already exists and names this deployment, the fence step is a proven no-op — same issuer, same generation, no new fence event.
• Foreign fence (do not soften): if a fence exists naming a different deployment, identity-init fails closed and writes nothing. A booting container must never take authority from the active issuer and must never “repair” or adopt that fence. That refusal is the security property the cutover fence (#130 / #219) rests on; an update/adopt branch here would make the fence meaningless.
• Generation: boot never advances authority_generation. Only the break-glass signing-tool fence-advance command moves it. Products pin the generation they accept, so after a fence-advance set SUITE_AUTHORITY_V2_ACTIVE_GENERATION in .env to the new value and run veltro up. On Helm, set the Sigmatic and Warden generation values and run helm upgrade; the publish Job writes VectorFlow’s from the fence.

Verify

docker exec veltro-suite-postgres \
  psql -U veltro_identity_owner -d veltro_identity -XAtc \
  "SELECT di.deployment_id, f.active_issuer_id, f.authority_generation
   FROM veltro.deployment_issuer di
   CROSS JOIN veltro.identity_authority_fence f
   WHERE di.id = 1 AND f.id = 1"

Expect one row where deployment_id = active_issuer_id and authority_generation >= 1. A successful login (POST /api/auth/login) must return HTTP 200 and mint a session cookie — never HTTP 503 SECURITY_CONFIGURATION_UNAVAILABLE on a correct credential.

Manual first provisioning (out-of-band identity install)

If you are standing up identity by hand without the composed core profile, after migrations and before starting the identity service:

export VELTRO_IDENTITY_MIGRATION_DATABASE_URL='postgresql://veltro_identity_owner:***@suite-postgres:5432/veltro_identity'
node apps/identity/signing-tool/boot-authority-fence.cjs
# -> issuer_created=true fence_created=true authority_generation=1

(From a source checkout before the image bundle exists, the TypeScript entry is pnpm --filter @veltrosecurity/identity exec tsx signing-tool/boot-authority-fence.ts.)

After an out-of-band provision, make up boots normally: identity-init sees the matching fence and keeps it.

Verify the signing-key ring

On the running stack, the ring must hold exactly one active key:

docker exec veltro-suite-postgres \
  psql -U veltro_identity_owner -d veltro_identity -XAtc \
  "SELECT key_id,status FROM veltro.identity_signing_keys WHERE realm_id='default' ORDER BY created_at"

Through the gateway, GET /sigmatic/ (and every other product route) must return its real application response — never HTTP 500.

Manual first provisioning (out-of-band identity install)

If you are standing up identity by hand (import tooling, recovery rehearsal, or a deployment without the composed core profile), provision the ring right after migrations, before starting the identity service:

export VELTRO_IDENTITY_MIGRATION_DATABASE_URL='postgresql://veltro_identity_owner:***@suite-postgres:5432/veltro_identity'
# VELTRO_IDENTITY_SIGNING_PASSPHRASE — never on a command line:
#   export VELTRO_IDENTITY_SIGNING_PASSPHRASE='<long, random, out-of-band>'
pnpm --filter @veltrosecurity/identity signing provision
# -> provisioned signing key kid-<hex> (active)

provision reads the passphrase from VELTRO_IDENTITY_SIGNING_PASSPHRASE (or a muted stdin prompt), requires the owner DSN, and refuses when an active key already exists — that refusal means the ring is already provisioned and is not an error to paper over. It prints only the key id.

The passphrase must be at least 12 characters and is the same value the identity runtime reads in .env as VELTRO_IDENTITY_SIGNING_PASSPHRASE; it is the key-encryption secret for every envelope in the ring, so it must be escrowed out-of-band and never committed.

After an out-of-band provision, make up boots normally: identity-init sees the active key and keeps it.

Rotation and passphrase change (do NOT do at boot)

Rotation is signing-tool rotate: it retires the active key to verify-only and mints a new active key. Restart identity afterwards so it loads the new ring. Products fetch identity’s public keys from its JWKS (/.well-known/jwks.json, veltro#1692), which lists the active key and every retired one, so assertions signed before the rotation still verify. A product that sees a key id it doesn’t know fetches the JWKS again, so it picks up the new key on its first assertion without a restart.

Passphrase rotation is rekey (see the same CLI’s help). Both are operator ceremonies with their own sequencing. identity-init never rotates: if the ring has no active key at boot and one is needed, restore/import it through the recovery tooling rather than fabricating a new key at startup.