Suite backup, restore, and disaster recovery

Source of truth: docs/runbooks/backup-restore.md (synced at build time — edit the original, not this copy). This runbook covers bulk state for Veltro identity and whichever products are installed: VectorFlow, Sigmatic, Warden, and their datastores. veltro backup is the only backup (ADR-016). Sigmatic’s Export / import detection content moves rules and their configuration between installs, but it isn’t a backup.

Machine entry points:

Entry	Purpose
veltro backup / veltro restore	Operator CLI (once veltro-cli is installed)
python3 scripts/veltro_backup.py backup|restore	Same library without the CLI package
scripts/veltro-backup.sh / scripts/veltro-restore.sh	Thin POSIX shims
compose/backup.yml (--profile backup)	Optional scheduled sidecar

Archive schema: veltro-backup/v1, datastore layout suite.

What a backup contains

A veltro-backup-<timestamp>.tar holds:

Artifact	Source	Format
postgres/suite-postgres-veltro_identity.dump	core: suite-postgres database veltro_identity	pg_dump -Fc as suite_admin
postgres/suite-postgres-vectorflow.dump	VectorFlow: database vectorflow (TimescaleDB)	pg_dump -Fc as suite_admin
postgres/suite-postgres-sigmatic.dump	Sigmatic: database sigmatic	pg_dump -Fc as suite_admin
postgres/suite-postgres-warden.dump	Warden: database warden	pg_dump -Fc as suite_admin
postgres/suite-postgres-roles.json	roles the databases reference that aren’t the suite’s own, with attributes, settings, memberships and SCRAM verifiers	JSON (only when such roles exist)
redis/suite-redis-db0.bin	VectorFlow: suite-redis index 0	Redis key dump (see below)
redis/suite-redis-db1.bin	Sigmatic cache: suite-redis index 1	Redis key dump
redis/suite-redis-db2.bin	Sigmatic queue: suite-redis index 2	Redis key dump
redis/suite-redis-db3.bin	Warden: suite-redis index 3	Redis key dump
opensearch/snapshot.tar.gz	Sigmatic: fs repository at /usr/share/opensearch/snapshots (path.repo in compose/sigmatic.yml), indices sigmatic-*	tar of snapshot files
volumes/{vf,sigmatic,warden}-data.tar.gz	selected products’ named volumes, by the name the Compose model gives them	tar.gz
volumes/suite-workload-secrets.tar.gz	core: product-peer workload secret drops	tar.gz
secrets/identity-secrets.envelope.json	the identity-keyring volume’s tar.gz, which holds identity’s TOTP keyring. Archives from before #1690 hold the keyring host file instead; restore puts either into the volume (and skips the legacy keyring and bootstrap-admins files even older archives sealed)	veltro-secret-envelope/v1 (openssl AES-256-CBC + PBKDF2)
identity/evidence.json	best-effort veltro ledger / realm probe	recovery-contract alignment
env/versions.env	pin file	plain text
manifest.json	digests, datastore_layout, datastores, selected products, module pins, backup_status, skipped artifacts, notes, and per database its database_properties (ACL, connection limit, database and role-in-database settings) and referenced_roles	veltro-backup/v1

Datastore layout

manifest.json records "datastore_layout": "suite" and a datastores block with the suite-postgres databases in the archive and a map of suite-redis index to product. An archive without datastore_layout comes from the per-product datastores used before ADR-014 step 5. This release refuses to restore those; see Archives from the per-product datastores.

The Postgres dumps run inside suite-postgres as suite_admin. SUITE_POSTGRES_ADMIN_PASSWORD from .env reaches pg_dump as PGPASSWORD through the environment, never through argv.

pg_dump without --create leaves out what belongs to the database itself. So each database’s artifact in the manifest also records its ACL, connection limit, ALTER DATABASE … SET and ALTER ROLE … IN DATABASE … SET settings, and every role it references. Roles that aren’t the suite’s own travel in postgres/suite-postgres-roles.json with their password verifiers. The verifiers appear nowhere else: not in manifest.json, not in any command line or output. Backup refuses a role it couldn’t recreate (superuser, CREATEROLE, CREATEDB, REPLICATION, a non-SCRAM password, or a membership of a role that isn’t carried).

Redis is dumped key by key while suite-redis keeps serving. A short-lived python:3.12-alpine3.21 container joins suite-redis’s network namespace (docker run --rm -i --network container:<id>) and runs a stdlib helper that SCANs one index and writes each key’s DUMP payload and PTTL. Keys that expire or get deleted during the walk are left out. The artifact is a framed binary file: a VELTRO-REDIS-DUMP/1 header with the index number, one record per key (key, remaining TTL in milliseconds or -1 for none, DUMP payload), and a trailer with the record count. Backup checks the framing and stores the count as keys on the artifact in the manifest.

What is deliberately excluded by default

• .env — not embedded. The manifest records env_sha256 so restore can detect a credential mismatch. Pass --include-env only when you accept the risk; the archive then contains secrets and must be protected like a key.
• Identity Redis rate-limit / failure-window state (transient protection state; see docs/runbooks/identity-database-recovery.md).
• Host Docker images and the compose checkout itself (reinstall from the suite release, then restore data).

Consistency model (honest)

Components are captured sequentially. The manifest sets consistency.is_coordinated_snapshot: false. Do not treat the archive as a single distributed snapshot; after restore, expect brief cross-product lag that workers and outboxes reconcile. Each Redis index is read with a SCAN walk over the live server, so a key that changes during the walk may be captured before or after the change.

When --skip-optional omits an artifact, the manifest records backup_status: partial and names it in skipped_artifacts. Restore surfaces that condition in its plan and reports restored-partial, never an ordinary complete restore.

Create a backup

Prerequisites: core and the selected products are running, including suite-postgres and suite-redis; versions.env and .env are present and .env has SUITE_POSTGRES_ADMIN_PASSWORD; Docker Compose ≥ 2.20; openssl is on PATH; the host can run (or pull) python:3.12-alpine3.21; and VELTRO_BACKUP_PASSPHRASE is set: it seals the identity keyring.

# Selection: .env VELTRO_PRODUCTS, then saved veltro.toml, then all products.
export VELTRO_BACKUP_PASSPHRASE='…'   # operator-held; never commit
./scripts/veltro-backup.sh --to ./veltro-backup.tar

# Override the selection for a core + Sigmatic install.
veltro backup --products sigmatic --to ./veltro-backup.tar

# Core only (an empty selector is intentional).
python3 scripts/veltro_backup.py backup --products '' --to ./veltro-core-backup.tar

The veltro_identity database, suite-workload-secrets and the sealed identity-keyring volume are always included. On an install that hasn’t started a release with identity-init yet (the backup veltro upgrade takes before moving to one), there’s no keyring volume, so the backup seals the host keyring file .env names instead. The tool also probes identity recovery evidence when available, connecting to veltro_identity as its owner role. Other artifacts follow the selected products: VectorFlow adds the vectorflow database and Redis index 0; Sigmatic adds sigmatic, indexes 1 and 2, and OpenSearch; Warden adds warden and index 3. A Sigmatic backup doesn’t touch VectorFlow or Warden data. The manifest records the product list.

Flags:

• --products sigmatic,warden — override .env and saved veltro.toml for one run; pass --products '' for core only.
• --include-env — embed .env (prints a warning; mode 0600 inside the tar).
• --skip-optional: continue when optional database, OpenSearch, volume or Redis backups fail. The archive is marked partial; the scheduled sidecar keeps the newest complete recovery point independently of its normal retention count.
• --schedule "0 3 * * *" — turn on the scheduled sidecar (see below). --backup-dir (an absolute host directory) and --keep set where it writes and how many it keeps.
• --unschedule — turn the scheduled sidecar off and remove it.
• --encrypt encrypts the entire Suite archive with the separately mounted archive password. VELTRO_BACKUP_ARCHIVE_ENCRYPTION=true enables it by default.

Scheduled backups

veltro backup --schedule "0 3 * * *" --keep 7

This writes BACKUP_CRON, the absolute VELTRO_SUITE_ROOT, and any supplied BACKUP_DIR/BACKUP_KEEP to .env, sets scheduled_backups = true in veltro.toml, and starts veltro-backup (Compose profile backup). Mounting the checkout at its host path lets nested Docker Compose jobs resolve the same secret and backup mounts. The sidecar accepts only M H * * * (hour may be *). It keeps the newest BACKUP_KEEP archives plus the newest complete one. Selection comes from .env, then veltro.toml, then all products.

veltro down stops the sidecar with the rest of the stack. veltro up, an upgrade and a restore start it again because veltro.toml says it’s on. veltro backup --unschedule turns it off. An install that started the sidecar by hand with docker compose --profile backup up -d veltro-backup has nothing recorded, so run veltro backup --schedule once to keep it across restarts.

Before an upgrade replaces the installed files, Veltro runs the verified new backup script against the current datastores. It uses the staged Compose definition with the existing Compose project name, so the archive still has the installed version pins and selection. An install still on the per-product datastores can’t upgrade straight to this release; it goes through suite 0.13.9 first (docs/runbooks/datastore-consolidation.md).

Encrypt the whole Suite archive

The identity secret envelope protects only the keyring. To protect database dumps, .env and the archive manifest, configure:

VELTRO_BACKUP_ARCHIVE_ENCRYPTION=true
VELTRO_BACKUP_ARCHIVE_KEY_ID=suite-2026-07
VELTRO_BACKUP_ARCHIVE_PASSWORD_FILE=/srv/veltro/secrets/backup/password
VELTRO_BACKUP_ARCHIVE_SECRET_DIR=/srv/veltro/secrets/backup
VELTRO_BACKUP_ARCHIVE_KEYRING_FILE=/srv/veltro/secrets/backup/keyring.json

Use an operator-owned mode-0600 password file with a randomly generated high-entropy password. The scheduled sidecar mounts the directory read-only at /run/veltro-backup; its password file is named password. It produces .tar.enc files. A manual backup can use veltro backup --encrypt --to backup.tar.enc. Encryption authenticates the entire file with AES-256-CBC, PBKDF2-SHA256 (600,000 iterations), and HMAC-SHA256. Passwords travel through files/process environment, never command arguments. Restore checks authentication before extracting or changing a datastore.

The unencrypted header records only format, scope and key ID. Retain old passwords while any archive needs them. keyring.json maps historical key IDs to password-file paths readable in the process doing the restore or retention check. Keep an independent escrow copy of these keys. Rotation changes the current key ID and password; it does not rewrite old archives. Plain .tar archives remain supported.

Restore

Irreversible-migration rule (Sentry pattern): restore a backup whose module pins are older than or equal to the installed pins, then upgrade the suite. Never restore a newer major/module pin onto older software.

Restore refuses when:

1. any artifact digest or size does not match manifest.json;
2. the archive uses the per-product datastore layout (no datastore_layout), or it holds a datastore artifact this release doesn’t know;
3. backup suite major > installed suite major;
4. any of VF_VERSION / SIGMATIC_VERSION / WARDEN_VERSION / VELTRO_VERSION in the backup is newer than the installed pin;
5. identity evidence (when present) violates the recovery contract in conformance/identity-persistence-v1.json:
   • realm must be the fixed literal default;
   • postgres major must be one the suite restores (postgres_majors_supported: 16 or 18), so a PG16 backup restores into PG18;
   • backup migration head must not be ahead of the installed contract head (schema regression → restore-then-upgrade the suite image first);
   • recovery remains repair-forward (no automatic down migrations).

# Plan only
./scripts/veltro-restore.sh ./veltro-backup.tar --dry-run

# Destructive apply (stops app services, rewrites datastores, starts the stack)
export VELTRO_BACKUP_PASSPHRASE='…'
./scripts/veltro-restore.sh ./veltro-backup.tar --yes

# Or write the data and leave the stack stopped, then start it yourself
./scripts/veltro-restore.sh ./veltro-backup.tar --yes --skip-start
veltro up   # or: make up

Restore starts the stack the way veltro up does, in one compose up. Products fetch identity’s public keys from its JWKS (#1692), so they verify request assertions against the ring the restore brought back with no extra step. With --skip-start, veltro up does the same. A stack that doesn’t start fails the restore after the data is written, and says so: the error says the data is back and gives the veltro up --dir … --products … command that finishes the job. Run that rather than restoring again.

Restore selection follows the same order: VELTRO_PRODUCTS in .env, saved veltro.toml, then all products. --products overrides these for one run. Select the products whose data is in the archive. Restore applies the artifacts it finds. It doesn’t migrate data between products, enable missing products, or downgrade schemas. Core services and selected product apps are stopped first. Apps of products you didn’t select keep running, and if one of them writes to a Redis index that’s being restored, the DBSIZE check fails.

After restore:

1. Confirm docker compose ps / veltro status show healthy services.
2. Spot-check identity login and the installed products’ data, such as a Sigmatic alert query, Warden case, or VectorFlow pipeline list.
3. If identity evidence was present, optionally re-run the identity recovery verification path in docs/runbooks/identity-database-recovery.md (veltro.recovery_evidence(), head/checksum/fingerprint).
4. If .env sha256 warned on mismatch, re-seal product secrets or restore the matching env before expecting encrypted fields to decrypt. An archive from before #1692 can differ only by the assertion block older releases wrote into .env; Compose ignores those lines now.

Restore onto a wiped host

If the volumes are gone (veltro down -v, a new host, a lost disk), run the same command. You don’t need to start anything first:

export VELTRO_BACKUP_PASSPHRASE='…'
veltro restore ./veltro-backup.tar --yes

Restore unseals the secrets envelope first, so a wrong passphrase fails before anything starts. Then it starts only the datastores the archive restores into (suite-postgres, suite-redis, and opensearch when the archive has a Sigmatic snapshot), using the install’s profiles, and waits until they’re healthy. It also waits until suite-postgres accepts TCP connections; on first boot the image runs a socket-only server while it initialises. If a datastore doesn’t come up within five minutes, restore stops and says nothing was restored. Next it creates the named volumes the archive restores into that don’t exist yet (vf-data, sigmatic-data, warden-data, suite-workload-secrets, identity-keyring). It creates the containers that mount them without starting them, so Compose makes each volume under its own name. Once the data and the keyring volume are back, it starts the rest of the stack as above.

Running veltro up first on a wiped host costs nothing now: identity-init makes a fresh keyring for the empty database, and the restore replaces it with the archived one along with the data.

How the shared datastores are restored

Before anything is stopped, restore creates the carried roles (from postgres/suite-postgres-roles.json) that suite-postgres doesn’t have yet, with their attributes, settings, memberships and password verifiers, then checks that every role any archived database references exists. A carried role that already exists with different attributes, a different verifier, settings or memberships is refused, not overwritten. Restore also refuses, already in --dry-run, an archive whose databases don’t record their properties and referenced roles; take a new backup instead.

Postgres goes one database at a time, veltro_identity, then vectorflow, sigmatic and warden, all as suite_admin:

1. The dump is streamed into suite-postgres as a mode 0600 file in a private directory under /tmp, and pg_restore --list reads its table of contents. A dump that can’t be read stops the restore here, before anything is dropped.
2. DROP DATABASE … WITH (FORCE), then CREATE DATABASE … OWNER <owner> TEMPLATE template0, using the owner role that compose/suite-postgres/init.sh created.
3. veltro_identity gets its database ACL back: REVOKE ALL … FROM PUBLIC, then GRANT CONNECT to veltro_identity_owner, veltro_identity_runtime, veltro_identity_importer and veltro_identity_recovery.
4. vectorflow gets CREATE EXTENSION timescaledb and SELECT timescaledb_pre_restore(). Its restore list drops the dump’s own timescaledb extension and extension comment entries.
5. pg_restore --exit-on-error --use-list … replays the dump. It runs without --no-owner, so schemas, tables and grants keep the owners recorded in the dump. For vectorflow, SELECT timescaledb_post_restore() runs afterwards even when pg_restore fails.
6. The database gets back what pg_dump left out, exactly as the manifest recorded it: its ACL (everything else revoked first), connection limit, database settings and role-in-database settings. Identity keeps its PUBLIC revoke and CONNECT grants on top. The result is read back and compared; a difference fails the restore.
7. The temporary directory is removed.

Volumes are found by the name the Compose model gives them; restore creates the missing ones first, as above. Each volume comes back with the owners and modes it had at backup time, since the helper container extracts the archive as root. The OpenSearch restore deletes the snapshot’s own sigmatic-* indices before restoring them, since OpenSearch won’t restore over an open index. A snapshot with no index, from an install where Sigmatic hasn’t written one yet, has nothing to restore and is skipped. The identity keyring comes back the same way as the other volumes, into identity-keyring, with the owner (UID and GID 1000, the node user identity runs as) and mode 0600 it had. An archive from before #1690 holds the keyring file itself; restore wraps it in a volume archive owned by 1000:1000 at 0600 and restores that. Nothing is written on the host, so the restore no longer has to run as root or UID 1000. If the install still has a host keyring file from before #1690 and it isn’t the one restored, the start afterwards stops and says so; see “Moving the TOTP keyring into its volume” in identity-signing-key-provisioning.

Redis goes one index at a time and suite-redis keeps running. The helper container reads the whole artifact and checks its framing, then runs FLUSHDB SYNC on the index and RESTORE key ttl payload REPLACE for each record. Those writes go through the live server, so the append-only file records them and the data survives a restart of suite-redis. Keys without a TTL stay persistent. A key that had a TTL gets the time it had left when the backup ran, counted from the restore. Afterwards DBSIZE has to equal the artifact’s record count. The only slack is for keys whose restored TTL ran out before DBSIZE answered; any other difference fails the restore. The restore output lists each index’s numbers under redis_verification. Indexes that aren’t in the archive are left alone.

Managed Lake backup and recovery have been removed. Restore refuses the whole archive if it contains lake.clickhouse, lake.lifecycle, a retired Lake artifact kind, or Lake archive members, even if the manifest doesn’t list those members. This check runs before unsealing identity secrets or changing services, volumes, configuration or datastores. It also applies to --postgres-only; there is no skip flag. Take a new backup without Lake content.

External ClickHouse sinks remain supported, but their data is outside the suite backup. Back up and restore that external service separately.

The PostgreSQL major-version copy inside veltro upgrade keeps schemas and migration ledgers unchanged until row-count and inventory comparison passes. The datastore-upgrade marker blocks writers throughout the copy.

Archives from the per-product datastores

Archives written before ADR-014 step 5 have no datastore_layout in manifest.json. They hold one dump per product PostgreSQL server plus RDB files from Sigmatic’s and Warden’s Redis; VectorFlow’s Redis was never in them. This release refuses them, in --dry-run and in a real restore, before it stops a service or touches a datastore. The refusal says where the archive came from and what to do with it. There’s no partial restore of such an archive.

Keep those archives until your retention decision covers them. To restore one, use scripts/veltro_backup.py from the release that wrote it, against a stack that still runs the per-product datastores. Moving that stack onto the shared datastores takes the migration in suite 0.13.9; see docs/runbooks/datastore-consolidation.md.

Empty-database / identity-only recovery

When you only need the independent identity database, use the dedicated identity recovery tool and runbook — not this suite archive:

• docs/runbooks/identity-database-recovery.md
• apps/identity/recovery-tool + veltro-identity-recovery-manifest/v1
• scripts/recovery_set.py / scripts/restore_plan.py for the narrower recovery-set planner

The suite backup complements that path: it adds product DBs, OpenSearch, Redis, and volumes under one digest-bound tar.

Restore test

Prove a restore on a disposable stack, never on production: a restore drops and recreates every database in the archive and flushes its Redis indexes.

1. Bring up the disposable stack at the installed release and seed representative data for every product you run.
2. Take a backup with scripts/veltro_backup.py backup. The manifest should show datastore_layout: suite, a dump for each selected database and a redis/suite-redis-db<N>.bin for each selected index.
3. Change a few records, so you can see the restore undo them.
4. Run restore --dry-run and check nothing blocks, then restore --yes.
5. Check the output: status is restored, and every index under redis_verification has a dbsize equal to that artifact’s keys in the manifest. Then check identity sign-in and each product’s readiness, the veltro_identity ACL (\l veltro_identity in psql as suite_admin), each database’s grants, connection limit and settings, table owners, VectorFlow’s hypertables, and that your changes from step 3 are gone.

Config-only vs bulk DR

Need	Tool
Move a VF pipeline graph or Sigmatic rule pack between environments	Product export/import UI/API
Rebuild a host after disk loss	This suite backup + veltro install/upgrade
Identity-only recovery evidence	Identity recovery runbook
Homelab pin rollback to a prior suite release	promote_homelab.py recovery mode (immutable manifest), not a data backup

Support bundle

For live incident diagnosis without datastores, use the secret-free support bundle (python3 scripts/support_bundle.py). It is not a restore input.

Acceptance (CI)

The dispatch-only dr job in verify.yml (when enabled) is the live proof:

1. seed demo on a disposable stack;
2. veltro backup (or scripts/veltro_backup.py backup);
3. wipe volumes;
4. veltro restore --yes;
5. golden-path spec green on restored data.

Unit coverage for digests, version refuse rules, identity evidence gates, the secret envelope, the suite-layout Postgres and Redis restore sequence, the Redis helper’s framing and RESP handling, and the old-layout refusal lives in scripts/test_veltro_backup.py (injected runners; no Docker on the developer laptop).