Backup and upgrade

Backup

veltro backup is the one backup for an install: the sign-in service and every installed product go into the same archive. The same library runs without the CLI package, which is what the Make targets and the POSIX shims use:

veltro backup --to ./veltro-backup-$(date +%Y%m%d).tar
# without the CLI package installed:
python3 scripts/veltro_backup.py backup --to ./veltro-backup-$(date +%Y%m%d).tar
# POSIX shim: scripts/veltro-backup.sh --to ./…

Artifact veltro-backup/v1 (datastore layout suite) contains:

• One pg_dump -Fc per suite-postgres database, taken as suite_admin: veltro_identity always, vectorflow, sigmatic and warden when selected
• One key dump per suite-redis index: 0 VectorFlow, 1 Sigmatic cache, 2 Sigmatic queue, 3 Warden (only selected products)
• OpenSearch snapshot of sigmatic-* indices
• Volume archives (vf-data, sigmatic-data, warden-data, workload secrets)
• The encrypted identity secret envelope and versions.env
• manifest.json with per-artifact sha256, module pins, backup_status, datastore_layout and the database and Redis index lists

.env is excluded unless --include-env (prints a warning). Components are captured sequentially, so the manifest records consistency.is_coordinated_snapshot: false — it is not a distributed snapshot. Scheduled backups use compose profile backup.

Use --encrypt or VELTRO_BACKUP_ARCHIVE_ENCRYPTION=true to encrypt the whole Suite archive, including database dumps and its manifest. The archive password and versioned key ID are separate from the identity secret-envelope password. Keep historical mounted keys and an independent escrow copy while any archive needs them.

Full procedure, including partial-backup semantics: backup and restore runbook.

Restore

veltro restore ./veltro-backup-….tar
# without the CLI package installed:
python3 scripts/veltro_backup.py restore ./veltro-backup-….tar

Refuses: backup suite major newer than installed; module versions newer than installed (restore, then upgrade).

Each database is dropped and recreated with its owner role, then restored with pg_restore as suite_admin, keeping the owners in the dump. The identity database gets its connect ACL back, and vectorflow is restored between timescaledb_pre_restore() and timescaledb_post_restore(). Each Redis index is flushed and refilled with RESTORE through the running suite-redis, so its append-only file records the writes, and DBSIZE must match the artifact’s key count.

Managed Lake backup and recovery have been removed. Archives containing lake.clickhouse, lake.lifecycle or other retired Lake artifacts are refused in full before any restore effects, including with --postgres-only. Take a new backup without Lake content. There is no flag to skip those artifacts.

External ClickHouse sink data isn’t part of the suite archive. Back it up and restore it separately.

Caution

Archives taken before the move to the shared datastores (no datastore_layout in the manifest) are refused before anything changes. Restore them with the release that took them into a stack that still runs the per-product datastores. An install on those datastores upgrades through suite 0.13.9 first; see the datastore consolidation runbook.

Upgrade

Update the CLI first. The upgrade runs the CLI you already have, not the one in the release, so it can’t pick up fixes the new release makes to the upgrade itself. veltro upgrade refuses to start when the CLI is older than the release you name, and prints the command to update it.

~/.veltro/venv/bin/python3 -m pip install -U veltro-cli==<release>
# or, if you installed it with pipx:
pipx install --force veltro-cli==<release>

veltro upgrade --to <release>          # --dry-run prints the plan

Use the first line if install.sh set up the CLI. Otherwise use the pip of the environment veltro lives in. A CLI installed from a clone reports a placeholder version, so the upgrade always refuses it. Install the published veltro-cli==<release> to upgrade.

Upgrading from 0.16.x

The 0.16.x CLI doesn’t have this check. Run veltro upgrade --to 0.17.0 with it and it replaces compose/ whole, which deletes compose/secrets, and identity won’t start. Update the CLI to 0.17.0 before you upgrade. If you can’t, take a veltro backup first so you have something to restore.

veltro upgrade fetches the release manifest, refuses downgrades, prints the release’s breaking-change and migration notes, takes a backup, replaces the version pins in versions.env, starts the new release, and checks that every running Veltro container runs the image digest the release signs. An enrolled VectorFlow agent is restarted on the new release before that check. Don’t edit those pins by hand.

It also checks that identity’s install id, written once on the first install, is the same after the upgrade. A different one means the suite came up on a freshly initialised datastore, and the upgrade stops before recording the new version.

Remove managed Lake deployments first

Before stopping services or replacing release files, the upgrade reads VectorFlow’s deployed and pending graphs from PostgreSQL. If any still contain vectorflow_lake, including disabled nodes, it refuses and lists the pipeline names and IDs. Pending deploy requests, promotions and active canaries count, as do versions still assigned to an agent.

Use the installed release to remove those nodes and their edges. Keep every surviving destination wired as it was, then deploy through the normal approval flow. Cancel pending requests or canaries that contain Lake. Undeploy a pipeline whose only destination was Lake until you’ve added a surviving destination. Then retry the upgrade. Generic external clickhouse sinks don’t need changing.

PostgreSQL 16 to 18

Releases after 0.26.x run PostgreSQL 18, which won’t open the data PostgreSQL 16 wrote. Upgrading from 0.26.x or earlier, veltro upgrade moves it: it stops every service but the datastores, takes the backup, starts PostgreSQL 18 on a new volume, restores into it and refuses unless every table’s row count and every migration ledger match. The old volume, veltro-suite-pgdata, isn’t touched, and veltro upgrade --rollback goes back to it. Expect downtime of a few minutes per few GB of PostgreSQL data. The PostgreSQL 18 upgrade runbook has the details, the manual steps and what to remove once you’re satisfied.

Bundled Let’s Encrypt TLS was removed

Releases up to 0.20.x could run Caddy with --tls auto, which got its own certificate from Let’s Encrypt. That mode is gone. Running behind your own reverse proxy is the recommended setup, and the bundled Caddy now only serves a certificate you supply (custom) or one from its own CA (internal).

On an install made with --tls auto, veltro upgrade and every other command that reads veltro.toml stop with an error and change nothing. They won’t start the install without TLS. Move the install to one of these first, then run veltro upgrade again:

• Your own proxy, the recommended route. Set it up as TLS and your reverse proxy shows. The gateway’s address is VELTRO_HTTP_PORT in .env: the auto install put it on loopback (127.0.0.1:8088 unless you’d set a port), so change it if the proxy runs on another host. With Sigmatic installed, the old Caddy also served log ingest on port 8443, so point your shippers at /api/logs/ on the public URL first, or have your proxy take over 8443 once the old Caddy is gone. Then set tls_mode = "off" in veltro.toml and remove the old Caddy so it frees ports 80, 443 and 8443: docker rm -f veltro-tls.
• Your own certificate in the bundled Caddy. Copy the certificate chain and key to tls/fullchain.pem and tls/privkey.pem and set tls_mode = "custom" in veltro.toml. To keep serving the certificate Caddy already has while you sort out renewal, copy <host>.crt and <host>.key from tls/caddy-data/caddy/certificates/<ca>/<host>/ (owned by root; for Let’s Encrypt <ca> is acme-v02.api.letsencrypt.org-directory). Nothing renews it, so it expires within 90 days.