Kubernetes and Helm

Source of truth: docs/kubernetes.md (synced at build time — edit the original, not this copy). Veltro publishes one Helm chart to install, veltro, as an OCI artifact at oci://ghcr.io/veltrosecurity/charts/veltro. An OCI artifact is a versioned package in a container registry. The chart brings the gateway, the sign-in service and the products you enable, at the versions its release pins.

Prerequisites

Before you install

• Helm 3 or later, and a namespace for the suite:

  kubectl create namespace veltro

• A default StorageClass. The charts create their volume claims from it. kubectl get storageclass should list one class marked (default). ReadWriteOnce is enough for VectorFlow, Sigmatic and Warden, including single-attach block storage (EBS, GCE PD, Azure Disk, Ceph RBD) on several nodes. Where pods share a claim, the charts keep them on the node it’s attached to. VectorFlow and Sigmatic also stop a pod before starting its replacement, so their upgrades have a short gap. VectorFlow chart 0.26.7 and Sigmatic chart 0.52.0 and earlier don’t do this (veltro#1402, veltro#1403): run them on one node, local-path or NFS. Look for vectorflow-server and sigmatic under dependencies in the command in the next item. Older Warden charts are the exception (next item).

• ReadWriteMany (RWX) storage for Warden, if your suite version pins Warden chart 0.18.8 or earlier. Suite 0.18.0 pins 0.18.8. To see which Warden chart a suite version pins, look for warden under dependencies in:

  helm show chart oci://ghcr.io/veltrosecurity/charts/veltro --version <suite-version>

  Warden’s api and worker both mount one data claim, and those charts refuse anything but RWX. Set warden.persistence.data.storageClass to an RWX class, or create an RWX claim yourself and set warden.persistence.data.existingClaim to its name. NFS (nfs-subdir-external-provisioner), Longhorn, Amazon EFS and Azure Files can all provide RWX.

  A single-node lab cluster such as kind has no RWX class, but a hostPath volume works there because every pod runs on that one node. Make the directory on the node, owned by Warden’s user (1000), then apply the volume and claim below with kubectl apply -f and set existingClaim: warden-data:

  # on the node, e.g. docker exec <kind-node> sh -c '...'
  mkdir -p /var/local/warden-data && chown 1000:1000 /var/local/warden-data

  apiVersion: v1
  kind: PersistentVolume
  metadata: {name: veltro-warden-data}
  spec:
    capacity: {storage: 10Gi}
    accessModes: [ReadWriteMany]
    storageClassName: ""
    hostPath: {path: /var/local/warden-data}
    claimRef: {namespace: veltro, name: warden-data}
  ---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata: {name: warden-data, namespace: veltro}
  spec:
    accessModes: [ReadWriteMany]
    storageClassName: ""
    volumeName: veltro-warden-data
    resources: {requests: {storage: 10Gi}}

  For a claim you made, this should print ["ReadWriteMany"]:

  kubectl -n veltro get pvc <claim> -o jsonpath='{.spec.accessModes}'

  Later Warden charts use ReadWriteOnce and keep every api and worker pod on one node (warden#285), so they need nothing extra. To spread those pods across nodes, set warden.persistence.data.accessMode to ReadWriteMany and give it an RWX class. A claim that already exists keeps its access modes, and moving it to RWX takes a new claim. If you render with veltro k8s render or helm template (for Argo CD, say), the chart can’t see the claim: set warden.persistence.data.accessMode: ReadWriteMany when an older chart made it.

• Registry access, while the charts and images are private. You won’t need this once they’re public (#925). Until then, use a GitHub token with read:packages to log Helm in and to make a pull secret for the cluster:

  helm registry login ghcr.io
  kubectl -n veltro create secret docker-registry ghcr \
    --docker-server=ghcr.io \
    --docker-username=<github-user> \
    --docker-password=<token>

  Add it to every chart that pulls a Veltro image, in the values file from Install the suite. Set each chart’s own key, because the umbrella’s global.imagePullSecrets doesn’t reach them. The gateway runs the public nginx image and doesn’t need it.

  veltro-identity:
    imagePullSecrets: [{name: ghcr}]
  vectorflow-server:
    imagePullSecrets: [{name: ghcr}]
  sigmatic:
    imagePullSecrets: [{name: ghcr}]
  warden:
    imagePullSecrets: [{name: ghcr}]

Secrets

Create one Kubernetes Secret for each product. Do not use one shared Secret for the product database keys. The product charts define their required keys in these files:

• vectorflow/charts/vectorflow-server/values.yaml
• sigmatic/charts/sigmatic/values.yaml
• warden/charts/warden/values.yaml
• veltro/charts/veltro-identity/values.yaml

The default umbrella names are vectorflow-secrets, sigmatic-secrets, warden-secrets, and veltro-suite-secrets.

vectorflow-secrets needs NEXTAUTH_SECRET, DATABASE_URL and VF_ENCRYPTION_KEY_V2, the key VectorFlow encrypts stored credentials with. VectorFlow doesn’t start without it. For a new install, generate one with openssl rand -base64 32 and keep it with your backups. An install whose data was encrypted before it had this key uses its current NEXTAUTH_SECRET as the value. REDIS_URL is optional. VectorFlow charts before vectorflow#837 list the encryption key as optional, but the image still requires it (#1387).

sigmatic-secrets needs SIGMATIC_ENCRYPTION_KEY (at least 32 characters) and your database’s POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_DB and POSTGRES_PASSWORD. Without POSTGRES_HOST, Sigmatic connects to localhost. Sigmatic charts 0.52.1 and earlier read those keys but leave four of them out of their values.yaml list (#1405). In its default push mode Sigmatic also needs REDIS_URL; REDIS_QUEUE_URL is optional and defaults to it. OPENSEARCH_SEED_URL tells Sigmatic where OpenSearch is on first boot. Add OPENSEARCH_SEED_USERNAME and OPENSEARCH_SEED_PASSWORD if it needs a login. Until Sigmatic has an OpenSearch connection, its worker exits and restarts.

kubectl -n veltro create secret generic sigmatic-secrets \
  --from-literal=SIGMATIC_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  --from-literal=POSTGRES_HOST=<host> \
  --from-literal=POSTGRES_PORT=5432 \
  --from-literal=POSTGRES_USER=sigmatic \
  --from-literal=POSTGRES_DB=sigmatic \
  --from-literal=POSTGRES_PASSWORD=<password> \
  --from-literal=REDIS_URL=redis://<host>:6379/0 \
  --from-literal=OPENSEARCH_SEED_URL=https://<host>:9200

You don’t copy identity’s public signing keys or any workload secret anywhere. Every product fetches identity’s public keys from its JWKS, http://<release>-veltro-identity:3000/.well-known/jwks.json, which you set per product in the values below. The chart refuses any other URL, since products trust whatever keys it serves. A signing-key rotation needs no product change: retired keys stay in the JWKS and products pick up the new one on their own.

Once identity’s database exists, a Job in the identity chart writes SUITE_AUTHORITY_V2_ACTIVE_GENERATION and the product’s own SUITE_WORKLOAD_SECRET into each installed product’s Secret. Sigmatic gets its workload secret only with veltro-identity.hooks.provisionSigmaticWorkload.enabled. Product pods wait until those keys are there, then start. No volume is shared between pods, so ReadWriteOnce storage on any number of nodes is fine for these. The Job runs again on every helm upgrade. The Job can patch only the Secrets named in veltro-identity.hooks.publishSuiteAuthority.secrets, and each name has to match that product’s existingSecret. If something else manages those Secrets and would undo the patch, set veltro-identity.hooks.publishSuiteAuthority.enabled=false and add the keys yourself. That includes SUITE_WORKLOAD_SECRET for VectorFlow and Warden, which nothing else writes. The identity image’s apps/identity/signing-tool/boot-workload-provision.cjs writes one product’s secret to a file: give it SUITE_WORKLOAD_CLIENT_ID (the product name), SUITE_WORKLOAD_SECRET_FILE, VELTRO_IDENTITY_MIGRATION_DATABASE_URL and VELTRO_IDENTITY_SIGNING_PASSPHRASE. It mints the secret the first time and returns the same value after that.

Only product charts with suite.authority.jwksUrl (VectorFlow, Sigmatic) or config.suiteAuthorityV2JwksUrl (Warden) read the URL. Older ones wait for a public-key entry in their Secret that the Job no longer writes, so a suite release has to pin charts that read it.

Warden’s Secret still needs WARDEN_SECRET_KEY. Sigmatic and Warden need the active issuer and positive generation in the values below (a fresh install starts at generation 1), and Warden needs identity’s Service URL for the workload token exchange. VectorFlow needs that URL too: it asks identity for the sidebar and header of every page, and without it, it shows an empty chrome.

TOTP keyring

Identity seals every enrolled authenticator secret with its TOTP keyring and doesn’t get ready without one. On first install the chart creates the Secret <release>-veltro-identity-totp-keyring with a random key. Upgrades keep the stored keyring, and helm uninstall leaves the Secret behind, because losing it locks out every user enrolled in TOTP. Back it up with identity’s database.

The chart checks for an existing keyring with Helm’s lookup, which sees nothing when manifests come from helm template (Argo CD, veltro k8s render). There, every render would mint a new key. Create the Secret yourself first and point the chart at it:

python3 -c 'import base64,json,secrets; print(json.dumps({"version":1,"activeKeyId":"identity-totp-2026-01","keys":[{"id":"identity-totp-2026-01","status":"active","keyBase64":base64.b64encode(secrets.token_bytes(32)).decode()}]}))' > totp-keyring.json
kubectl -n veltro create secret generic veltro-identity-totp-keyring --from-file=totp-keyring.json
rm totp-keyring.json

veltro-identity:
  totpKeyring:
    existingSecret: veltro-identity-totp-keyring

Install the suite

Create a values file for the public URL and your Secret names. While the packages are private, add the imagePullSecrets lines from Before you install to it, and on Warden charts 0.18.8 and earlier your RWX storageClass or existingClaim for Warden.

global:
  publicBaseUrl: https://soc.example.com

vectorflow-server:
  existingSecret: vectorflow-secrets
  suite:
    # identity's Service in a release named `veltro`
    exchangeUrl: http://veltro-veltro-identity:3000
    authority:
      jwksUrl: http://veltro-veltro-identity:3000/.well-known/jwks.json

sigmatic:
  existingSecret: sigmatic-secrets
  appUrl: https://soc.example.com/sigmatic
  suite:
    authority:
      issuer: https://soc.example.com
      activeGeneration: 1
      jwksUrl: http://veltro-veltro-identity:3000/.well-known/jwks.json

warden:
  existingSecret: warden-secrets
  config:
    suiteAuthorityV2Issuer: https://soc.example.com
    suiteAuthorityV2ActiveGeneration: 1
    suiteAuthorityV2JwksUrl: http://veltro-veltro-identity:3000/.well-known/jwks.json
    # identity's Service in a release named `veltro`
    suiteExchangeUrl: http://veltro-veltro-identity:3000

veltro-identity:
  existingSecret: veltro-suite-secrets
  # Your first admin. Identity logs their one-time sign-in link at startup.
  bootstrapAdminEmail: you@example.com

After the first start, kubectl -n veltro logs deploy/<release>-veltro-identity-server | grep 'First admin' shows the first admin’s one-time link.

Install one exact suite version.

helm install veltro oci://ghcr.io/veltrosecurity/charts/veltro \
  --version <suite-version> \
  --namespace veltro \
  --values values.yaml \
  --wait --wait-for-jobs

With Helm 3 on suite 0.19.1, --wait can return before VectorFlow’s server and Sigmatic’s backend and worker are ready, on install and on upgrade (#1406). Helm 4 waits for them, and so do later suites. On 0.19.1 with Helm 3, check them yourself:

kubectl -n veltro rollout status deployment/veltro-vectorflow-server
kubectl -n veltro rollout status deployment/veltro-chad-backend
kubectl -n veltro rollout status deployment/veltro-chad-worker

The umbrella uses these path prefixes:

• /vectorflow
• /sigmatic
• /warden

The gateway keeps the browser-request assertion boundary on every product path. Do not expose product services as an alternative suite entry point.

Release names

Helm names a chart’s resources <release>-<chart>, except when the release name already contains the chart name — then it uses the release name alone. A release called sigmatic-prod therefore gives Sigmatic the Service sigmatic-prod-backend, not sigmatic-prod-sigmatic-backend.

The umbrella refuses a release name containing more than one enabled chart name (including veltro): those charts can collapse onto the same resource name. With Sigmatic enabled, its untruncated -frontend suffix limits names without sigmatic to 49 characters; names containing sigmatic can use Helm’s 53-character limit. The suite gate renders valid release names and checks that colliding or overlong names fail.

The gateway names every upstream as <service>.<namespace>.svc.<cluster domain>. nginx resolves these names itself and ignores the pod’s DNS search domains, so a bare Service name never resolves. If your cluster’s DNS domain isn’t cluster.local, set veltro-gateway.clusterDomain.

If you point a route at a service outside this release, set the gateway upstream yourself, with the full name:

veltro-gateway:
  upstreams:
    sigmaticBackend: sigmatic-backend.other-namespace.svc.cluster.local:8000

Datastores

Production installations use external Postgres and Redis services. Sigmatic also requires OpenSearch.

Each product chart can install its bundled Postgres or Redis dependency for a disposable lab. Read the product values.yaml file before you enable a bundled datastore.

Identity’s Secret (veltro-suite-secrets) always needs VELTRO_IDENTITY_SESSION_SECRET and VELTRO_IDENTITY_SIGNING_PASSPHRASE. What else it needs depends on where the database is.

With external Postgres, add both DSNs. URL-encode any reserved character in a password (@, :, /, ?, #, %):

• VELTRO_IDENTITY_MIGRATION_DATABASE_URL: postgresql://veltro_identity_owner:<password>@<host>:5432/veltro_identity
• VELTRO_IDENTITY_RUNTIME_DATABASE_URL: postgresql://veltro_identity_runtime:<password>@<host>:5432/veltro_identity

The identity chart also includes a small Postgres StatefulSet for lab installations. Set veltro-identity.postgresql.enabled=true and put the four role passwords in the Secret: VELTRO_IDENTITY_OWNER_PASSWORD, VELTRO_IDENTITY_RUNTIME_PASSWORD, VELTRO_IDENTITY_IMPORTER_PASSWORD and VELTRO_IDENTITY_RECOVERY_PASSWORD. The chart builds both DSNs from those itself and ignores any DSN keys in the Secret. The passwords don’t need URL encoding. It runs PostgreSQL 18; an install still on 16 moves its data as Bundled Postgres 18 describes.

Migrations

Sigmatic, Warden, and identity create migration Jobs. Helm waits for these Jobs when you use --wait-for-jobs.

VectorFlow runs Prisma migrations during startup by default. Its optional migration Job requires an external database Secret.

Monitoring

Turn on monitoring when your cluster has the Prometheus Operator and Grafana dashboard sidecar.

veltro-monitoring:
  enabled: true

The chart installs four Grafana dashboards and Veltro alert rules. The PrometheusRule resource stays disabled only when you set veltro-monitoring.prometheusRule.enabled=false.

Each product chart also supports an optional ServiceMonitor resource.

Render manifests

Use the Veltro CLI for a local render.

veltro k8s render --values values.yaml

Gateway and identity always render. With no --products, the CLI uses VELTRO_PRODUCTS from .env, then the selection saved in veltro.toml, then all three products, the same order as Compose. Name a subset to render only those products; an empty list renders gateway and identity alone (ADR-014):

veltro k8s render --products sigmatic,warden --values values.yaml

The CLI sets each product’s enabled flag and global.products, which identity and the product charts read to know which peers exist. With plain Helm, set both to the same selection or the chart refuses to render. Use --set-json 'global.products=["sigmatic","warden"]' for a subset, or --set-json 'global.products=[]' for core alone, and disable the other product subcharts. Gateway and identity cannot be disabled.

You can also use Helm directly.

helm template veltro oci://ghcr.io/veltrosecurity/charts/veltro \
  --version <suite-version> \
  --namespace veltro \
  --values values.yaml

Upgrade

Read the release notes for migration warnings. Then upgrade to one exact version.

helm upgrade veltro oci://ghcr.io/veltrosecurity/charts/veltro \
  --version <suite-version> \
  --namespace veltro \
  --values values.yaml \
  --wait --wait-for-jobs

With Helm 4, going from suite 0.19.0 or earlier straight to 0.19.2 or later can fail with spec.strategy.rollingUpdate: Forbidden. On ReadWriteOnce storage, VectorFlow and Sigmatic now roll out with Recreate, and Helm 4’s server-side apply can’t remove the default rollout settings the older Deployments carry. Run the same upgrade again with --server-side=false; later upgrades don’t need it. Helm 3, and Helm 4 upgrades that went through 0.19.1, don’t hit this. Under Helm 4, the same error and fix apply when an upgrade first puts VectorFlow or Sigmatic on a ReadWriteOnce claim, for example by turning on persistence.data.enabled or pointing persistence.data.existingClaim at an RWO claim.

Suites that pin VectorFlow chart 0.26.6 or earlier gave VectorFlow its workload secret through a claim, <release>-veltro-identity-workload-secrets. Later suites don’t create it: the upgrade deletes the claim once VectorFlow stops mounting it, and identity writes the same secret into vectorflow-secrets instead. With publishing on (the default) there’s nothing to copy, since identity keeps the secret in its database. With it off, add VectorFlow’s SUITE_WORKLOAD_SECRET before upgrading (see Secrets), or the VectorFlow pod waits for it. A values file that still sets vectorflow-server.suite.workloadSecrets.existingClaim or veltro-identity.workloadSecrets works unchanged; the charts ignore both. Unset sigmatic.suite.workloadSecrets.existingClaim if you set it: the chart refuses it, and veltro-identity.hooks.provisionSigmaticWorkload.enabled puts Sigmatic’s secret in sigmatic-secrets instead.

The suite manifest records the immutable digest of every chart the release installs. Promotion fails if a registry tag moves to a different digest.

The same manifest also records the keyless Cosign signature behind each of those charts, next to the signature of each first-party image. The chart set that must be signed is read from the manifest’s own chart pins, so a chart cannot be pinned without evidence, and evidence cannot be recorded for a chart the manifest does not pin. Each signature must come from the one reviewed workflow that publishes that chart: the product charts from their own repository’s release workflow, veltro-identity from the protected identity-v* component release, and veltro, veltro-gateway and veltro-monitoring from the suite release train. suite_manifest.py verify re-checks the recorded signature with cosign verify unless you pass --offline.

Bundled Postgres 18

This applies only if you run identity’s bundled Postgres (veltro-identity.postgresql.enabled=true). The identity chart now runs postgres:18.6-alpine. PostgreSQL 18 can’t open a PostgreSQL 16 data directory, so the chart runs it as a new StatefulSet, <release>-postgresql-18, on a new claim, pgdata-<release>-postgresql-18-0. The Service keeps its name, so identity’s connection settings don’t change.

A plain helm upgrade would start identity on that new, empty database, so the chart refuses it and names the StatefulSet it would replace. Move the data with identity_postgres_upgrade.py from the release you’re upgrading to. It needs kubectl, helm and Python 3. Without a checkout, download it from that release’s tag:

curl -fsSLO https://raw.githubusercontent.com/veltrosecurity/veltro/v<suite-version>/scripts/identity_postgres_upgrade.py

It takes the chart and the arguments you’d give helm upgrade, after --. Flags only helm upgrade has, such as --install or --history-max, go to the upgrades and are left out of the render the script checks first:

python3 identity_postgres_upgrade.py \
  --release veltro --namespace veltro \
  --chart oci://ghcr.io/veltrosecurity/charts/veltro -- \
  --version <suite-version> \
  --values values.yaml

Add --dry-run first to see the plan without changing anything. The script:

1. Checks that the major changes, that the new claim doesn’t exist yet, and that there’s a bundled Postgres to move.
2. Reads identity’s install id from /api/suite/peer-reads, if your identity reports one.
3. Stops identity and dumps the old Postgres (roles and the veltro_identity database) into a new local directory, identity-postgres-upgrade-<time> unless you pass --backup-dir. It also records the row count of every table and the migration ledger.
4. Runs helm upgrade with identity and its setup Jobs held back. Helm deletes the old StatefulSet. Its claim stays.
5. Restores into the new, empty PostgreSQL 18 and checks the row counts, ledger and schema against the dump. Then it labels the old claim veltro.dev/postgres-moved-to.
6. Runs helm upgrade again with only your arguments. Identity starts on the restored data, and its install id has to match the one from step 2.

The backup directory holds your users and their password hashes. Keep it private, and delete it once you no longer need it. Pass values as files or --set: the script refuses --reuse-values, since the second upgrade would inherit the first one’s held-back settings. To keep the values you run with now, save them with helm get values veltro -n veltro -o yaml > values.yaml.

When it finishes, or stops after step 4, it prints the rollback: helm rollback to the revision before the move brings back the PostgreSQL 16 StatefulSet on its old claim, as it was at the dump. Anything written after the dump is lost. The printed rollback also removes the veltro.dev/postgres-moved-to label from the old claim; leave it there and the PostgreSQL 18 guard would later let an empty database start beside that claim. Once you’re happy with the upgrade, delete the old claim with the kubectl delete pvc data-<release>-postgresql-0 command it prints.

If you can’t move the data yet, keep PostgreSQL 16 by pinning its tag: veltro-identity.postgresql.image.tag: "16.15-alpine". The chart keeps a 16 tag on the original StatefulSet and claim, and identity runs on 16 and 18.

The refusal relies on Helm’s lookup, which sees nothing under helm template, so it can’t stop an Argo CD sync or a veltro k8s render you apply yourself. Two more checks cover those. The Postgres Service only selects the current StatefulSet’s pod, so identity never spreads across two databases. The PostgreSQL 18 pod also starts behind a check of its own: it won’t start while a PostgreSQL 16 StatefulSet of the release exists, or a PostgreSQL 16 claim without that label. Identity then waits for its database rather than starting an empty install. The script needs a Helm release, though, so if a GitOps tool applies the chart, pin the 16 tag before you bump the chart version. There’s no scripted move for an install that isn’t a Helm release yet, so keep the pin; identity runs on 16 as well as 18.

With your own Postgres (veltro-identity.postgresql.enabled=false), the database and its upgrades are yours: the chart doesn’t touch it, and the script has nothing to move. Identity supports PostgreSQL 16 and 18. To move to 18, use your provider’s major-version upgrade or pg_upgrade, or dump and restore, keeping the veltro_identity_* roles and their grants.

The product charts need nothing. VectorFlow, Sigmatic and Warden’s bundled Postgres comes from the Bitnami postgresql chart 18.x, which already runs PostgreSQL 18.

Roll back

Database migrations can prevent an application rollback. Restore a compatible backup before you install an older chart.

For a chart-only rollback, select a prior Helm revision.

helm history veltro --namespace veltro
helm rollback veltro <revision> --namespace veltro --wait