Truthful Suite Operations

Source of truth: docs/truthful-suite-operations.md (synced at build time — edit the original, not this copy). The identity launcher publishes one public, PII-free suite status document for VectorFlow, Sigmatic, and Warden. A product is only ready when its authenticated product-peer status probe succeeds and any comparable running release version matches the deployment pin.

Signal	States	Authority
Component status	ready, degraded, down, unknown	Exchange-backed /api/peer/v1/status responses and deployment pins

A timeout, unreachable endpoint, unsupported schema, malformed response, unavailable service credential, or unconfigured peer is never silently converted to ready.

Status contract

Identity exposes GET /api/suite/status without authentication. The response contains no user data. Cached refreshes call each product’s authenticated POST /api/peer/v1/status endpoint with a short-lived token for the fixed suite-status@service.veltro.invalid service principal. Identity never enumerates users or borrows a human grant for this route. Requests are rate-limited, refreshes are cached for 10 seconds, and every refresh is audit-logged.

The shipped v1 conformance file remains immutable in conformance/suite-status-v1.json. The public aggregation response is versioned separately as conformance/suite-status-v2.json.

The status document carries no onboarding block. Setup steps are on Home, where each product answers its own setup list (ADR-016 D9).

Peer reads

A product can pass its own health check and still fail every read identity makes of it for a person. Identity renders the rest of the page and names that product unavailable, so the failure stays quiet. GET /api/suite/peer-reads runs identity’s nav, Home and pending-approvals reads of each installed product, through the same readers and timeouts a page load uses, and answers:

{
  "products": {
    "sigmatic": { "nav": "ready", "home": "ready", "approvals": "ready" },
    "warden": { "nav": "ready", "home": "ready", "approvals": "unavailable" }
  },
  "exchanges": {
    "sigmatic": { "warden": "unavailable" },
    "warden": { "sigmatic": "ready" }
  },
  "declarations": {
    "sigmatic": { "status": "refused", "refused": { "warden": ["warden:cases:write"] } },
    "warden": { "status": "ready", "unused": { "sigmatic": ["sigmatic:rules:deploy"] } }
  },
  "teamPublish": {
    "sigmatic": { "status": "failed", "reason": "revoked_credential" },
    "warden": { "status": "published" }
  },
  "unavailable": ["sigmatic", "warden"],
  "installId": "8fa9a09a-192d-4be2-9972-df7b3b656217"
}

It runs as the same suite-status@service.veltro.invalid principal, whose tokens carry only <product>:peer:status:read. The approvals read asks for the nil UUID 00000000-0000-0000-0000-000000000000. Identity issues random UUIDs, so no person holds it, but each product still asks identity what that subject may decide before it lists nothing. The route takes no credential, returns no user data, and has the same per-IP rate limit, 10-second cache and refresh audit as the status route (suite.peer_reads.refreshed).

exchanges covers calls between products. Each product’s reviewed workload matrix (apps/identity/src/exchange/exchange.ts) lists the scopes it may mint on other products: Sigmatic’s rule editor reads and writes Warden runbook bindings, Warden comments on Sigmatic alerts, and so on. For each installed product that calls another installed product, identity reads the stored workload credential and runs the exchange’s own workload x audience x scope check on every scope in that matrix. A missing or revoked credential, or one identity-init didn’t reconcile, reads unavailable. Nothing is minted.

exchanges starts from the matrix, so it can’t see a scope a product asks for that its matrix never listed. That was #1340. declarations starts from the product (#1359). Each product serves the pairs it mints at GET /api/peer/v1/exchanges, read with the same status token:

{ "contract_version": 1, "product": "sigmatic", "exchanges": { "warden": ["settings:write", "warden:cases:read"] } }

The product builds that answer from the one constant its own mint checks before calling the exchange, and a test in each product repo fails when a call site mints a pair the constant doesn’t list. Identity checks every declared pair against the reviewed matrix and the stored credential. ready means it would mint all of them. refused lists the ones it wouldn’t. undeclared is a product release from before the route: Sigmatic and Warden answer 404, VectorFlow its not-found page with a 200. unavailable is any other failed or malformed answer. unavailable at the top lists products with a failed read, a failed exchange, or a declaration that’s refused or unavailable.

Each declaration also lists, under unused, any pair the reviewed matrix grants on another product that the declaration doesn’t name (#1399). Nothing mints those, so they only make a stolen workload credential worth more. An unused pair is an over-grant to trim from the matrix, not a failure: it doesn’t change the status or put the product in unavailable.

teamPublish says whether each installed product has published its teams into identity’s directory (#1205). A product publishes once with POST /api/authority/teams and retries every five minutes until it lands. Identity audits every publish as authority.teams.published and every refusal of a known workload as authority.teams.publish_refused with the reason. The product’s latest row decides its status: published, or failed with the reason (malformed_request, revoked_credential or not_a_product). With no row yet it’s pending, and it’s unavailable when identity can’t read its audit table. None of these put a product in unavailable.

Homelab promotion (scripts/promote_homelab.py) reads it after every deploy and no-op. It fails unless every deployed product is listed with all three reads ready, every reported exchange is ready and every declaration is ready, allowing two minutes for a just-deployed product to settle. undeclared passes only for a product release no newer than the last one before the route, which UNDECLARED_EXCHANGES_THROUGH names. The step records every unused pair as over_grants (product: audience scope) and never fails on them.

Then it checks teamPublish as its own team-publish step. A failed publish fails promotion straight away, with identity’s reason. pending and unavailable get seven minutes, one missed retry plus margin, to turn published. After that unavailable fails, and so does pending, unless the product release is no newer than the last one before the publisher, which TEAM_PUBLISH_PENDING_THROUGH names.

It also guards against a deploy that comes up on an empty datastore (#1512). Identity writes a random install id into its database once and reports it as installId, or null when it can’t read it. After the deploy, and on the no-op path, the install-id step compares what identity reports with a baseline. A different id, a null, or a missing field fails promotion: the datastore looks freshly initialised, and the old data is probably still on the previous volume. Every health check passes in that state, which is why it gets its own check.

The baseline is the after id of the install-id step in the newest published release’s promotion evidence (its promotion-*.json asset). The live id isn’t enough on its own: a refused deploy leaves the stack on the fresh datastore, so a re-run would read the fresh id, find nothing to deploy and pass as a no-op. When no published promotion recorded an id, the baseline is the id identity reports before promotion changes anything. The install-id-before step records the baseline, where it came from (source) and the live id. If you reset identity’s datastore on purpose, dispatch the next release with accept_install_id set to the id identity now reports (--accept-install-id on the script). Its published evidence makes that id the baseline from then on. A recovery accepts the input too, but publishes nothing, so it only covers that run: the next release still needs the input. When there’s no baseline at all, the step records why in unchecked and passes: identity didn’t answer before the deploy, or predates the field (“identity predates this check”). A recovery may also deploy an identity that predates the field, and that’s recorded the same way. When the deployed identity doesn’t answer, the peer-reads failure is what fails the run.

installId is the id identity’s datastore was given when it was created (migration 035, #1512). It lives in veltro.installation, one row nobody can update, delete or truncate, and it travels with every dump and restore. So it changes only when the datastore was wiped and created again. It’s null when identity has no runtime or can’t read the row, and an identity from before 035 leaves the field out.

A cut deploys the newest green source, which can be older than the commit promote_homelab.py runs from. Each check that reads a peer-reads field runs only when the promoted identity reports that field. When it doesn’t, the step’s unreported names the field with “identity predates this check” and the check passes, forward or recovery (#1458). That covers exchanges, declarations (and the over_grants taken from them, recorded as null) and teamPublish. Suite 0.22.0 failed a good deploy because the team-publish step read a missing teamPublish as pending.

Gateway router requests

Every product page location in the gateway asks identity’s mint for a request assertion through auth_request before it reaches the product. Identity is a Next.js app, and Next answers a request carrying RSC: 1 with a 307. auth_request turns that 307 into a 500. Suites up to 0.17.0 forwarded the Next router headers to the mint, so every soft navigation and prefetch in Sigmatic, Warden and VectorFlow failed while each health check passed (#1366). The mint locations, and the product-unavailable fallbacks that also call identity, now clear those headers.

Promotion sends an anonymous router request (RSC: 1, Next-Router-Prefetch: 1) to /sigmatic/, /warden/ and /vectorflow for each deployed product. It needs no credential, because the mint answers an anonymous request with 401 and the gate passes it on to the product. Any answer below 500 passes, redirects included; a 5xx fails the promotion. A recovery records the answers without failing, since its target suite may predate the fix.

Health evidence

Compose healthchecks cover every long-running service. Backend checks exercise product dependencies, frontend checks fetch the image’s existing nginx root, and the gateway checks the identity readiness document through nginx. Sigmatic and Warden workers refresh per-container loop-heartbeat files; freshness proves the consumer loop is making progress rather than merely proving PID 1 exists. The optional VectorFlow agent uses the same loop-progress rule.

A product’s peer response maps healthy to ready, degraded to degraded with reason product_degraded, unhealthy to down with reason product_unhealthy, a timeout to down, and stale or incompatible evidence to degraded. An unavailable peer or service credential stays unknown.

Version state and upgrades

versions.env is the deployment pin authority. Compose passes VF_VERSION, SIGMATIC_VERSION, WARDEN_VERSION, and VELTRO_VERSION into identity. Product-peer release versions are compared with those pins; a mismatch is degraded with reason manifest_mismatch. Identity reports the source revision baked into its immutable image as sha-…, rather than relabelling its deployment pin as observed runtime evidence. A revision and a release version are not falsely compared.

VELTRO_VERSION is the release version, so the public v2 document’s suiteVersion is that release as vX.Y.Z: the same install version the chrome footer and Settings > About show (ADR-016 D33). On Helm the identity chart sets it from the chart’s appVersion, which a release stamps. veltro status compares it with the newest GitHub Release when online and reports offline without degrading local component health when the release lookup is unavailable. Version-state precedence for comparable release versions is:

1. missing running/pinned version → no mismatch claim;
2. comparable running and pinned versions differ → manifest_mismatch;
3. latest release unavailable or malformed → unknown;
4. running older than latest → update_available;
5. otherwise → current.

There is no browser update action. Operators use veltro upgrade, which verifies the immutable release manifest, takes a backup, replaces the release files, waits for health, and verifies running image digests. Module and suite publishers still update exact pins through reviewed release workflows.

Recovery and restore path

Operational status is read-only; it does not heal, restart, pull, or roll back services. If a new module release or pin is defective:

1. stop promotion and retain the failing status/HTTP evidence;
2. repair forward with a new immutable module release whenever possible;
3. if rollback is explicitly authorized, restore the compatible pre-change database snapshot and select the prior immutable module versions through the same reviewed release workflow;
4. verify schema compatibility before starting old images;
5. rerun exact-version, Compose health, and HTTP acceptance checks.

Never directly mutate Komodo, SSH into the homelab, or edit a deployed Compose stack outside release-suite.yml. A status value is evidence, not deployment authority.