Monitoring

The monitoring profile runs an observability stack for the install itself: Prometheus, Grafana, Alertmanager, node-exporter and cAdvisor, next to the core and your products.

sudo veltro up --monitoring
# or at install time:
sudo veltro install --monitoring ...

Raw docker compose ... --profile monitoring up -d works too, once the credential files below are in place.

Prometheus, Alertmanager and Grafana bind to loopback on 9090, 9093 and 3001. Grafana’s admin login is GRAFANA_ADMIN_USER and GRAFANA_ADMIN_PASSWORD in .env.

To reach Grafana from your workstation, tunnel to it over SSH. veltro up --monitoring prints the exact command:

ssh -L 3001:127.0.0.1:3001 you@your-host
# then open http://localhost:3001

Leave GRAFANA_ROOT_URL empty while Grafana is loopback-only. Once you put it behind an operator-facing URL that isn’t loopback, set GRAFANA_ROOT_URL to that URL and Settings > About shows an Open Grafana link.

node-exporter mounts the host’s / with rslave, which Docker only allows when / is a shared or slave mount. systemd hosts already are. A WSL distro without systemd isn’t, so veltro install --monitoring and veltro doctor fail with the fix: sudo mount --make-rshared /, again after each restart.

Scrape credentials

Prometheus reads each product’s metrics with that product’s own credential, from ignored files under observability/prometheus/secrets/ (only .gitkeep is tracked). Prometheus and Alertmanager run as UID 65534, so every file there is 0600 and owned by that user. That’s why the command needs root.

Product	Credential	File	Who writes it
Sigmatic	SIGMATIC_METRICS_TOKEN from .env	sigmatic_metrics_token	The CLI
Warden	WARDEN_METRICS_TOKEN from .env	warden_metrics_token	The CLI
VectorFlow	VF_METRICS_TOKEN from .env	vf_metrics_token	The CLI

The CLI copies the three tokens from .env on every --monitoring install or up. If .env doesn’t have them yet, it makes them first. Change a token in .env and the next veltro up rewrites its file; a file that already matches is left alone.

An install upgraded from before this release may hold a minted VectorFlow service-account key in vf_metrics_token. The next sudo veltro up --monitoring replaces it with VF_METRICS_TOKEN, and you can revoke the old account in VectorFlow’s Settings > Service Accounts. A Prometheus outside the suite can still scrape VectorFlow with a metrics.read service-account key.

With Warden installed, veltro up --monitoring starts the rest of the suite but not Prometheus until the Warden webhook token is in place, and its error names the missing file and where to mint it. A fresh veltro install --monitoring with Warden can’t have it yet, so it finishes and prints the same steps; sign in as the first admin, mint it, then run sudo veltro up --monitoring. Without Warden nothing needs minting, and monitoring starts straight away.

Where Alertmanager sends

With Warden installed, Alertmanager opens Warden cases for platform problems. In Warden, open Webhook sources (<your URL>/warden/webhook-sources) and add a source with slug prometheus-alertmanager, using the field map in observability/warden-alert-field-map.json. Save the token it shows once to observability/alertmanager/secrets/warden_webhook_token, then run sudo veltro up --monitoring again. Alertmanager forwards only firing alerts, so a resolved problem can’t reopen a closed case.

Re-installs and upgrades keep everything under observability/*/secrets/.

On Kubernetes, turn on monitoring in the veltro chart instead; see Kubernetes.