Veltro benchmark harness (veltro#401)

Source of truth: docs/benchmark.md (synced at build time — edit the original, not this copy). Slice one of #48: the instrument. A reproducible benchmark harness plus a single versioned workload contract. This slice deliberately publishes no numbers — no supported tier, no capacity claim, no limit. What it produces is measurements of one named deployment at one point in time, with every input fully identified.

What it measures

Against a named, fully identified deployment (default: veltro, the Compose project veltro install creates):

Metric	How	Phase
Sustained ingest rate	queued ingest at a fixed target rate; events/sec actually acknowledged by the ingest listener, plus per-request queue depth	sustained
Burst ingest rate	max-effort queued ingest; peak accepted events/sec (1s buckets), backlog, and post-run drain time	burst
Detection latency (inline)	sync-path sentinel events: request duration, alert created_at − event @timestamp, and time-to-searchable in OpenSearch	sync_latency
Detection latency (queued)	sentinel events in the queued stream: alert created_at − event @timestamp after worker processing, observed via the run’s own drain window	sustained sentinels
Alert throughput	sync phase: alerts_created from Sigmatic’s own response per request (exact). Queued phases: deterministic alert-doc ids reconciled against OpenSearch after drain (one doc per (rule, event))	all
Active IOC cardinality	the run provisions its own MISP-protocol TI source (compose/misp-mock.yml), seeds the contract’s declared indicator population, syncs it through Sigmatic’s pull-mode MISPSyncService, and counts the indicators active on the deployment’s own sigmatic-indicators index after a refresh	TI leg (pre-load)
#35 terms-join boundary	the run provisions a run-scoped pull-mode Data Source with a deployed gate rule (Sigmatic’s run_poll_job returns early when no deployed rules exist), writes events into its log index, and invokes Sigmatic’s own run_poll_job once — executing the real chunked inline-terms join (fetch_active_values pages × bounded queries per mapped field) at a cardinality above the live derived limit; zero pull-lane matches refuses the run	TI leg (pre-load)
TI sync time	wall-clock duration of that pull-mode sync, plus fetched/cached/indexed counts, from MISPSyncService.sync_iocs itself	TI leg (pre-load)
IOC match reconciliation	every queued event carrying a seeded indicator must produce exactly one alert evidencing the match — an enriched Sigma alert (top-level ioc_matches) when a rule matched, otherwise an IOC-only alert reconciled by id under Sigmatic’s sentinel ioc-detection rule id. Sigmatic also raises that IOC-only alert when it loses all of an event’s Sigma alerts; it counts as IOC evidence, and those events are reported as sigma_lost_ioc_only_fallback_events. Zero observed matches refuses the run	queued phases
Correlation alerts	the contract declares a correlation over the two fixture rules with a deterministic pair plan (correlation_pair_plan); the harness emits one rule-a-only and one rule-b-only event per pair sharing the entity field, then reconciles correlation alerts by mirrored deterministic id. Zero observed correlations refuses the run	response leg
Alert-to-case throughput	correlation (and primary) alerts are escalated into Warden via process_sigmatic_alert; cases are attributed only when linked to this run’s alert external ids. Throughput is attributed-cases / response-leg duration, gateable as response.alert_to_case_throughput_per_second	response leg
Response/audit writes	a run-scoped manual-trigger runbook (case.update stamp + comment) executes inline on each attributed case; audit rows are counted only for those cases (never provisioning-time audits). Zero run-stamped comments or zero attributed audit rows refuses the run	response leg

Explicitly not measured, and why (recorded in every result set as excluded_metrics):

• Normalization latency in isolation — Sigmatic exposes no per-stage timing; normalization runs inside the ingest path with percolation and persistence. The closest observable proxy is the sync-path ingest-to-alert latency (event @timestamp → alert searchable in OpenSearch), which includes normalization, detection, storage, and refresh. It is reported as exactly that, never renamed.
• IOC cardinality — measured as of veltro#722 (see the table above); removed from this list. The workload contract now declares a real, deterministic indicator population, and every run provisions the TI source it needs rather than assuming one.
• Response latency (human-gated) — analyst triage, approval gates, and interactive case actions wait on a person; a synthetic number over a UI nobody drives would be fiction. The automated path is measured: alert- to-case throughput across the Sigmatic→Warden boundary and run-attributed audit rows from the manual-trigger runbook (see the table above). Only the human-gated portion stays excluded.

The workload contract

scripts/benchmark/workload-contract-v1.json is the single, versioned definition of the benchmark: event shape (fields and generation rules), rates and durations per phase, and the rule/IOC population the events exercise (one high-signal auth-failure rule, one narrow sentinel rule, and a deterministic IOC population whose values derive from the contract’s own seed). Later tiers reuse this contract as-is — that is #48 criterion 5 — and a results set embeds the exact contract document it ran.

The ioc_population member is the threat-intel leg (#48 criteria 1, 2 and 6): count indicators of ioc_type, injected into the declared event field on a deterministic stride derived from match_rate (one event in every round(1/match_rate)), with every value and MISP attribute uuid a pure function of seed — two runs of the same contract produce the identical indicator set, so reconciliation can assert exact counts instead of averages.

Pass/fail thresholds

Thresholds are data, not code. They are declared in the workload contract’s optional thresholds member — one gate per measured metric, each naming the metric id, the threshold value, and the comparison direction:

"thresholds": {
  "metrics": {
    "sustained.ack_rate_events_per_second": {"value": 150, "direction": "at_least"},
    "sync_latency.ingest_to_alert_p95_ms": {"value": 2000, "direction": "at_most"}
  }
}

The member is validated with the rest of the contract: the workload validator’s exact-key rule accepts thresholds as the one optional member and refuses malformed gates (wrong shape, unknown metric id, unknown direction, non-numeric value) at load. The shipped scripts/benchmark/workload-contract-v1.json carries the member with an empty metrics map — declaring real gates in that file needs no code change.

Metric ids the writer can evaluate today. Percentile metrics additionally name the observation count that evidences them:

Metric id	Direction	Measured value	Observation evidence
sync_latency.ingest_to_alert_p95_ms	at_most	sync phase ingest_to_alert_ms.p95_ms	ingest_to_alert_observations
sync_latency.searchable_delta_p95_ms	at_most	sync phase searchable_delta_ms.p95_ms	searchable_delta_observations
sustained.ack_rate_events_per_second	at_least	sustained phase ack rate	—
sustained.alerts_throughput_per_second	at_least	sustained phase alert throughput	—
burst.peak_ack_rate_events_per_second	at_least	burst phase peak ack rate	—
drain.queued_sentinel_ingest_to_alert_p95_ms	at_most	drain window queued-sentinel latency, p95	queued_sentinel_observations
ti.active_cardinality	at_least	indicators active on the deployment after the run’s sync	—
ti.sync_duration_ms	at_most	pull-mode sync duration reported by MISPSyncService	—
response.alert_to_case_throughput_per_second	at_least	attributed cases per second across the Sigmatic→Warden boundary	—

A gate declared against any other id is refused; so is a gate whose metric produced no measurement in the run, and so — for percentile metrics — is a run that recorded zero observations behind the number (summarize_ms([]) zeroes out, and a zero silently passes any at_most gate). An unevaluable gate is never silently counted as met.

The result document carries one threshold_records entry per declared gate — metric id, threshold, direction, measured value, met — and the run-level verdict under pass_fail is derived from those records, never restated: every record met is passed: true; any miss sets passed: false and names the missed metrics in failed_metrics. With no gates declared the document says evaluated: false and publishes measurements only — a missing gate is never a defaulted pass.

results.load_results(path) reads a published document back and rejects any document whose schema version differs from what this harness writes (veltro-benchmark-results/v2) — older documents are refused with their version named, never silently mis-parsed.

Measurement matrix

The matrix is the set of operating points the campaign measures, expressed as contract data — never as code paths. Today’s axes:

Axis	Points
Phase mix	fixed by the contract (sync_latency, sustained, burst)
Rule population	high-signal auth-failure + narrow sentinel (fixed)
IOC cardinality	one point below and one point above the deployment’s OpenSearch index.max_terms_count

The shipped contract executes the boundary point: its ioc_population.count sits deliberately above the terms limit that veltro#35’s chunked pull-mode join exists to survive, and the run enforces the premise instead of trusting it — the limit is read live from the target cluster (index.max_terms_count: the index setting, else the cluster’s published default) and a contract whose count does not exceed the observed limit is refused. The limit’s value appears in no harness literal; a sub-limit control point runs by editing the contract copy, not the harness — and the pull leg EXECUTES the chunked join itself, because reading a setting alone would verify nothing.

Supported, tested, configurable

Every published capacity statement must say which of three distinct claims it is making. They are not interchangeable, and each requires different evidence:

• Supported — a limit this suite stands behind: a deployment operated at or below it is our problem, and incidents at that load get fixed rather than declined. Evidence required: repeatable runs of this harness on fully identified deployments, at the claimed load with operational headroom (not run-to-failure), across the workload contract’s phases, plus an explicit decision to own the number. A single green run is not a supported tier.
• Tested — a ceiling a specific measured run actually observed. Evidence required: the published result document itself (image digests, topology, resources, contract id/version). It promises nothing about any other deployment, duration, tenancy shape, or data mix beyond what was measured.
• Configurable — the maximum value a setting accepts (queue depth, worker count, rate limits) as defined by the module’s configuration surface. Evidence required: the configuration schema itself — and nothing about performance. Most configurable maximums are untestable or destructive at full scale; a configurable maximum is neither supported nor tested until a benchmark result says so.

The campaign layer is scripts/benchmark/campaign.py plus the small- and medium-SOC contracts. It evaluates gates against real run documents, refuses a deployment digest mismatch, and writes a tier statement with per-document SHA-256 and workflow/artifact links. The operator-facing tier status lives in the root README. Both tiers remain unmeasured until their protected host-side dispatches succeed; no number is published from a local-only or unlinked result.

Version identification: digests, never tags

Image tags are labels a registry can move, so naming one in a result would look precise without proving which image ran.

The harness therefore records, for every module image and every dependency service, the resolved OCI image digest at measurement time (docker image inspect RepoDigests): the image that actually ran.

Consequences, stated in every result set:

• A run is reproducible in the sense that its inputs are fully identified.
• To rerun a measurement, install the same suite release and check that the digests match the recorded ones. The harness never moves tags; it only records what it measured.

Refusal semantics

The harness refuses to run (with a REFUSED: message and nonzero exit) when it cannot fully identify the deployment or the workload:

• no container carries com.docker.compose.project=<project> for the named deployment;
• any required service is missing, not running, or unhealthy (sigmatic-backend, sigmatic-worker, suite-redis, suite-postgres, opensearch); other project containers are recorded but do not gate the run;
• any project container’s image has no OCI repo digest (locally built/unknown provenance — a measurement whose inputs are unknown is not evidence);
• the --ingest-url is not Sigmatic’s direct ingest listener (an unauthenticated POST must be refused with 401);
• the workload contract is unsupported (wrong schema/version) or invalid;
• the Sigmatic detection path is broken: the pre-run smoke (one queued sentinel → searchable alert) does not complete within 120s;
• fixture provisioning is refused by Sigmatic policy (e.g. an approval-gated environment refuses the deploy) — the harness reports the native error and exits;
• the TI source the contract requires cannot be honoured: no running misp-mock service in the project, the fixture unreachable (from the host for seeding or from inside sigmatic-backend for syncing), the pull-mode sync fails or fetches fewer indicators than declared, the post-sync active cardinality differs from the contract’s count, or the contract’s cardinality does not exceed the deployment’s observed index.max_terms_count;
• the IOC reconciliation observes zero matches for events that carried seeded indicators, or Sigma alerts carrying matches without their ioc_matches enrichment — a workload declaring IOC matching while nothing matches is the failure mode the run exists to catch, and it refuses rather than reporting a clean run.
• the response leg cannot run: Warden services (warden-backend, suite-postgres, suite-redis) are missing/unhealthy, fixture provisioning fails, correlation pairs produce zero correlation alerts, cases cannot be attributed to this run’s alert ids, or run-attributed audit rows / run-stamped comments are zero — never report a zero reading as a clean run.

Running it

Single run

Run from the docker host that owns the deployment (docker, docker exec, and docker stats must work and 127.0.0.1:8001 must reach Sigmatic’s direct ingest listener, the default loopback bind of SIGMATIC_INGEST_BIND_ADDRESS):

python3 scripts/benchmark/run.py                      # default project veltro
python3 scripts/benchmark/run.py --project veltro --output results.json --verbose
python3 scripts/benchmark/run.py --keep-fixture       # leave the fixture for inspection

The full default run takes on the order of 2–4 minutes (60s sustained at 200 ev/s, 15s burst, drain, reconciliation). The results JSON lands at veltro-benchmark-<run-id>.json by default.

Capacity campaigns (small and medium SOC, veltro#724 / #921)

The small campaign requires three independent runs at the shared workload’s declared rates. The medium campaign requires one tested run with rate_multiplier: 10; the campaign runner writes that effective workload and passes it to every single-run invocation. A run captured at 1× is rejected by the 10× campaign even though both reuse the same versioned base contract.

python3 scripts/benchmark/campaign.py validate \
  --campaign scripts/benchmark/campaign-small-soc-v1.json
python3 scripts/benchmark/campaign.py validate \
  --campaign scripts/benchmark/campaign-medium-soc-v1.json

Run each campaign by hand on the Docker host that owns the test deployment. The CI capacity path was retired on 2026-09-23, and capacity tiers were dropped from 1.0. Validate the campaign contracts above, then collect runs with campaign.py:

python3 scripts/benchmark/campaign.py run \
  --campaign scripts/benchmark/campaign-small-soc-v1.json \
  --project veltro --output-dir ./benchmark-small-runs
python3 scripts/benchmark/campaign.py run \
  --campaign scripts/benchmark/campaign-medium-soc-v1.json \
  --project veltro --output-dir ./benchmark-medium-runs

veltro is the project a default veltro install creates (Compose names it after the install directory). Pass --project when your test stack lives in a differently named directory. The retired CI testbed is not provisioned by these commands.

The runner writes the effective workload, calls run.py for each trial, and evaluates the resulting documents. Keep the raw results and tier statement with the deployment’s image digests so each measurement can be traced to the tested stack. You can also evaluate previously captured run documents offline:

python3 scripts/benchmark/campaign.py evaluate \
  --campaign scripts/benchmark/campaign-small-soc-v1.json \
  --results run-01.json run-02.json run-03.json \
  --output tier-statement.json

Without durable workflow and artifact links, the evaluator reports publication.ready: false. Neither tier has a published measurement here. Leave README capacity values absent until the measurements and their evidence are reviewed and linked.

What a run does, in order:

1. Identify the deployment (containers, digests, resources, health) — refuse on any gap. Digest evidence is written before any load.
2. Check the ingest listener.
3. Provision a dedicated, run-scoped fixture through Sigmatic’s own services (same pattern as verify_ingest_listener.py): a push-mode Data Source veltro-bench-<8hex>, the machine actor that owns the fixture’s rows, the contract’s rules deployed via apply_sigma_rule_deployment (with the mandatory positive-match evidence recorded the way the rule-testing API records it), and the OpenSearch indices seeded with the workload’s field shape. Real data sources and rules are never touched. The actor is one Sigmatic users row, bench@veltro.local, created by the first run and reused by every run after it; teardown never deletes it, because Sigmatic’s audit and version rows reference users without ON DELETE CASCADE. It’s built the way Sigmatic builds its own machine actors (role admin, default organization, no identity_sub). Identity owns sign-in, so the row carries no credentials, nothing signs in as it, and no identity account is created. The harness drives Sigmatic’s services inside sigmatic-backend and never goes through the HTTP API, so it needs no identity session.
4. Smoke the detection path (one queued sentinel must become a searchable alert) or refuse.
5. Provision the threat-intel leg when the contract declares indicators: derive the deployment’s terms limit and verify the boundary premise; seed the deterministic population onto the run’s MISP fixture (host side) under a seed-stable population tag — so reruns against the same fixture volume reuse the same tagged events instead of duplicating them. Then, inside sigmatic-backend: prepare_ti snapshots, replaces and commits the global MISP config (its payload reaches the harness before anything can fail, so teardown always has provenance), and sync_ti syncs through Sigmatic’s pull-mode service, refusing on any shortfall. Finally the pull-mode boundary leg: provision a run-scoped pull source, bulk-write events into its log index, invoke Sigmatic’s own poll entry point once, and reconcile the chunked join’s matches against the carrying events — zero observed matches refuses the run.
6. Run the phases (sync_latency → sustained → burst), sampling container stats throughout.
7. Drain the ingest queue to zero (or the window expires), measure queued sentinel detection latency from the stored alerts, reconcile every queued event’s deterministic alert ids against OpenSearch together with the IOC-leg evidence, and record the storage delta of the run’s own indices.
8. Response leg when the contract declares response_leg: emit the guaranteed correlation pairs, wait for correlation alerts by mirrored id, provision a run-scoped Warden runbook, escalate the alerts into cases, execute the runbook inline, and reconcile attributed cases + audit rows against the run stamp — zero correlation alerts or zero attributed audit rows refuses.
9. Clean up the fixture and the TI state (undeploy + delete data source/rules/correlation + delete the run’s OpenSearch indices; restore or delete the prior MISP config according to provenance — a config whose pre-run state was never captured is left untouched, never deleted on a guess; evict the population’s indicators from cache and index), unless --keep-fixture. If any of it fails, the run still writes its results, then names what it left behind and exits 1.
10. Write the results JSON — schema veltro-benchmark-results/v2, with digests, topology, resources, contract id/version + the contract document, run duration, per-phase measurements, the threat-intel section, the response section, exclusions, limitations, and the pass/fail verdict derived from the contract’s declared thresholds (or pass_fail.evaluated: false when none are declared).

Something recoverable (e.g. --output to a read-only path) fails loudly; a core guarantee (identification, ingest boundary, worker smoke) refuses the whole run.

Alert observability design

Sigmatic stores alert log_document with indexing disabled, so alert existence cannot be queried by content — but Sigmatic derives deterministic alert ids from the raw event (raw_event_key + generate_deterministic_alert_id in backend/app/services/alerts.py). The harness mirrors those two pure functions exactly (benchmark/contract.py) and polls OpenSearch _doc/_mget by id. The mirror is verified against Sigmatic’s source in scripts/test_benchmark.py; a drift would make sentinels undetectable and the run would fail closed, never silently report wrong numbers. OpenSearch reads run via docker exec into the opensearch container, reusing its own compose-provisioned $OPENSEARCH_INITIAL_ADMIN_PASSWORD (the password never leaves the container).

Development

Unit tests (stdlib-only, no docker/network needed):

python3 -m unittest scripts/test_benchmark.py scripts/test_benchmark_campaign.py \
  scripts/test_benchmark_product_models.py -v

All three run in verify’s check job as part of the scripts test list. See AGENTS.md for the suite’s pre-push gate discipline. Run live measurements by hand against a disposable test deployment.

The provisioners run inside sigmatic-backend and warden-backend and use the products’ ORM models directly, so nothing in veltro executes them before a live run does. test_benchmark_product_models.py reads both scripts’ syntax trees and checks every app.* import, every keyword set on a product model, every attribute read on a product class or on an instance the script creates or imports, and every call to a product function or such an instance’s method against the Sigmatic and Warden source verify checks out at the pinned releases (.composition/<product>, from versions.env). In CI a missing checkout fails the test. Locally it skips unless you populate the checkout, for example:

mkdir -p .composition/sigmatic .composition/warden
git -C ../sigmatic archive "v$(sed -n 's/^SIGMATIC_VERSION=//p' versions.env)" backend/app | tar -x -C .composition/sigmatic
git -C ../warden archive "v$(sed -n 's/^WARDEN_VERSION=//p' versions.env)" backend/app | tar -x -C .composition/warden