ADR-001: CHAD's detection pipeline and integration boundaries

Source of truth: sigmatic/docs/adr/001-sigmatic-detection-and-integration-boundaries.md (synced at build time — edit the original, not this copy). Status: Accepted (2026-07-27). Product-local ADR for veltrosecurity/chad at 58a68b7 (chad-backend 0.26.2).

Status note (2026-09-04): D1 is shipped. Detection semantics for every ingress now live in one module — backend/app/services/detection_pipeline.py (DetectionPipeline) — and the three callers are wired onto it: synchronous push (backend/app/api/logs.py), the Redis-Streams worker (backend/app/services/log_processor.py), and scheduled pull (backend/app/services/pull_detector.py). The triplication recorded in Context §1 is closed; ingress-owned transport/batching/checkpointing stays outside the shared module. This note amends the ADR in place rather than rewriting it.

It records one load-bearing decision — where CHAD’s detection semantics live and where its edges stop — and pins the gap between what ships today and the approved target. Suite-level identity, apps.<product> roles, session/actor principals, revocation and signing are governed by veltrosecurity/veltro ADR-008 and are not restated here; integration/service-token framing is veltro ADR-004, the MCP/agent plane is veltro ADR-006.

Context

CHAD owns detection: rules, correlation rules, Data Sources, field mappings, alerts, and detection execution (see CONTEXT.md). Four boundary questions became load-bearing once CHAD shipped inside the Veltro suite, and each had a different answer in code, docs, and product intent.

1. Detection semantics are implemented three times. The canonical pipeline (percolate → suppress via exception → threshold-gate → enrich → dedupe alert id → create alert → correlate → notify) exists independently in backend/app/api/logs.py:receive_logs (synchronous push), backend/app/services/log_processor.py:LogProcessor.process_batch (Redis-Streams worker), and backend/app/services/pull_detector.py (scheduled pull). The duplication is acknowledged in code comments (“Mirrors LogProcessor…”), not hidden, but it means a semantic fix can land in one ingress and silently miss two. This is the repo’s largest drift risk.

2. The enrichment seam was designed push, then rebuilt pull. CHAD once pushed alert context to a Warden endpoint. Today the only case-context path is analyst-driven pull: backend/app/api/warden.py:list_related_cases proxies GET {warden}/api/related-cases with an actor-scoped warden:cases:read token. INTEGRATIONS.md still described the removed push shape, and also claimed suite mode exempts the WARDEN_SERVICE_URL host from the outbound SSRF guard — which the code and backend/tests/services/test_webhook_ssrf.py::TestSuiteEgressGuard::test_warden_peer_host_remains_blocked_in_suite contradict: the Warden peer stays blocked.

3. CHAD carried a local Cases surface. backend/app/api/cases.py, backend/app/models/case.py, and frontend/src/pages/Cases.tsx were first-class in standalone CHAD. In the suite, Warden is the sole case authority, and CHAD only holds a bare case pointer written back over backend/app/api/external.py:set_alert_case_ref_external. That surface has since been removed under D3.

4. Raw ingest and the control plane are published on different doors. docker-compose.yml publishes "8001:8001" — a raw, nginx-bypassing uvicorn listener intended for high-volume log ingest — alongside the nginx-fronted :8000 interactive API. Port :8000 runs the complete app.main:app; :8001 runs app.main:ingest_app, which mounts only the shared /api/logs/* router.

Decision

D1 — One shared semantic detection pipeline (approved target, veltro#140). Detection semantics — matching, suppression, threshold gating, enrichment invocation, alert-id derivation, alert construction, correlation, notification — must live in exactly one implementation shared by all three ingresses. Ingress-specific concerns may legitimately differ and stay separate: transport, batching, retry/acknowledgement, and checkpointing. The current triplication is recorded as drift, not as an accepted design.

That implementation now exists as backend/app/services/detection_pipeline.py (veltro#303), and it fixes the order every ingress will follow for one candidate rule match: rule enabled → exception suppression → threshold gate → enrichment → deterministic alert id → alert construction → correlation fan-out → broadcast → notification → MISP sighting. An ingress supplies a DetectionScope (where the batch’s alerts land and who owns them), a DetectionServices bundle (its domain collaborators) and an AlertSink (write each alert as it is built, or buffer for one bulk call), and owns nothing else about detection.

Synchronous push (backend/app/api/logs.py:receive_logs, veltro#304), the Redis-Streams worker (veltro#305), and scheduled pull (veltro#306) now run this module. The pull path keeps its transport, pagination, windowing, and checkpoint behavior in pull_detector.py. The baseline in backend/tests/services/test_detection_parity.py pins shared semantic behavior and intentional ingress-owned differences.

Its correctness properties are already load-bearing and must survive convergence unchanged. In particular, pull polling (backend/app/services/pull_detector.py:run_poll_job) advances the last_poll_at watermark only on a fully successful poll and only to the queried window_end — never to wall-clock now; on failure the watermark stays at window_start so the failed window is re-scanned rather than silently skipped; catch-up is bounded by DEFAULT_MAX_CATCHUP_MINUTES (60); and the half-open window (window_start, window_end] is applied identically to Sigma (PullDetector.build_time_filtered_query) and IOC (PullDetector._run_ioc_detection) detection.

D2 — CHAD↔Warden case enrichment is pull-only (implemented, veltro#138). CHAD does not push alert context to Warden for enrichment. Case context is fetched on demand through backend/app/api/warden.py:list_related_cases. The removed push endpoint must not return, and stale push documentation, config keys, and token references are retired. Alert escalation is a separate seam and is unaffected: CHAD still pushes alert evidence over POST {warden}/api/events/chad and POST {warden}/api/webhooks/chad (backend/app/services/notification.py, redelivered by backend/app/services/warden_reconciler.py).

Corollary: suite mode grants no SSRF exemption to the Warden peer host. The generic enrichment-webhook egress guard applies to every target including WARDEN_SERVICE_URL, as pinned by TestSuiteEgressGuard.

D3 — Warden is the sole case authority; CHAD’s local Cases surface is removed (implemented, veltro#135). Cases, dispositions, approvals, and response runs exist only in Warden. CHAD emits alert evidence and stores a bare case pointer. The former api/cases.py / models/case.py / Cases.tsx surface was implementation drift, not a competing authority, and carried no second case model to reconcile. Migration backend/alembic/versions/20260807b_drop_legacy_cases.py dropped the four legacy tables — deliberately one-way, since no export retains the dropped investigation data — and /api/cases* now returns 404 (backend/tests/api/test_case_paths.py).

D4 — The host-published :8001 listener is an ingest-only trust boundary (implemented, veltro#342; network hardening roadmap veltro#134). backend/app/main.py:ingest_app mounts only the shared logs router and the request validation, redacted error, and organization-scope middleware that ingest requires. Requests remain authenticated against exactly one Data Source bearer token, with an optional IP allowlist, by backend/app/api/logs.py:validate_log_shipping_token; no interactive session, suite cookie, or MCP actor token grants ingest. Control-plane routes are absent, not merely rejected by their normal route-level auth. Host policy, infrastructure-level rate limits, TLS, and network placement remain under #134.

Consequences

Implemented today

• D1 convergence (veltro#140): the shared module backend/app/services/detection_pipeline.py (veltro#303) drives synchronous push (veltro#304), the Redis-Streams worker (veltro#305), and scheduled pull (veltro#306); detection semantics have one implementation. See the 2026-09-04 status note above.
• Pull-only case enrichment and the un-exempted SSRF guard are live and test-pinned (D2).
• D3’s legacy local Cases surface is removed; migration backend/alembic/versions/20260807b_drop_legacy_cases.py is deliberately one-way, and backend/tests/api/test_case_paths.py pins /api/cases* to 404.
• Per-Data-Source authentication for /api/logs/* on :8001 is live, and the dedicated app.main:ingest_app exposes no non-ingest API routes (D4, veltro#342).
• Data Source is the only entity vocabulary: /api/index-patterns* → 404 (backend/tests/api/test_data_source_paths.py), migration 20260726a, confirmed by production probes [TARS-runtime].
• The closed loop log → detection → alert → escalate → disposition/exception writeback works end to end.

Approved target, partially built

• D4 application-level route isolation is implemented (veltro#342); host policy, TLS, and suite network exposure remain under veltro#134.

Costs accepted

• D1 requires reconciling three drifted implementations, including the pull path’s watermark/catch-up semantics, which have no analogue in the push paths — the shared module keeps persistence pluggable (AlertSink) and leaves checkpointing entirely to pull_detector.py. Three of the drifts it removes are observable, and the characterization tests name each. Pull-mode alert ids are derived today without the raw event key, so one event seen by both push and a poll is two alert documents; converging them re-derives ids once, across a single poll window. IOC-match severity, title and tags have three different answers per ingress today and become one. Correlation alerts raised by a poll gain the correlation tag and both linked rule titles they currently lack, so tag-scoped Warden routing stops depending on which door the event came through. Push and pull still write to differently-named alerts indices for the same Data Source; converging that is a data migration, not a code change.
• D2 means CHAD cannot proactively notify Warden of case-relevant context; the analyst pull is the only path, and Warden availability becomes a foreground UI concern rather than a background retry.
• D4 keeps two listeners with different exposure in one image; the explicit ASGI application split is a scaling and blast-radius choice and must be documented wherever deployment is described.

Out of scope (deferred to suite ADR-008 / suite issues)

Per-analyst attribution of writebacks (today fixed backend/app/api/external.py:SUITE_SERVICE_USER_EMAIL, target veltro#141), 60s grant revocation (veltro#129), asymmetric per-service signing (veltro#130), and preserving product-native roles through MCP delegation (veltro#131).