ADR-017: Plain-English questions become reviewed structured queries

Source of truth: docs/adr/017-plain-english-queries.md (synced at build time — edit the original, not this copy). Status: Accepted (2026-10-04). The build is tracked in #1591, with veltrosecurity/sigmatic#819 and veltrosecurity/vectorflow#1010.

The VectorFlow Lake portions below are historical. Managed Lake has been removed; the CHAD Explore decision still applies.

Amended for VectorFlow (2026-10-04, vectorflow#1029): decision 5 still keeps what was asked and what came back, but VectorFlow no longer stores the question and the query in the audit row itself. Everyone in the team can read VectorFlow’s audit rows, including members a pipeline ACL keeps off that pipeline. The row keeps the pipeline, provider, model, query kind or refusal code, and a salted SHA-256 digest that binds the generation id, the pipeline, the question and the query, so the hash chain still covers them. The text is stored with the salt in a record tied to the row. VectorFlow hands it out only to a caller with VIEWER on the pipeline (the check decision 4 runs before generating), or by generation id to an org admin, which is the only way to it once the pipeline is deleted. The answer says whether the record still matches the digest (intact). A record that’s gone comes back without text and not intact. The check is only as good as the row, so evidence also needs a verified chain export. The dashboard, chain export, audit file and Veltro activity feed show the row without the text. The ai.invocation row for these calls is team readable too, so its prompt and output digests use the same per-generation salt and its token counts are rounded up to multiples of 64. It can’t confirm a guessed question and says little about the question’s length. CHAD doesn’t change.

Decision owner: Danny

Context

CHAD’s Explore page (hunting) and VectorFlow’s lake page both expect the user to know the query model: Explore’s filter clauses, Lucene query_string or PPL, and the lake’s Search and Summarize inputs. We want people to ask a question in plain English and get a query back.

ClickHouse has nothing server side for this. clickhouse-client has ?? and ClickHouse Cloud has Ask AI, and neither is available to a bundled lake behind VectorFlow’s API. OpenSearch’s query assistant needs ML Commons agents and sends sample documents to the model. So we build it, and Danny chose one design for both products.

Both products already have the pieces. Each calls its configured provider through the shared AI package (veltro-ai in CHAD, @veltrosecurity/ai in VectorFlow) with the kill switch, data-class gate, redaction and ai.invocation/ai.refusal audit rows from AI governance. Each has a query path that already enforces scope: CHAD’s POST /api/explore/query resolves the index from data sources the caller’s teams can read, and VectorFlow’s lake.search and lake.summarize run behind withTeamAccess with the organisation taken from the session. Earlier, Danny ruled out a lake SQL editor.

The risk is the model. A question can carry injected instructions, and anything the model reads from the data (log lines, sample values, field values) is a second injection path. Whatever the model writes has to be treated as untrusted input.

Decision

1. The model fills in the product’s existing query model as JSON. A strict validator checks the JSON before anyone sees it, and refuses unknown keys, a missing or unknown operator, fields outside the grounded list, values of the wrong type and anything over the caps below. Values are strings, numbers or booleans, or a short list of those where the operator takes a list. Objects are never values. The model never writes SQL, DSL or PPL that runs as written. Targeting a native query language (PPL, say) would need a superseding ADR.
2. Grounding is the schema and the time range, nothing else. The prompt carries field names, the current time and the page’s selected range. It never carries log contents, sample values or field statistics. Field names that come from indexed data (dynamically mapped fields, attrs keys) are still attacker influenced: someone who can write logs can choose them. So they only reach the prompt if they match a strict identifier pattern of at most 128 characters, they sit inside untrusted-data markers with a standing instruction, the list is capped (300 names) and curated names go first (CHAD’s field mappings for the data source, the lake’s fixed columns) rather than in alphabetical order. The question is capped at 500 characters before the prompt is built. There are no tools and no agent loop: the user asks, the model answers once.
3. The user runs it. The page fills the generated query into its normal query controls and runs nothing. A review notice says the query came from AI and lists any clause that excludes results (neq, not_exists) or a time range narrower than the one the page had, because an injected exclusion is the likeliest way to hide events. The model may add a note of up to 300 characters about anything it couldn’t express. The note is plain text, rendered without markdown or links and labelled as AI written. The user can edit the query and then runs it with the page’s own Run button.
4. Scope is checked before the schema is read, and again on the run. The generation endpoint resolves the data source or pipeline with the same check and the same permission as the query path (CHAD: Explore’s data-source visibility check, a 404 for anything the user can’t read; VectorFlow: withTeamAccess("VIEWER") on the pipeline with the organisation from the session) before it reads a single field name. CHAD generation needs a selected data source; with “All visible” selected the control is disabled. The query then runs through the same endpoint or procedure as a hand-written one, with the same tenancy, team scope, permissions and caps. The model never picks the index, data source, pipeline, team or organisation, and its time range is bounded (CHAD 90 days, the lake’s existing 31 days) and shown in the page’s time controls. CHAD’s 90 days binds only the generated range: Explore’s run path has no span cap, so a user can still widen it by hand.
5. Every generation is audited. The provider attempt writes the usual ai.invocation or ai.refusal row, which holds digests only. The feature writes its own row with the user’s question (redacted and capped), the provider and model, and the validated query or the refusal code, because Danny wants the trail to show what was asked and what came back. Questions and queries often carry hunting values such as usernames, hosts, IPs and hashes, and those rows are readable by anyone who can read the product’s audit log for as long as audit retention keeps them. In CHAD that means every view_audit holder, whichever team owns the data source, and the rows sit in a hash-chained log that also feeds the Veltro activity feed, so a secret the redactor missed can’t be removed later. The control tells the user that the question is recorded. The generation id is issued by the server and bound to the user and the data source or pipeline, and it expires after 24 hours. A run that carries it is compared with the stored query in a canonical form, server side, and the run’s audit records accepted or edited, at most once per generation and outcome. CHAD adds this to its existing explore.query row. VectorFlow’s lake.search and lake.summarize write no audit row today, so VectorFlow adds a run row for generated runs only.
6. The configured provider only. The feature uses each product’s configured provider, including Ollama for air-gapped installs. With no provider configured, AI switched off or the kill switch engaged, the control is hidden and the endpoint refuses with the usual named refusal. Field names drawn from indexed data count as the events data class, as CHAD’s field-mapping suggestions already do. CHAD’s fields all come from index mappings, so it needs events allowed. VectorFlow’s fixed lake columns are product schema with no data class, and the lake adds attrs keys only when events is allowed. Generation is rate limited per user and per installation, in a shared store where the product has one (CHAD’s Redis), and a limited request gets a named refusal.
7. Translated. Every UI string goes through the product’s i18n catalogue and checks.
8. Tested at the guardrails.
   • Deterministic tests feed the validator recorded model outputs, malicious ones included: other indices or pipelines, write commands, SQL or DSL fragments, object values, oversized lists and instructions injected into the question or into field names.
   • Each product keeps an eval set of about 30 question and expected-query pairs that runs live against a provider on demand. Its result is accuracy, reported, not a CI gate.

Per product

• CHAD targets Explore’s filter mode. The model returns only a list of {field, op, value} clauses (at most 25, ANDed) and an optional time range. The data source comes from the page. The ops are Explore’s existing allowlist (eq, neq, contains, startswith, endswith, exists, not_exists, gt, gte, lt, lte, in, cidr). A value is at most 512 characters, an in list holds at most 50 values, cidr needs a valid network, and at most 5 clauses use the leading-wildcard ops (contains, endswith). The clauses then compile through the same translate_filter_clause as a hand-written query. query_string and PPL are not targets.
• VectorFlow targets the lake’s guided Search and Summarize inputs: event type, free text, time range and, for Summarize, metric, metric field and group-by. The output is parsed with the lake router’s own input rules. The raw where search (admin only) is not a target.

Consequences

• Users can start from a question and still see and own the query that runs.
• A prompt injection can at worst produce a wrong query inside the user’s own scope. In a SIEM that still matters, because a query that quietly excludes the attacker’s host is detection evasion. The review notice in decision 3 is there for that case.
• The model only sees field names, so it can’t filter on a value it hasn’t been told and may miss what the data calls things. It doesn’t see field types either, so it can pair an op with the wrong kind of field (cidr on a text field, say), and the query comes back empty. The user fixes that in the editor.
• filter mode can’t express everything query_string or PPL can (OR across fields, for one). Lake Search has no field predicates at all, only event type and one free-text term, so attrs keys only help Summarize’s metric field and group-by. A question that needs more gets the closest query and a note that says what’s missing.
• In CHAD the audit log now holds question text and hunting values that the hand-written query path doesn’t record. In VectorFlow a record beside the audit row holds them, readable only by pipeline viewers and org admins.
• Each product adds one audited feature row, a rate limit and an eval set. AI governance and AI features list both features with the events data class.

Rejected alternatives

1. Let the model write SQL, DSL or PPL and run it. Validating a full query language after the fact is the hard problem this design avoids, and Danny already ruled out a lake SQL editor.
2. Ground on sample rows or field values. It would improve accuracy, and it puts attacker-controlled log content in the prompt.
3. An agent that runs queries, looks at results and refines. It runs queries the user hasn’t seen and feeds results back to the model.
4. OpenSearch’s query assistant or ClickHouse Cloud’s Ask AI. The first needs ML Commons agents and sample documents; the second isn’t available for the bundled lake. Neither covers both products.