Replay, buffers and the DLQ

Source of truth: vectorflow/docs/replay-and-dlq.md (synced at build time — edit the original, not this copy). VectorFlow gives you three ways to avoid losing events: sink buffers, a dead-letter queue (DLQ) for events your transforms drop, and replay. This page covers what each one catches and, just as important, what the DLQ doesn’t.

Sink buffers

Every sink gets a buffer. An environment can set a default buffer policy (memory or disk, a size cap, and when_full as block, drop_newest or overflow), and VectorFlow applies it to each sink that doesn’t set its own. when_full defaults to block, so a slow destination applies backpressure instead of dropping events.

Use a disk buffer on any sink whose destination can be down for a while. It holds events that haven’t been sent yet, so they survive an outage, a backlog of retriable errors (timeouts, 5xx, 429) and an agent restart. It does not help with events the destination permanently rejects; see below.

What the DLQ holds

The DLQ takes events your remap transforms drop, and nothing else.

When a pipeline (or its environment) has a DLQ enabled, VectorFlow turns on reroute_dropped for every remap transform and sends each transform’s dropped output to the DLQ destination: a JSON-lines file on the agent, an S3 prefix, or both. An event lands there when the remap drops it, which means a VRL error with drop_on_error set or an abort with drop_on_abort set.

The DLQ does not hold events a sink rejects. If a destination refuses events for good (a mapping conflict, a non-429 4xx, an auth failure), Vector drops them without retrying and they never reach the DLQ. Vector has no dead-letter output for sinks yet; it’s tracked upstream in vectordotdev/vector#1772 and vectordotdev/vector#10870. VectorFlow will route sink rejections to the DLQ once Vector can.

Until then, catch rejections early and fix the cause. VectorFlow parses Elasticsearch and OpenSearch bulk rejections from the agent’s sink logs and shows the field, the index and a fix you can copy in the pipeline’s error context. Add a rule on the sink_rejection metric, or subscribe your team to the Sink rejection notification event, to hear about them as they happen. Then correct the mapping or the VRL and, if the pipeline also writes to the lake, replay the affected window from there.

Browsing and replaying the DLQ

Open a pipeline and choose DLQ to see its configured destinations. For a file destination, Replay reads the file back through the pipeline. Each replay runs as a job you can follow on the same page.

DLQ replay has some preconditions:

• The page reads the pipeline’s own DLQ setting. A DLQ that only comes from the environment default still works, but the page shows it as not configured.
• You need the Admin role on the pipeline’s team.
• The environment can’t be in read-only GitOps mode or require deploy approval.
• The file must live under /var/lib/vector/dlq.

Replay sources

A replay job feeds stored events back into a pipeline. The source can be:

• the lake, for a time window of a pipeline’s events (the same pipeline or another one), with an optional filter
• S3, GCS or Azure Blob data, read as newline-delimited JSON from a single HTTPS endpoint you give it, such as a signed object URL. Replay doesn’t list a bucket prefix.
• a file on the agent, which is how DLQ replay works

For object-storage and file replays you can also pick the agent that runs the job and set a rate cap. The cap is a throttle: events over the limit in each second are dropped, not delayed, so leave it unset unless you’d rather lose the excess than load the destination. These replays mark each event with replay = true and a replay_dedupe_key. Lake replays carry replayJobId and replayDedupeKey instead.

Searching the lake

The lake gives you guided search plus, for admins, a filter expression: you write the WHERE part of a query and VectorFlow always adds your organisation, pipeline and time-window predicates. It refuses subqueries, semicolons and comments. There’s no free-form SQL editor, by design.