Progressive rollouts

Source of truth: vectorflow/docs/progressive-rollouts.md (synced at build time — edit the original, not this copy). A canary release can deploy in ordered stages. Each stage picks a slice of the environment’s fleet, bakes for a fixed window, then its gates decide whether the rollout advances to the next stage, completes, holds for review, or rolls back.

Author a plan in the pipeline’s deploy dialog (“Progressive Rollout”), or send one to release.canary.create as rolloutPlan. Without a plan the canary keeps its single-stage shape: one selector, one health-check window, manual broaden — but delivery fencing (below) now applies to it as well.

Stages

Selector	Targets
percentage	The given percent of the environment fleet. Nodes are ranked by sha256(releaseId:nodeId), so a larger percentage always contains every node a smaller one selected — a node never loses the canary as the rollout widens.
labels	Nodes whose labels match every given key/value pair.
nodeGroup	Nodes matching a node group’s criteria.

Stage targets are cumulative: stage n targets the union of stages 1..n, and only the newly-added nodes are pushed a config change. bakeSeconds is at least 30 s and at most 24 h.

Delivery fencing

Targeting governs delivery, not just push order. While a canary release is in flight (CANARY_DEPLOYED or HEALTH_CHECK), both config paths — POST /api/agent/config for vf-agents and the OpAMP remote-config handler for OTel collectors — serve a node the version it is targeted to:

Node	Served
Inside the cumulative stage targets	the canary version
Not targeted yet	the version deployed before the canary
Not targeted, and the canary is the pipeline’s first deploy	nothing — the pipeline is withheld until its stage opens

Targeting is derived from the live Release row at serve time (canaryNodeIds plus the node ids of stages that have started), so a node polling every few seconds cannot pick up a version its stage has not opened. Once the release leaves flight — broadened, or rolled back — nothing is fenced and every node converges on the pipeline’s newest version, which after a rollback is the reverted config the rollback published.

Node selection still applies first: a node the pipeline’s nodeSelector excludes never receives the pipeline at all, canary or not.

The version untargeted nodes are held on is the one the fleet was running when the rollout started (the newest version before the canary was minted), which is also the rollback target. So withheld only happens when the canary is the pipeline’s very first version — nodes that were not running it anyway.

This applies to plan-less canaries too, and that is a behaviour change. Before, a single-selector canary pushed the new config to the canary nodes and any other matching node picked it up on its next poll regardless; now every in-flight CANARY release fences delivery the same way, so a non-canary node stays on the pre-canary version until the release is broadened or rolled back. Broaden and rollback are unchanged.

The rollout view lists the not-yet-targeted nodes and states which version they are held on (or that the pipeline is withheld from them).

Gates

Gates are evaluated once the stage’s bake window expires. All gates must pass.

Gate	Passes when	No data
healthWindow	The bake window elapsed.	n/a
errorRate	errors / eventsIn across the pipeline’s node statuses is ≤ maxPercent.	Fails closed — including when the nodes have processed no events, so a canary that never served traffic cannot pass.
throughputDrop	The drop from the baseline throughput captured at rollout start is ≤ maxDropPercent.	Skipped when no baseline throughput was recorded.

throughputDrop reads the pipeline’s aggregated metric rollups and cannot tell an idle pipeline from a collapsed one: absent rollups read as zero throughput, which against a non-zero baseline is a 100 % drop. Use it on pipelines with continuous traffic, and prefer a customPromQL gate when you need to distinguish the two. | customPromQL | The instant-query result compares as configured (lt/lte/gt/gte) against threshold. | Fails closed. |

autoAdvance: false holds the rollout in HEALTH_CHECK after a passing stage so an operator advances it by hand. autoRollbackOnGateFail: false holds instead of rolling back on failure; the failing gate’s reason is recorded on the release either way. A manual advance never rolls back — it refuses and records the failing gate.

PromQL gates

customPromQL needs a Prometheus-compatible instant-query API:

Variable	Meaning
VF_PROMETHEUS_URL	Base URL, e.g. http://prometheus:9090. Unset ⇒ PromQL gates are unavailable.
VF_PROMETHEUS_BEARER_TOKEN	Optional bearer token. Sent as a header only; never logged or returned by an API.
VF_PROMETHEUS_TIMEOUT_MS	Per-query timeout, default 5000.

When VF_PROMETHEUS_URL is unset the plan editor does not offer the gate and release.canary.create refuses a plan that uses one — a gate that could never read its signal would fail closed and roll every rollout back.

A query must resolve to a single number: a scalar, or a vector with exactly one sample. A multi-sample vector is rejected (the gate would otherwise compare an arbitrary series) and the gate fails closed.

Watching a rollout

/pipelines/<id> shows the live rollout: per-stage status, the stage’s nodes with their agent and pipeline-process status and error counters, the gate results recorded at each bake, and the bake countdown. Buttons for “Advance next stage”, “Complete all remaining” and “Rollback Canary” appear when the release is in the corresponding state.

End-to-end coverage

e2e/tests/progressive-rollout.spec.ts authors a plan through the deploy dialog, asserts that while stage 1 is active the wave-a node is served the canary version and the wave-b node is held on the previous version, then proves the scheduler rolls the release back when stage 1’s error-rate gate fails, that the rollout never opened stage 2, and that delivery is unfenced again once the release leaves flight. e2e/docker-compose.e2e.yml has a two-agent pair under the agents profile (labels rollout-wave=a|b) for manual runs of the Playwright spec. The end-to-end suite was removed from CI on 2026-09-23.