Skip to content
Veltrosecurity

Contributing VectorFlow catalogs and packs

Source of truth: docs/contributing/packs.md (synced at build time — edit the original, not this copy). VectorFlow has two related contribution surfaces: component definitions that make real Vector/OpenTelemetry components available in the builder, and pipeline packs that bundle reusable pipelines, mappings, VRL and tests.

Component definitions

Section titled “Component definitions”

VectorFlow pins Vector 0.54.0 (src/lib/vector-version.ts). Its checked-in component inventory is docs/vector-components-0.54.json; the builder schemas live under:

  • src/lib/vector/schemas/sources/
  • src/lib/vector/schemas/transforms/
  • src/lib/vector/schemas/sinks/
  • src/lib/otel/schemas/ for the pinned OTel contrib catalog

Component definitions are executable TypeScript product code, not inert pack content. They remain AGPL-3.0 and run VectorFlow’s full normal CI; the content-only validation workflow accepts content/** only.

A Vector definition is a VectorComponentDef: type, kind, display metadata, input/output types, and a JSON-schema-like configSchema. Reuse the shared framing, decoding, TLS and auth fragments instead of creating a second local shape. Defaults must match the pinned upstream component; a UI-friendly default that changes emitted Vector YAML is not presentation, it is behaviour.

Before adding a component, check docs/vector-unsupported.md. That file is the reviewed list of deliberate gaps (with a reason); remove the row when you add the real schema. Do not add an unsupported row merely to make the audit green.

Terminal window
node scripts/vector-catalog-audit.mjs
node scripts/otel-catalog-audit.mjs

These commands compare the schemas to the pinned upstream inventories.

Pipeline packs

Section titled “Pipeline packs”

Pipeline packs live under content/packs/ and are Apache-2.0 content. The pack implementation owns the manifest schema and validator; run the repository’s packs:validate script (or scripts/validate-packs.*, as declared by the checkout) rather than validating YAML/JSON by eye.

A useful pack is an installable unit, not a screenshot of one deployment. It contains only what the loader declares: pipeline graphs, shared components, VRL snippets/unit tests, lookup files and Sigmatic field-mapping fixtures. Environment IDs, node IDs, enrolment tokens, connection secrets and absolute local paths do not belong in a pack.

Every content file starts with:

SPDX-License-Identifier: Apache-2.0

The manifest carries semantic version, source/provenance and signature fields. Signature validity is derived from the manifest bytes when an operator public key is configured; never persist or claim a verified boolean independently of that calculation.

Fixture tests

Section titled “Fixture tests”

A pack contribution earns its place with a sample event and an observable result:

  • normalization packs assert the output document, including types and required ECS/OCSF fields;
  • source-family packs include a representative event from that source and a Sigmatic field-mapping fixture;
  • VRL snippets include a persisted VRL unit test whose expected output changes if the mapping breaks;
  • secrets are references only; fixtures use test values that cannot authenticate.

Run:

Terminal window
pnpm packs:validate
node scripts/validate_content.mjs

.github/workflows/content-validation.yml runs on the existing shared self-hosted X64 pool for pull requests. It refuses a content/packs/ tree when no pack validator exists: a green job with no validator would be worse than no job.

Review bar

Section titled “Review bar”

A reviewer checks: the manifest parses with the product’s loader; the version is semantic and changes when content changes; every referenced file stays inside its pack directory; no secret or environment-specific identifier is present; fixture tests prove a consumer-visible transformation; and upstream schemas/content include their exact version, provenance and licence.