Auditbeat on server — tuning runbook

Source of truth: docs/runbooks/auditbeat-server.md (synced at build time — edit the original, not this copy). Live config: root@server:/etc/auditbeat/auditbeat.yml Rollback copies: /etc/auditbeat/auditbeat.yml.pretune-<timestamp> Tuned and verified: 2026-07-26 (auditbeat 8.19.15, audit userspace 3.0.9, Debian 12).

server runs ~40 containers under Docker/Komodo. auditd the daemon is inactive by design — Auditbeat owns the kernel audit netlink socket (socket_type: unicast) and loads the ruleset itself. Editing /etc/audit/rules.d/* therefore does nothing; the ruleset lives in audit_rules: inside auditbeat.yml.

Path to Sigmatic: Auditbeat writes NDJSON to /var/log/auditbeat/ → the vectorflow-agent container tails it (host /var/log is bind-mounted read-only at /data/logs/server) → VectorFlow auditbeat_remap → OpenSearch index vector-auditbeat and the Sigmatic ingest queue. Sigmatic data source vector-auditbeat = 243cf8bf-90af-422c-807b-d04bd9f95834.

---

1. Three auditd behaviours that were verified on this host

These were each tested with a control variable. They are counter-intuitive and two of them silently break rulesets, so re-test before “simplifying” Section 0 of the config.

(a) auditd is first-match-wins. A never rule only works if it appears before the always rule it overrides. auditctl -a appends, so a never rule added that way after a matching always rule is dead code. Use -A to prepend when testing interactively; inside audit_rules: the file order is the rule order.

This is not theoretical — it bit this very config. The privesc rules were initially placed after the catch-all execve rules, so setpriv --reuid=65534 /usr/bin/passwd was captured but keyed exec, and nothing ever received the privesc key. The specific rule was dead code while looking perfectly correct in the file. Order the execution rules most-specific-first.

(b) -F exe= matches the binary being executed, and does not suppress its children. Excluding /usr/bin/timeout still logged the /usr/bin/id it spawned. So excluding a noisy wrapper does not blind you to what it runs.

(c) -F exe= is inode-resolved at rule-load time, not path-matched. Proven by copying /usr/bin/uname to a second path: the copy’s path was suppressed while the original was not, despite identical contents. Two consequences:

• A path that does not exist on the host silently matches nothing. Container-only binaries cannot be suppressed here. /runc, /usr/local/bin/core, pg_isready, redis-cli, valkey-cli, mariadb-admin are all absent from the host filesystem — rules naming them were confirmed to be no-ops and were removed. Filter those in VectorFlow instead.
• A package upgrade replaces the inode and the rule stops matching until Auditbeat restarts and re-resolves it. Restart Auditbeat after upgrading a suppressed binary.

Check before adding a suppression:

stat -Lc '%d:%i links=%h' /usr/bin/git /usr/lib/git-core/git

/usr/bin/git and /usr/lib/git-core/git are distinct inodes here, which is why the config can suppress git’s internal plumbing while leaving user-facing git fully audited. If a future image makes them hardlinks, that suppression would blind the supply-chain detections — re-check after Docker/git upgrades.

---

2. What the tuning changed

Baseline was 2604 events/min (~3.75M/day), a 1.1 GB on-disk ring rotating every ~13 minutes, and auditctl -s reporting lost 468 (kernel backlog overruns).

Change	Why
Suppress runc, containerd, containerd-shim-runc-v2, docker-proxy	Largest single noise source: ~36% of shipped exec events. Also removed the /tmp/runc-process* churn that dominated the /tmp watches (8600 → 2 events per sample window).
Suppress Komodo periphery + git-core helpers	Komodo polls Gitea every few seconds. User-facing /usr/bin/git stays audited.
Suppress connect() only for newt, olm, dockerd	Service-discovery DNS was ~63% of all connect events. Scoped to connect, so their execution and file activity are still audited.
backlog_limit 8192 → 16384	The old ruleset overran the kernel backlog.
failure_mode: silent → log	Silent drops are indistinguishable from an attacker disabling telemetry.
include_warnings: true → false, and event.original dropped	See §4 — this was a credential-leak path.
Added connect auditing (saddr_fam=2/10, success=1)	The only network telemetry on the box. saddr_fam skips the AF_UNIX flood.
Added file_integrity module over the binary dirs	Detects replaced/backdoored system binaries — nothing else covered this. Emits only on change after baseline.
Added privesc (dual-keyed -k privesc -k exec), ptrace, PAM, LD_PRELOAD, shell-profile, apt, /var/spool/cron rules	Standard persistence/privesc gaps. All low-volume. Privesc rules sit before the catch-all exec rules — see §1a.

Result: ~1434–1553 events/min (~2.1–2.2M/day), a 41–45% reduction, with strictly broader coverage. No new lost events. Auditbeat CPU ~3.5%. On-disk spool fell from 1.1 GB to ~125 MB.

Deliberately not suppressed

Removed from an earlier draft after review. These are all plausible post-exploitation tools, and their combined volume was only ~6%:

redis-cli, valkey-cli, pg_isready, mariadb-admin, mysqladmin (DB/cache clients), xtables-nft-multi (firewall tampering to permit C2 egress is a real technique).

Suppress the specific healthcheck argv in VectorFlow instead — see §5.

---

3. Network coverage

connect() is audited for AF_INET/AF_INET6, successful connects only. A processor then drops loopback / multicast / unspecified destinations.

RFC1918 destinations are deliberately kept. Container-to-container and host-to-host connects are exactly what lateral movement looks like, and keeping them was measured at only 1.2% of connect events (4 per 25s). Dropping by address class would have blinded the main internal threat on a 40-container host for almost no saving.

Loopback was 70% of connect volume, dominated by Docker’s embedded resolver 127.0.0.11:53 — intra-host service chatter with no C2 signal.

Verified in OpenSearch: destination.ip.keyword and destination.port.keyword both resolve (897 docs), and Sigmatic’s generated form destination.port.keyword:443 returns 439 hits.

Tuning lever if this grows: the system resolver (100.96.128.1:53, from /etc/resolv.conf) is the next-largest contributor. Drop by tuple (known resolver IP + port 53), never by address class, so “process talking to an unexpected resolver” stays detectable.

---

4. Credential exposure (open item — needs a decision)

Komodo runs git remote set-url origin https://token:<secret>@gitea.terrifiedbug.com/... on every repo sync. That put a live Gitea push token in process.args, measured on 1266 of 45990 events (2.75%) — roughly 107k events/day. argv is world-readable via /proc, and these events were being written to disk.

Mitigated at the source by a script processor that redacts URL credentials, --flag=value and --flag value (argv is an array, so the split form needs a stateful pass), and tool-specific -a/-p. Redacted events are tagged labels.credential_redacted: true.

Three non-obvious bypasses were found and closed during verification:

• include_warnings: true makes Elastic attach the raw contributing audit records independently of include_raw_message. PROCTITLE records are hex-encoded, so a text regex cannot scrub them — event.original is dropped outright instead.
• The first regex only covered the password family, so a space-separated --token X in process.title still leaked. The flag list is now shared across both forms.
• curl -u user:pass was unhandled. Found the hard way: routine docker exec … curl -u admin:<pw> diagnostics against OpenSearch wrote the admin password into the spool 42 times. The first fix gated -u on argv[0] being curl, which still missed it — under docker exec, argv[0] is docker. The -u handling is now shape-driven, not tool-driven: any -u/--user followed by something of user:secret shape has the secret half destroyed and the username kept, so it works through any wrapper (docker, sudo, env, ssh). Authorization: Bearer|Basic headers are covered too. Verified against a bare -u FOO, which is left intact.

Verified with canaries grepped across the entire serialized event, not just process.args; a benign control string confirmed events still flow unmodified.

Still outstanding — your call:

1. Rotate the Gitea token. It sat in cleartext in /var/log/auditbeat/*.ndjson for a long time. Redaction stops new exposure; it does not undo old.
2. Old copies still exist in the local NDJSON ring (~1 GB, rotates out on its own) and in any vector-auditbeat documents indexed before today.
3. The real fix is Komodo not putting the token in argv (e.g. a credential helper), but that is upstream behaviour.

Those 42 spool entries were sanitised in place. Do not repeat this on the active spool. Auditbeat is appending while Vector tails by checksum+offset; perl -i replaces the inode, and because <redacted> is shorter than the secret every substitution shrinks the file, shifting byte offsets (here by up to ~430 bytes) against Vector’s stored checkpoint. Prefer rotating to a fresh file and editing only closed copies. Never pass the secret on the command line — read it from env or stdin, or the cleanup command re-logs the very thing you are erasing.

What was actually verified afterwards, and what it does and does not prove:

Check	Result	Proves
jq over every spool line	29994 lines, 0 invalid	no corruption
auditbeat + vector logs	0 errors/warns	no failure surfaced
Vector auditbeat_file checkpoint	hash advancing	ingestion resumed
auditd.sequence duplicates, 30m spanning the rewrite	22423 docs / 22380 distinct = 0.19%	—
same, baseline 3h–2h before	118873 docs / 118584 distinct = 0.24%	duplication is pre-existing (at-least-once into two sinks); no spike attributable to the rewrite

Event loss was not proven either way. A lost line simply is not there, and auditd.sequence gaps cannot isolate it because the ruleset deliberately drops events (never rules, the loopback processor, Vector’s docker-default abort). The exposure is bounded to the offset shift above — a few lines at most — but treat it as “no corruption observed, ingestion resumed”, not “nothing was lost”.

Also outstanding: /data/komodo/stacks/opensearch/vector/vector.yaml holds the OpenSearch admin password and a Veltro bearer token in plaintext. That password was the one exposed above, so consider rotating it alongside the Gitea token.

---

5. VectorFlow changes still needed

Vector’s auditbeat_remap currently does this:

if exists(.user) && exists(.user.selinux) && .user.selinux.user == "docker-default" {
  abort
}

That discards every audit event originating inside a container — measured at 42% of the stream, and confirmed with user.selinux.user=docker-default returning 0 documents in vector-auditbeat. On a box whose entire workload is containers, Sigmatic is blind to attacker execution inside them: the deployed supply-chain, discovery and React2Shell rules can only ever fire on host processes.

This is a coverage bug, not an efficiency measure. Replace the blanket abort with targeted, argv-aware drops (these are the container-side binaries the kernel cannot suppress, per §1c):

exe  = to_string(.process.executable) ?? ""
argv = join(array(.process.args) ?? [], " ") ?? ""

# Komodo control plane + its git polling
if exe == "/usr/local/bin/core" || exe == "/usr/local/bin/periphery" { abort }
if starts_with(exe, "/usr/lib/git-core/") || starts_with(exe, "/usr/libexec/git-core/") { abort }
if exe == "/runc" { abort }

# Container healthchecks — argv-matched, so an interactive redis-cli still lands
if ends_with(exe, "/pg_isready") { abort }
if (ends_with(exe, "/redis-cli") || ends_with(exe, "/valkey-cli")) && contains(argv, "ping") { abort }
if (ends_with(exe, "/mariadb-admin") || ends_with(exe, "/mysqladmin")) && (contains(argv, "status") || contains(argv, "ping")) { abort }

Net effect: container attack visibility is restored while the timer-driven churn is still dropped. Expect indexed volume to rise somewhat — that increase is the previously-missing detection coverage.

---

6. DNS detections — fixed 2026-07-26

Corrects an earlier version of this runbook that claimed DNS was dead. It is not. That conclusion came from range-querying @timestamp; the live docs use timestamp.

Deployed rule “Axios NPM Compromise Malicious C2 Domain DNS Query” targets vector-auditbeat with logsource: category: dns. Auditbeat emits no DNS query events, so on that data source the rule cannot fire at any config. Same for “Linux Crypto Mining Pool Connections” (undeployed), which matches DestinationHostname — connect() yields an IP, never a hostname. Both belong on the DNS source, not on auditbeat.

The DNS feed is healthy

Technitium query logs flow through a VectorFlow-managed pipeline (vectorflow.pipeline_version: 13, HomeLab → Production) into vector-dns:

• ~17,200 queries/hour, newest document current to the minute.
• 66.1M docs total, and the count visibly increments (+75 in 20s).
• Documents carry question.questionName, clientIp, answers.*, responseCode.

Two timestamp fields exist, and this is the trap:

Field	Docs	Meaning
timestamp	64.2M	the live path (VectorFlow pipeline 13)
@timestamp	1.92M	dead legacy path — the static vector.yaml dns_remap, stale since 2025-12-06

The static vector.yaml technitium_dns source is genuinely dead: its input file /data/komodo/stacks/technitium1/technitium-logs/dns_logs.json stopped at exactly 1 GiB (1073743126 bytes) on 2025-12-06 and Vector’s ignore_older_secs: 600 skips it. That dead path is what produces the misleading stale @timestamp maximum. Ignore it.

What is already wired up

Sigmatic data source vector-dns (a6025a40-813e-49a7-84b4-41e84c245efd), pull mode, 5-minute poll, timestampField: timestamp, geoip on clientIp. Field mappings added:

Sigma field	Target
query, QueryName	question.questionName.keyword
answer	answers.recordData.keyword
SourceIp	clientIp.keyword

Mapping proven against real data: question.questionName.keyword:misp.terrifiedbug.com returns 5,072,573 hits.

The blocker, and how it was fixed (done 2026-07-26)

Sigmatic’s detection query hardcodes a range filter on @timestamp, ignoring the data source’s configured timestampField. Verified with a positive-control rule matching a domain known to be present: total_scanned: 0, emitted query carried {"range":{"@timestamp":{...}}}. Since 0 of the last hour’s 17,264 docs had @timestamp, nothing was in scope.

Fixed in the DNS Logs pipeline (cmmjg3p4z04c401man9lqf7k1, HomeLab → Production, production/dns-logs.yaml) by appending to its remap transform:

if exists(.timestamp) {
  ."@timestamp" = .timestamp
}

Set in addition to timestamp — no del — so existing consumers keep working. Both ."@timestamp" (quoted, canonical) and bare .@timestamp compile in this Vector build; the quoted form is used because it is unambiguous across versions.

Shipped as pipeline version 14 via vf_edit_pipeline_graph → vf_validate_pipeline (valid, 0 errors/warnings) → vf_deploy_pipeline. Nodes pick up config on a ~30s poll.

Verified end to end:

Check	Before	After
Fresh docs carrying @timestamp	0	336 in a 3-minute window
Positive-control historical test	scanned 0 / matched 0	scanned 390 / matched 88

The Axios NPM Compromise Malicious C2 Domain DNS Query rule was then repointed from vector-auditbeat to vector-dns and redeployed. It validates clean and compiles to question.questionName.keyword:(sfrclak.com OR calltan.com OR callnrwise.com). (percolator_index: null is expected — pull-mode sources are polled, not percolated.)

Still open: “Linux Crypto Mining Pool Connections” remains undeployed on vector-auditbeat. It matches DestinationHostname, which connect() never yields. To make it work, rewrite it against question.questionName on vector-dns.

7. Sigmatic field mappings

vector-auditbeat had zero field mappings, so Sigmatic fell back to its built-in Sigma taxonomy. That works for process/file rules (Image → process.executable.keyword, TargetFilename → file.path.keyword) but broke network rules: DestinationIp resolved to auditd.data.socket.saddr.keyword (the raw hex sockaddr — never matches), while DestinationPort and User resolved to nothing at all and failed validation.

Added:

Sigma field	Target
DestinationIp	destination.ip.keyword
DestinationPort	destination.port.keyword
User	user.name.keyword

---

8. Operating notes

Change the ruleset: edit /etc/auditbeat/auditbeat.yml, then

auditbeat test config -c /etc/auditbeat/auditbeat.yml \
  --path.home /usr/share/auditbeat --path.config /etc/auditbeat
systemctl restart auditbeat
auditctl -l | wc -l          # expect 78
auditctl -s | grep -E 'lost|backlog '

test config only checks YAML/schema — it does not execute the redaction script. After any change to that processor, re-run a canary and grep the whole serialized event:

/usr/bin/env --token CANARYvalue123 true
# then confirm 0 hits for CANARYvalue123 in /var/log/auditbeat/*.ndjson

Likewise verify the privesc key still applies after touching Section 1 (as root uid == euid, so it can only be triggered from a non-root uid):

setpriv --reuid=65534 --regid=65534 --clear-groups /usr/bin/passwd --help
# expect one event with tags[] containing "privesc", uid=65534, euid=0

Measuring volume — always take a post-restart boundary, or the sample straddles the restart and gives nonsense:

F=$(ls -t /var/log/auditbeat/auditbeat*.ndjson | head -1)
A=$(wc -l < "$F"); sleep 60; B=$(wc -l < "$F"); echo "$((B-A)) events/min"

Rollback: cp /etc/auditbeat/auditbeat.yml.pretune-<ts> /etc/auditbeat/auditbeat.yml && systemctl restart auditbeat

Gotcha: the audit rule key lands in tags[], not auditd.data.key — that field is not indexed. Filter with tags: exec / tags: netconn.