Skip to content

Scaling real-time file monitoring with eBPF (Datadog, 2025-11-18)

Datadog's File Integrity Monitoring (FIM) team describes how they reduced an eBPF-collected file-event stream of >10B events/minute down to ~1M events/minute (≈94% reduction) by pushing rule evaluation into the Linux kernel via eBPF, using two filter classes — approvers (statically compiled from rules) and discarders (dynamically learned at runtime).

Why eBPF (vs. inotify / auditd / periodic scans)

Legacy Linux FIM approaches each failed on a different axis:

  • Periodic filesystem scans — miss tamper-then-revert attacks between scan intervals; report that a file changed but not how/why/who. Fundamentally snapshot-based, not event-based.
  • inotify — event-based, but lacks system-level context: no process/container correlation for the file event.
  • auditd — more comprehensive context, but "high performance overhead" and "scalability struggles under heavy system loads."

eBPF provides the combination the team needed: real-time, process + container context, and safe in-kernel execution (verifier-gated, no custom kernel modules).

The scale problem

Once eBPF gave per-syscall observability, the event stream itself became the bottleneck:

  • ~10B file-related events / minute across Datadog's infrastructure on a normal Friday afternoon.
  • ~5 KB per serialized event (file path + process + container + other metadata).
  • Naive upstream forwarding: multiple TB/s of outbound network traffic, of which "most wouldn't match any detection rules and would ultimately be discarded."
  • Per-host outbound: hundreds of Mbit/s sustained just for security monitoring.
  • Some sensitive workloads: up to 5,000 security-relevant syscalls per second — not background noise, each needs analysis with zero drops.

The agent struggled on some hosts: "the stream of events flowing through the ring buffer could outpace the Agent's ability to process them, leading to dropped events. Every missed event meant a potential blind spot."

The two-stage evaluation model

Architectural shift: move as much rule evaluation as possible into the eBPF programs themselves, so only events that might match a rule cross the ring buffer into user space.

  1. In-kernel filtering — lightweight, bounded by eBPF verifier limits (especially on older kernels). Cheap per-event decisions.
  2. User-space evaluation — full rule engine: correlations, richer context, logic that would be unsafe or impossible in the kernel.

This is the core design pattern: a cheap first pass protects the expensive second pass from drowning.

Approvers: compile-time static filters

Approvers are generated at rule compile time. The rule engine analyses each rule's conditions and extracts concrete values that deserve a deeper look in user space, then pushes them into the kernel via eBPF maps for fast O(1) lookup per event.

Example:

open.file.path == "/etc/passwd" && open.flags & O_CREAT > 0

/etc/passwd is a concrete, bounded value → approver. Any open whose path is in the approver map is forwarded; everything else is candidate for dropping.

Problem: "when you're managing hundreds [of rules], each with different fields and conditions across different syscalls, it quickly becomes a complex optimization problem." And some rules don't yield any static value — e.g. open.file.path == "/etc/*" — the wildcard gives no concrete extractable filter.

Discarders: runtime-learned negative filters

Discarders are the dynamic counterpart. The rule engine decides at runtime: given a specific observed value, could it ever match any rule in the active ruleset? If no → mark it as a discarder, push it into a kernel LRU eBPF map for future in-kernel dropping.

Example: for a ruleset targeting /etc/*, an observed /tmp file-access will never match → /tmp becomes a discarder. Subsequent /tmp events are filtered in-kernel without ever entering the ring buffer.

LRU is load-bearing: the discarder map has bounded memory, so least-recently-used negatives age out — the kernel only carries the hottest noise filters at any moment.

"Figuring out what can safely be discarded requires non-trivial algorithms" — the rule engine has to reason about rule coverage over an open-ended value space.

Reported outcome

  • ~94% of events filtered directly in the kernel via approvers + discarders combined.
  • Down from ~10B events/minute at the source to ~1M events/minute forwarded upstream.
  • "No dropped events" — user-space agent keeps up even with the ring buffer under load.
  • "Dramatically lower CPU usage."

Architectural takeaways

  1. eBPF as a programmable data plane for host observability — kernel-safe, process+container-aware, replacing both inotify (too little context) and auditd (too heavy).
  2. Edge filtering for volume problems: match at the producer, not the consumer. Parallels observability pipeline design — a security agent evaluating rules locally is structurally the same move as a metrics pipeline dropping per-instance labels in-transit (concepts/streaming-aggregation).
  3. Two-stage evaluation — a cheap fast path that protects a rich slow path is a general shape (patterns/two-stage-evaluation); here bounded by eBPF verifier constraints on the kernel side.
  4. Approvers + discarders as a named dual: static positive filters compiled from rules + dynamic negative filters learned from traffic. Together the per-event filter approaches the decision boundary that the full rule engine would draw.
  5. Control plane / data plane separation — the rule engine (user-space, rich state, compiles rules, learns discarders) is the control plane; the eBPF programs + maps are the data plane (concepts/control-plane-data-plane-separation). Static + dynamic filter maps are the contract between them.

Caveats

  • eBPF verifier limits — especially on older kernels. The post explicitly notes this is why complex logic can't move into the kernel, even when it'd be cheaper per-event.
  • Approver/discarder correctness is a rule-engine invariant — a bug that discards a would-match event is a security blind spot. The post acknowledges the non-trivial algorithms but doesn't describe validation.
  • Sensitive-workload numbers (~5K syscalls/sec) hint that tail hosts still require care; the 94% filter rate is fleet average.
  • Datadog's public tier slot here is not in AGENTS.md's formal list; treated as Tier-3-equivalent and on-topic per scope filter (distributed-systems internals, production-measured scaling trade-offs, security-agent architecture).

Original

Last updated · 200 distilled / 1,178 read