Skip to content

PATTERN Cited by 1 source

Two-stage evaluation

Two-stage evaluation is the pattern of splitting a per-event match/decision pipeline into:

  1. A cheap, constrained first stage that resolves the common case at very low cost, and
  2. A rich, expensive second stage that handles only the events the first stage could not decide.

The first stage exists to protect the second stage from being drowned. The split is driven by a cost asymmetry (CPU, context switch, memory, bandwidth) between where the event is produced and where the full decision logic lives.

Canonical instance: eBPF + user-space rule engine

Datadog's Workload Protection FIM (Source: sources/2025-11-18-datadog-ebpf-fim-filtering):

  • Stage 1 (kernel, eBPF programs): O(1) map lookups against approver / discarder maps (patterns/approver-discarder-filter). Bounded by eBPF verifier limits — no loops, bounded instructions, more restrictive on older kernels. Resolves ~94% of events.
  • Stage 2 (user-space Agent): Full rule engine with correlations, cross-event state, arbitrary logic. Runs only on the ~6% that passed stage 1. Matches are then forwarded to the backend; everything else is dropped.

Without stage 1 the ring buffer outpaces the Agent on busy hosts (~5K syscalls/sec) → dropped events → security blind spots. Without stage 2 the rule engine's expressiveness is capped by the verifier → false-positive explosion. Both are required.

Structural properties

  • Stage 1 must be conservative. It is allowed to forward false positives (they'll be rejected in stage 2); it is not allowed to drop a would-match event. This asymmetry is what makes stage 2 a safety net rather than a duplicate of stage 1.
  • Stage 1 state is compiled from the rules in stage 2. A control plane (the rule engine) projects its policy into the data plane (eBPF maps, filter tables, etc.) — see concepts/control-plane-data-plane-separation.
  • Rule changes require stage 1 rebuild. Approvers recompile; discarders must be flushed/relearned.
  • Stage 1 memory is bounded (e.g. LRU maps); ruleset + traffic growth can push beyond that bound, silently reducing the filter rate (not correctness).

When to reach for it

  • Event production rate at the edge exceeds stage 2's budget.
  • Most events are uninteresting to the rule engine (low match rate).
  • The edge has a cheap O(1) filter mechanism available (hash table, Bloom filter, regex trimmed FSA, eBPF map, tcpdump BPF).
  • Correctness of the filter can be proven against the rule set.

Adjacent patterns

  • Bloom-filter-then-verify — generalization outside eBPF; false positives only, verify on hit. Husky's per-fragment patterns/trimmed-automaton-predicate-filter is in the same family.
  • SIMD-scan + scalar-verify — cheap vectorized filter, then scalar confirmation.
  • Sampling + full-capture — production profiling / tracing; different goal but same shape.

Seen in

Last updated · 200 distilled / 1,178 read