Skip to content

PATTERN Cited by 1 source

Ad-hoc bpftrace profiler

Ad-hoc bpftrace profiler = the specific instance of concepts/ad-hoc-profiler where the DSL is bpftrace and the orchestrator compiles + schedules + deploys the user-authored script as if it were a first-class in-tree profiler.

It is the escape hatch that a profiler orchestrator exposes so engineers can answer narrow diagnostic questions — "latency of this one C++ function on this call path", "count of this syscall in this service" — in hours instead of weeks.

The shape

  1. Engineer writes a small bpftrace script specifying the kernel hook (kprobe / uprobe / tracepoint / perf event), the data to capture, and optional output filtering.
  2. Script is committed to the source repo alongside first-class profilers.
  3. Orchestrator picks it up — compiles the script to eBPF, deploys it under the same safety rules (concurrency, PMU-counter coordination, rate limits, symbolization) applied to every other profiler.
  4. Output routes to the same UI as any other profiler — flame-graph, time-series, metadata-attached records — no custom ingestion.
  5. Hours, not weeks, to run on the fleet.

Canonical Meta instance

"However, engineers can write a single bpftrace script (a simple language/tool that allows you to easily write eBPF programs) and tell Strobelight to run it like it would any other profiler. An engineer that really cares about the latency of a particular C++ function, for example, could write up a little bpftrace script, commit it, and have Strobelight run it on any number of hosts throughout Meta's fleet — all within a matter of hours, if needed."

— Meta Engineering, 2025-01-21 Strobelight post (Source: sources/2025-03-07-meta-strobelight-a-profiling-service-built-on-open-source-technology)

The Strobelight orchestrator treats the bpftrace script as a first-class profiler: full safety rules apply, the output is symbolized via the same service, the UI renders the data the same way.

Why bpftrace specifically

bpftrace (github.com/bpftrace/bpftrace) is a high-level tracing DSL modelled on DTrace + SystemTap. It compiles to eBPF programs + a user-space consumer. Its load- bearing properties for ad-hoc-profiler use:

  • Compact. A useful profiler can be written in 10-30 lines.
  • Fast to learn. Syntax is close enough to awk / DTrace that an engineer with kernel familiarity can ramp in an afternoon.
  • eBPF-native. Compiles to verifier-gated kernel programs, so the orchestrator's sandbox already applies.
  • Standard library. Has primitives for kprobe/uprobe attachment, histograms, counts, stack captures — the common profiler shapes.

Why this works (vs "just add a profiler to the platform")

The platform-side profiler path is expensive:

  1. Code change to the orchestrator.
  2. Review + rollout cycle.
  3. Test coverage + safety review.

For narrow, one-off questions that cost is over-the-top. The ad-hoc bpftrace path inherits the orchestrator's safety posture for free — the script is still kernel-verifier-gated, still subject to concurrency + rate rules — but skips the first-party-profiler review cycle.

Safety interplay

The orchestrator's safety story must cover ad-hoc profilers too — a bpftrace script that accidentally attaches to a hot kprobe at 10 kHz can still degrade the host. Meta's post explicitly acknowledges this:

"If all of this sounds powerfully dangerous, that's because it is. However, Strobelight has several safeguards in place to prevent users from causing performance degradation for the targeted workloads and retention issues for the databases Strobelight writes to."

(Source: sources/2025-03-07-meta-strobelight-a-profiling-service-built-on-open-source-technology)

When to use this pattern

  • You have a centralised profiler orchestrator already.
  • Adding new first-class profilers has a multi-week lead time.
  • Engineers frequently have narrow one-off diagnostic questions where a bpftrace script is enough.
  • The orchestrator's sandbox + safety rules cover eBPF / bpftrace adequately.

When not to use it

  • No centralised orchestrator — running bpftrace ad-hoc on prod hosts without the safety rules is the mode this pattern obsoletes.
  • Question is wide enough to deserve a first-class profiler — commit the DSL-script-equivalent as platform code.

Seen in

Last updated · 550 distilled / 1,221 read