Skip to content

CONCEPT Cited by 1 source

Delayed symbolization

Delayed symbolization is the practice of deferring the address-to-symbol resolution step until after the profile collection step on the profiled host has completed — typically by writing raw (pc, stack_addr[]) records to disk and shipping them off-host for later resolution by a centralised symbolization service.

It is the architectural response to two structural problems with on-host symbolization at fleet scale (Source: sources/2025-03-07-meta-strobelight-a-profiling-service-built-on-open-source-technology):

  1. DWARF is megabytes to gigabytes per binary. Parsing it on the profiled host would "cause memory issues for the host's workloads" even with optimal caching.
  2. Producer-consumer decoupling. If the user-space consumer that symbolizes in-line can't keep up with the eBPF kernel producer's sample rate, samples get dropped. Quote:

"Strobelight also delays symbolization until after profiling and stores raw data to disk to prevent memory thrash on the host. This has the added benefit of not letting the consumer impact the producer — meaning if Strobelight's user space code can't handle the speed at which the eBPF kernel code is producing samples (because it's spending time symbolizing or doing some other processing) it results in dropped samples."

(Source: sources/2025-03-07-meta-strobelight-a-profiling-service-built-on-open-source-technology)

How it works (Strobelight canonical form)

  1. eBPF captures raw stacks (addresses only) using frame pointers.
  2. User-space writes raw samples to a buffer → local disk.
  3. Later, stacks are shipped to a central symbolization service.
  4. The service returns (function, file, line, type, inline-sites) tuples — serving the symbolized profile into Scuba or the flame-graph UI.
  5. The service has pre-parsed DWARF for every Meta production binary using gsym + blazesym, so symbolization is a cheap DB lookup — not a DWARF parse.

Why the naive "symbolize inline" shape breaks

  • Memory pressure. DWARF blobs compete with the workload for host RAM.
  • CPU competition. DWARF parsing spends CPU the workload would otherwise get — defeats the profiler's "invisible" goal.
  • Variable latency. Symbol-table misses stall in-line consumers and cause ring-buffer overflow → dropped samples.
  • Per-host cache churn. Binaries change more often than individual profiles; per-host caching is hot-reload-heavy.

Design shape

  • Raw on-host sample records are addresses + minimal metadata (build ID, PID, cgroup) — small and homogeneous.
  • Pre-indexed symbolization facts live in a single database keyed by (build_id, pc) → (function, file, line, …).
  • Off-host lookup scales horizontally; profiles don't block on it.
  • Bonus: inline-site info the compiler preserves in DWARF (which cheap runtime unwinds lose) is reattached by the service.

Seen in

Last updated · 550 distilled / 1,221 read