Skip to content

Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects

Summary

Cloudflare added on-demand CPU and memory profiling for Workers and Durable Objects, surfaced from the Workers Observability page (or the cf CLI / REST API). A developer requests a time-boxed profile of an actively-running production Worker, inspects it as an interactive flamegraph, and downloads the .pprof for offline analysis. The headline thesis is that "the best way to understand your application is to profile it in production" — local Chrome-DevTools profiling never reproduces real traffic shape or volume. The post doubles as an edge-systems deep dive: profiling a Worker in production requires routing the profiling request to the exact data center / metal / isolate (and, for a Durable Object, the exact live primary actor) that recently ran the chosen version, and the Workers Runtime must drive the V8 profiler without holding the isolate lock for the sampling window, or the profile would freeze the very execution it is trying to observe. Two production case studies (the R2-binding Worker's CPU profile; an internal Worker's OOM heap profile) show the loop end-to-end.

Key takeaways

  1. Profile in production, on-demand, time-boxed. From Workers Observability → Flamegraph, pick CPU or memory + a duration + a deployed version; Capture Profile runs the profiler for that window and renders a flamegraph (rectangle = function call, width = CPU time or memory). CLI equivalent: cf workers versions profile latest --worker-id "$NAME" --duration-ms 5000 --profile-type cpu > worker-cpu.pprof. (Source: this article)
  2. The profiler observes real execution — no fresh isolate is started. "A new isolate is not started to produce a profile, because the goal is to observe real production execution." The direct consequence: a Worker (or version) with little traffic is hard to profile — you must pick a version that is actually deployed and receiving requests. This is a deliberate hot-path fidelity choice.
  3. Flamegraph as target-selector, confirmed by a real CPU win. On the Worker implementing the R2 binding (very high traffic, so "eliminating even the smallest amount of wasted CPU cycles can have a massive impact"), a 50-second CPU profile's table view sorted by Samples surfaced genericR2JsonReplacer at >5% of CPU — recursing into itself. The replacer, already called by JSON.stringify on every node, also walked the tree, so a value nested five levels deep was processed five times. Removing the duplicate walk made the function 2.7× faster. A second candidate: a metrics call heavy enough that one call = ~1% of CPU; memoizing the first result removed the second call. Canonical measurement-driven loop — profile picks the function, the fix is structural.
  4. Heap profiling found an OOM root cause metrics couldn't. An internal Worker sat at P999 memory ~133 MB against the 128 MB limit, evicted frequently with "Exceeded Memory". Errors + metrics pointed at memory but not which code. A production heap profile opened in pprof showed Prometheus instrumentation accounting for ~66.7% of allocations — code believed disabled but only partially disabled, so it instrumented many code paths and paid the memory cost "even though the collected data never actually left the Worker." Removing the path:
Percentile Before (MB) After (MB)
P50 70 54
P90 94 79
P99 113 97
P999 133 118

→ ~10 MB of P999 headroom below the 128 MB limit. 5. Routing a profiling request to the edge is the hard part. Workers are placed near the client, replicated across data centers and metals, and a hot Worker can be replicated multiple times on one metal; Durable Objects add dynamic placement + movement. So before sampling, the system answers: which version? which data center ran it recently? is that isolate still loaded there? does the isolate belong exclusively to the requesting account? (and for a DO) where is the exact live primary actor? The request URL pins both script name and version (.../versions/latest/profile with {"duration_ms":...,"profile_type":...}). 6. Profiling without stopping the Worker = minimal isolate-lock hold. For a CPU profile the Workers Runtime: (1) acquire the isolate + its lock, (2) create a V8 CPU profiler sampling at 1 ms, (3) release the lock so normal requests run, (4) wait the requested duration, (5) reacquire the lock + stop profiling, (6) release + serialize the result outside the critical section. "Holding the lock through the duration of the profile would prevent JavaScript execution from continuing, which would make the profile useless." A clean control-vs-data-plane discipline at the isolate-lock altitude: lock only for lifecycle, never for the sampling window. 7. Durable Objects profile a named actor anywhere in the world. Regular Workers are stateless — "we pick a running isolate of the requested Worker at random." A DO is stateful and named, so you choose which object by name; the runtime routes the profiling request to the metal that owns that specific actor and profiles that specific isolate. 8. Enable source maps or read obfuscated frames. TypeScript Workers need source maps enabled or the flamegraph shows mangled function names. 9. Two named limitations motivate continuous profiling next. (a) You must explicitly start a session, so you can miss the window in which the Worker misbehaved; (b) the memory profiler only shows allocations during the profiling window, so start-up allocation is invisible. Cloudflare is building continuous profiling (always-on sampling, explore after the fact) to cover rare events.

Systems / concepts / patterns extracted

  • Systems: Cloudflare Workers (the V8-isolate compute being profiled; isolate-lock lifecycle), Durable Objects (named stateful actors, routed-by-name profiling of a specific isolate), V8 (the V8 CPU profiler + heap profiler under the runtime), cf CLI (cf workers versions profile), workerd (the open-source runtime embodying the isolate-lock model). pprof named as the heap-profile analysis tool (prose only — single mention, not minted).
  • Concepts: flamegraph profiling (CPU + memory, interactive + table view, width = resource share), observability (profiling as the layer that logs/metrics can't reach — which function, which allocation), hot path (R2-binding Worker; profile-in-production fidelity; needs real traffic), control/data-plane separation (hold the isolate lock only for profiler lifecycle, not the sampling window), working-set memory / allocation (heap profile = per-function allocation share; 128 MB isolate limit; OOM).
  • Patterns: measurement-driven micro-optimization (profile → pick by resource share → fix → re-measure; genericR2JsonReplacer 2.7× + the metrics memoization).
  • Tagged, not minted (taxonomy gate): on-demand-profiling, profiling-request-routed-to-the-edge, lock-only-for-profiler-lifecycle, profile-a-named-actor, continuous-profiling (named as the next step, not yet shipped) — all article-specific or single-source named ideas, recorded as prose + tags; candidates for Lint promotion at ≥3 citations.

Operational numbers

  • V8 CPU profiler sampling interval: 1 ms.
  • R2-binding Worker: 50 s CPU profile; genericR2JsonReplacer >5% CPU, made 2.7× faster; a single metrics call ~1% of CPU.
  • OOM Worker: Prometheus code ~66.7% of allocations; P999 memory 133 → 118 MB against the 128 MB isolate limit; full before/after table above.
  • Example CLI duration: --duration-ms 5000 (5 s).

Caveats

  • Low-traffic Workers / versions are hard to profile — the profiler observes real execution and starts no fresh isolate.
  • Session must be started explicitly → risk of missing the misbehaving window (the motivation for continuous profiling).
  • Memory profiler only captures allocations during the window → start-up allocation invisible.
  • TypeScript needs source maps or frames are obfuscated.
  • No fleet-wide / always-on coverage yet (continuous profiling is future work); no quantitative overhead numbers for the profiling act itself.

Source

Last updated · 854 distilled / 2,448 read