Skip to content

CONCEPT Cited by 1 source

Flamegraph profiling

Definition

Flamegraph profiling is the practice of sampling a running process's stack at high frequency and rendering the aggregated stacks as a flamegraph — a horizontal-axis-is-samples, vertical-axis-is-stack-depth visualisation where the width of each frame is proportional to the time spent with that frame on the stack. Invented / popularised by Brendan Gregg.

The point is not the image per se but the rank-ordering of where CPU goes. Bugs of the "something is burning a core but we don't know what" shape — canonically CPU busy-loop incidents — are diagnosed almost entirely by reading the top of the flamegraph.

Async-state-machine signature

In async-Rust / Tokio stacks, the tell-tale signature of a spurious-wakeup busy-loop is that the flamegraph is dominated by infrastructure, not business logic:

  • tracing::Subscriber::enter / exit frames (span enter/exit is supposed to be very fast)
  • Tokio poll frames with no meaningful work beneath them
  • libc syscalls that return almost immediately without doing I/O

As Fly.io describes:

"If the mere act of entering a span in a Tokio stack is chewing up a significant amount of CPU, something has gone haywire: the actual code being traced must be doing next to nothing."

The inversion — infrastructure in the hot path, business logic invisible — is the fingerprint.

Using the type signature

Modern languages with strong generic monomorphisation (Rust, Scala, templated C++) emit flamegraph frames with the fully-qualified type of each stack frame. For async Rust that often means the whole nested-Future type shows up as a single frame. Fly.io's 2025-02 case:

&mut fp_io::copy::Duplex<&mut fp_io::reusable_reader::ReusableReader<
  fp_tcp::peek::PeekableReader<
    tokio_rustls::server::TlsStream<
      fp_tcp_metered::MeteredIo<
        fp_tcp::peek::PeekableReader<
          fp_tcp::permitted::PermittedTcpStream>>>>>,
  connect::conn::Conn<tokio::net::tcp::stream::TcpStream>>

Reading this top-to-bottom gives the exact wrapper chain around the bug — and since Fly's own wrappers (Duplex, ReusableReader, PeekableReader, MeteredIo, PermittedTcpStream) could be audited for recent changes + reproducibility, the suspect narrowed to one foreign layer: tokio-rustls::TlsStream.

Seen in

  • sources/2025-02-26-flyio-taming-a-voracious-rust-proxy — Pavel on Fly.io's proxy team pulled a flamegraph from an angry fly-proxy; tracing::Subscriber dominance was the "something is wrong" indicator; the Future type signature pointed at tokio_rustls::server::TlsStream as the guilty layer. Textbook flamegraph-as-diagnostic.
Last updated · 200 distilled / 1,178 read