SYSTEM Cited by 1 source

Micrometer Tracing

Micrometer Tracing is Spring's vendor-neutral tracing abstraction (successor to Spring Cloud Sleuth). It provides a Tracer + Span + @Observed + SpanAspect API and bridges to concrete tracer implementations via io.micrometer.tracing.brave.bridge.BraveSpan (Brave) and equivalent OpenTelemetry bridges.

Relevant classes for the Netflix incident

• SpanAspect — AspectJ aspect around @NewSpan / @ContinueSpan -annotated methods.
• ImperativeMethodInvocationProcessor.proceedUnderSynchronousSpan — invoked by the aspect. Executes the target method inside a started span, then calls span.end() (which on the Brave bridge delegates to RealSpan.finish). The span-end call is wrapped in a synchronized block.
• AbstractMethodInvocationProcessor.after — final call on the way out of the aspect; calls span.end().

Why the synchronized block matters

Under Java 21 virtual threads, a blocking call made while holding a monitor (inside synchronized) causes the VT to pin to its carrier OS thread. RealSpan.finish transitively calls CountBoundedQueue.offer in systems/zipkin-reporter, which takes a ReentrantLock. That lock acquisition inside the Micrometer Tracing synchronized block is what pins the VT.

Per the thread dump Netflix captured:

java.util.concurrent.locks.ReentrantLock.lock
zipkin2.reporter.internal.CountBoundedQueue.offer
zipkin2.reporter.internal.AsyncReporter$BoundedAsyncReporter.report
zipkin2.reporter.brave.AsyncZipkinSpanHandler.end
brave.internal.handler.NoopAwareSpanHandler.end
brave.internal.recorder.PendingSpans.finish
brave.RealSpan.finish(RealSpan.java:134)
io.micrometer.tracing.brave.bridge.BraveSpan.end
io.micrometer.tracing.annotation.AbstractMethodInvocationProcessor.after
io.micrometer.tracing.annotation.ImperativeMethodInvocationProcessor.proceedUnderSynchronousSpan
io.micrometer.tracing.annotation.ImperativeMethodInvocationProcessor.process
io.micrometer.tracing.annotation.SpanAspect.newSpanMethod

The bottom of the stack is the AspectJ invocation; the middle is the Micrometer synchronized-guarded span end; the top is the Zipkin-Reporter lock acquisition that pins the VT.

Seen in

• sources/2024-07-29-netflix-java-21-virtual-threads-dude-wheres-my-lock — The canonical wiki instance of Micrometer Tracing's synchronized-guarded span-end path composing with systems/zipkin-reporter's ReentrantLock to produce fleet-wide VT-pinning hangs under Java 21.

Related

• systems/zipkin-reporter — The downstream library whose lock is the contention point.
• systems/spring-boot — Ships Micrometer Tracing as the default observability wiring.
• concepts/virtual-thread-pinning — The failure class triggered by the synchronized wrap.
• companies/netflix — Incident adopter.