Skip to content

NETFLIX 2024-07-29 Tier 1

Read original ↗

Netflix — Java 21 Virtual Threads: Dude, Where's My Lock?

Summary

A Netflix TechBlog post (2024-07-29, Tier 1; Vadim Filanovsky, Mike Huang, Danny Thomas, Martin Chalupa of Netflix's Performance Engineering + JVM Ecosystem teams) documenting a production deadlock on Java 21 + Spring Boot 3 + embedded Tomcat microservices that had just enabled virtual threads. Affected instances stopped serving traffic despite a healthy JVM, accumulated an ever-growing closeWait socket count, and showed thousands of "blank" virtual threads in jcmd-generated dumps. Root cause: virtual-thread pinning — four VTs blocked on a ReentrantLock inside a synchronized block (brave.RealSpan.finish called from Micrometer Tracing's SpanAspect), each pinning one of the 4 carrier threads in the VT fork-join pool on a 4-vCPU instance. No OS thread was available to run any other VT, including the incoming Tomcat request-VTs — Tomcat accepted connections, queued them as unmounted VTs, and bled socket file descriptors.

The diagnostic arc is a textbook example of patterns/diagnose-via-heap-dump-lock-introspection: thread dumps were inadequate (Java 21's jcmd Thread.dump_to_file omits lock-owner / waiting metadata); a heap dump under Eclipse MAT was needed to reverse-engineer the AbstractQueuedSynchronizer state and identify both the unowned ReentrantLock and the condition queue. The lock holder had entered Condition.awaitNanos (releasing the lock), but when it tried to reacquire after the wait, the writer-preference queueing protocol blocked it behind the 5 waiting threads (4 pinned VTs + 1 queued writer). Classic self-starvation — but invisible without the heap dump. Fix: Netflix is waiting on JDK changes (both the pinning-during-synchronized removal and richer thread-dump metadata are on the JDK roadmap) — canonical patterns/upstream-the-fix instance at the language-runtime layer.

Key takeaways

  1. Virtual thread model recap: a VT is "a task that is scheduled to a fork-join thread pool… When a virtual thread enters a blocking call, like waiting for a Future, it relinquishes the OS thread it occupies and simply remains in memory until it is ready to resume." The underlying OS thread is the carrier thread; a VT is "mounted" while executing and "unmounted" while waiting. The mount/unmount primitive is continuations. Reference: JEP 444.

  2. Tomcat's blocking model doesn't change with VTs — it just substrates on them. "In our environment, we utilize a blocking model for Tomcat, which in effect holds a worker thread for the lifespan of a request. By enabling virtual threads, Tomcat switches to virtual execution. Each incoming request creates a new virtual thread that is simply scheduled as a task on a Virtual Thread Executor." Canonical patterns/blocking-model-per-request-tomcat instance — VT adoption is the minimal-code-change path to the blocking-per-request shape at higher concurrency.

  3. Pinning is the hidden failure mode. Per the JDK docs: "a VT will be pinned to the underlying OS thread if it performs a blocking operation while inside a synchronized block or method." The VT stays mounted on its carrier OS thread through the block — defeating the entire point of virtual execution. Canonical virtual-thread-pinning instance on the wiki.

  4. 4 vCPUs → 4 carrier threads → 4-deep pinning kills the fleet. The app runs on a 4-vCPU instance; the fork-join pool that underpins VT execution has 4 OS threads. Four VTs pinned inside synchronized blocks == 100% of the carrier-thread pool exhausted. "No other virtual thread can make any progress." Tomcat keeps accepting connections, keeps creating request-VTs — but those VTs can't mount because every carrier thread is held. They remain blank tasks in the queue.

  5. closeWait socket accumulation is the external symptom. "Sockets remaining in closeWait state indicate that the remote peer closed the socket, but it was never closed on the local instance, presumably because the application failed to do so." Tomcat accepted sockets, created request-VTs, passed them to the executor — the VTs never got to run the close path. Canonical closeWait accumulation as a leading indicator of application-layer hangs.

  6. Thousands of "blank" virtual threads in the dump. "These are the VTs (virtual threads) for which a thread object is created, but has not started running, and as such, has no stack trace. In fact, there were approximately the same number of blank VTs as the number of sockets in closeWait state." A 1:1 correspondence between queued-but-unmounted VTs and leaked sockets — a load-bearing architectural datum for anyone debugging VT fleet hangs.

  7. jstack doesn't see virtual thread stacks — use jcmd Thread.dump_to_file. "We knew that virtual thread call stacks do not show up in jstack-generated thread dumps. To obtain a more complete thread dump containing the state of the virtual threads, we used the jcmd Thread.dump_to_file command instead." Canonical jcmd thread-dump guidance.

  8. Java 21 jcmd thread dumps omit lock metadata — a named JDK limitation. "Usually a thread dump indicates who holds the lock with either - locked <0x…> (at …) or Locked ownable synchronizers, but neither of these show up in our thread dumps. As a matter of fact, no locking/parking/waiting information is included in the jcmd-generated thread dumps. This is a limitation in Java 21 and will be addressed in the future releases." The primary JVM observability tool silently drops the data you need to diagnose lock-based hangs.

  9. The pinning path: all four pinned VTs enter synchronized(this) in brave.RealSpan.finish, then call zipkin2.reporter.internal.CountBoundedQueue.offer which calls ReentrantLock.lock() on a bounded-queue lock. Offered by Micrometer Tracing's ImperativeMethodInvocationProcessor.proceedUnderSynchronousSpan wrapping a span finish inside synchronized. All four block trying to acquire the same ReentrantLock — but pinned to their carrier threads, not parked. 100% carrier-thread exhaustion.

  10. The non-pinned fifth waiter: thread dump also shows a 5th virtual thread (#119516) on the same ReentrantLock but reached via brave.RealScopedSpan.finish — which does not go through a synchronized block. This VT is properly parked (unmounted), not pinned. Confirms the bug is specifically the four synchronized-gated paths, not the ReentrantLock itself.

  11. The 6th waiter — the AsyncReporter platform thread. A regular (non-virtual) thread named AsyncReporter: 

    zipkin2.reporter.internal.AsyncReporter$Flusher.run
→ AsyncReporter$BoundedAsyncReporter.flush
→ CountBoundedQueue.drainTo
→ AQS$ConditionObject.awaitNanos  ← here
→ AQS.acquire (post-await reacquire)
    Stack trace read carefully: "this thread seems to be blocked within the internal acquire() method after completing the wait. In other words, this calling thread owned the lock upon entering awaitNanos(). We know the lock was explicitly acquired here. However, by the time the wait completed, it could not reacquire the lock." The flusher was the owner, released on awaitNanos, timed out, and now can't get back in.

  12. 5 virtual threads + 1 platform thread all waiting for the same lock, and no thread shows as holder. "There's still no information on who owns the lock. As there's nothing more we can glean from the thread dump, our next logical step is to peek into the heap dump and introspect the state of the lock." The thread dump is exhausted as an evidence source.

  13. Heap-dump introspection via Eclipse MAT on the AsyncReporter thread's stack objects identifies the ReentrantLock + its AbstractQueuedSynchronizer state. Canonical patterns/diagnose-via-heap-dump-lock-introspection instance: when the thread dump can't tell you who owns a lock, the heap dump can — the lock object is on-heap, its state (reader count / writer identity), exclusiveOwnerThread, and condition queue are all traversable fields. Post-quote: "we reverse-engineered enough of it to match against what we see in the heap dump."

  14. The self-starvation mechanism. The AsyncReporter flusher released the lock by entering awaitNanos. When the wait timed out, the flusher re-queued for the lock — but AQS's FIFO write-preference semantics placed it behind the four pinned VTs and the fifth (non-pinned) VT already in the queue. None of the pinned VTs can ever acquire the lock — because they're pinned, and the carrier threads they need to run on are held by themselves. None of the carrier threads can be freed because the VTs holding them are waiting on this lock. The one thread that could release the lock (the flusher) is also waiting on this lock (behind the others in the queue). Fully starved — with no process owner.

  15. Fix path: upstream JDK changes. Two JDK-level changes would have avoided the bug: (a) pinning-during-synchronized is being removed — VTs will unmount across synchronized the same way they do across ReentrantLock; (b) richer thread-dump metadata (lock owner + condition queue) will return in jcmd dumps. Netflix reports the bug + works around it in-application (typically by switching synchronized blocks in the tracing hot path to ReentrantLock, or disabling VTs selectively for request handlers that transit through tracing), but the structural fix is in the language runtime. Canonical patterns/upstream-the-fix instance at the language-runtime layer (distinct from library-layer upstream fixes like Fly.io's parking_lot PR #466 or Cloudflare's V8 / Go / rustls fixes).

Systems

  • systems/java-21-virtual-threads — Java 21's VT implementation (JEP 444). VT-as-fork-join-task model, carrier-thread mount/unmount via continuations, pinning during synchronized.
  • systems/zipkin-reporter — zipkin2.reporter.*, specifically BoundedAsyncReporter + CountBoundedQueue. Uses a ReentrantLock + Condition for producer-consumer handoff. The structural lock contention point.
  • systems/embedded-tomcat — Spring Boot's default servlet container. Virtual-thread executor integration via VirtualThreadExecutor, registered in AbstractEndpoint.
  • systems/spring-boot — The Spring Boot 3 stack that ships Micrometer Tracing, enables VTs via a single config, and wires Brave via io.micrometer.tracing.brave.bridge.
  • systems/micrometer-tracing — Spring's observability abstraction layer. SpanAspect + ImperativeMethodInvocationProcessor pair. Wraps span-finish logic in synchronized on the Brave bridge path.

Concepts

  • concepts/virtual-thread — Lightweight JVM threads multiplexed over OS carrier threads via a fork-join pool.
  • concepts/virtual-thread-pinning — Root cause class. When a VT blocks inside a synchronized block or method (or certain native-code regions), it pins itself to its carrier OS thread. Canonical wiki first instance.
  • concepts/carrier-thread — The OS-backed worker thread in the fork-join pool that a VT mounts onto for execution.
  • concepts/fork-join-pool — The JVM-internal scheduler pool for VTs; by default sized to vCPU count. Canonical wiki instance.
  • concepts/closewait-socket-state — Observable external symptom of application-layer hangs: remote peer closed, local process never called close().
  • concepts/jcmd-thread-dump — Diagnostic tool that does see VT stacks (unlike jstack) but doesn't include lock-owner/parking metadata in Java 21.
  • concepts/heap-dump-lock-introspection — Debugging technique: when the thread dump can't tell you who owns a lock, traverse the lock object's fields on the heap.
  • concepts/deadlock-vs-lock-contention — This incident is a starvation deadlock, not a true cycle; every waiter is waiting on the same lock with no cyclic owner. Extended with the VT-pinning axis.

Patterns

Operational numbers

  • Instance shape: 4 vCPUs ⇒ 4 carrier threads in the VT fork-join pool ⇒ 4-deep pinning fully exhausts the pool.
  • Pinned virtual threads: 4 (all in brave.RealSpan.finish → CountBoundedQueue.offer).
  • Additional waiters on the same ReentrantLock: 1 non-pinned VT (RealScopedSpan.finish path) + 1 platform thread (AsyncReporter flusher, lock owner pre-awaitNanos).
  • Total waiters: 6 (5 VT + 1 platform), 0 current owners.
  • closeWait socket count grows unboundedly; 1:1 correspondence with blank VT count in the dump.
  • Stack: Java 21 + Spring Boot 3 + embedded Tomcat (blocking executor) + Micrometer Tracing + Brave + Zipkin Reporter.

Caveats

  • Tier-1 Netflix TechBlog post; architectural density is near-100%. No marketing voice.
  • Netflix doesn't disclose which services hit this or how many instances were affected. The symptom pattern (intermittent timeouts, hung instances, closeWait pile-up) is described as affecting multiple services in the migration.
  • The post doesn't quantify the pre-fix impact (request-loss percentage, customer-visible incident count) — it's a debugging narrative more than a post-incident retrospective.
  • The bug is not a true deadlock — the AsyncReporter flusher would eventually time out of its next awaitNanos and try again. But because its retry gets queued behind the pinned VTs (which will never run), the system is effectively deadlocked from an operator's standpoint. This is a starvation deadlock via queue position. Worth distinguishing from classic cyclic deadlock.
  • The fix Netflix actually deployed in-application isn't explicitly disclosed in the post. Obvious candidates: replace synchronized in the tracing hot path with ReentrantLock (they don't pin); pin the tracing library version to a release where RealSpan.finish doesn't go through synchronized; or disable VTs for request handlers that hit this code path.
  • The JDK fix (removing pinning-during-synchronized) is referenced as "in future releases" but the post (2024-07-29) doesn't name a target JDK version. Subsequent JEPs + JDK 24 early-access notes have addressed this.
  • The heap-dump introspection technique Netflix used is knowledge-dense ("we don't claim to fully understand the inner workings of it, we reverse-engineered enough of it to match against what we see in the heap dump"). Not a substitute for better tooling — it's the fallback when tooling fails.
  • Pinning can also occur on native-code blocking paths beyond synchronized; the post is specifically about the synchronized-block class.
  • No cross-vendor comparison (Kotlin coroutines / Go goroutines / .NET async — each has its own pinning story).

Source

Last updated · 319 distilled / 1,201 read