Skip to content

CONCEPT Cited by 1 source

jcmd thread dump

jcmd Thread.dump_to_file is the Java 21 diagnostic command that captures a thread dump including the state of virtual threads. It's the replacement for classical jstack when working with applications that have VTs enabled — jstack does not see VT stacks.

The jstack limitation

"We were surprised to find that all our thread dumps show a perfectly idle JVM with no clear activity. Reviewing recent changes revealed that these impacted services enabled virtual threads, and we knew that virtual thread call stacks do not show up in jstack-generated thread dumps. To obtain a more complete thread dump containing the state of the virtual threads, we used the jcmd Thread.dump_to_file command instead." (Source: sources/2024-07-29-netflix-java-21-virtual-threads-dude-wheres-my-lock)

If you jstack a VT-enabled hung JVM, you'll see "idle" — the platform threads (carriers) are idle because they're holding parked VTs that aren't in jstack's iteration.

Usage

jcmd <pid> Thread.dump_to_file -format=text /tmp/threads.txt
jcmd <pid> Thread.dump_to_file -format=json /tmp/threads.json

The resulting file includes: - All platform threads (same as jstack). - All VTs that have started at least once, with their mount status and stack traces. - "Blank" VTs — created but never started — as entries without stack traces: e.g. #119821 "" virtual with no frames.

The Java 21 limitation — missing lock metadata

jcmd Thread.dump_to_file in Java 21 omits the lock-owner and parked-on metadata that classical jstack included:

"Usually a thread dump indicates who holds the lock with either - locked <0x…> (at …) or Locked ownable synchronizers, but neither of these show up in our thread dumps. As a matter of fact, no locking/parking/waiting information is included in the jcmd-generated thread dumps. This is a limitation in Java 21 and will be addressed in the future releases." (Source: sources/2024-07-29-netflix-java-21-virtual-threads-dude-wheres-my-lock)

For a bug centered on lock ownership (Netflix's 2024-07-29 incident), this is a critical gap — the primary diagnostic tool for VTs silently drops the data you need. Netflix had to fall back to heap-dump introspection via Eclipse MAT to identify the lock state.

Blank-VT detection

Blank VTs in the jcmd output (#<id> "" virtual with no frames) are a specific diagnostic signal — VTs that have been created but never mounted. High blank-VT counts suggest carrier-thread exhaustion. Netflix saw a 1:1 correspondence between blank-VT count and closeWait socket count during the incident.

Complement: tracing pinned threads

Pair jcmd with -Djdk.tracePinnedThreads=full (or =short) JVM flag — emits log events whenever a VT would pin, with the full stack trace. Good for catching pinning events before they produce a carrier-exhaustion outage.

Seen in

Last updated · 319 distilled / 1,201 read