Jstack and Jcmd: What Are Your Java Threads Even Doing?

Luke Hebert

- 4 minutes read - 848 words

When a Java application slows down or hangs, you need visibility into what each thread is doing right now. The jcmd and jstack utilities give you that visibility directly from the command line, no GUI required. This post walks through the key jcmd commands, a reusable script for collecting correlated thread dumps and CPU data, and the common pitfalls you will hit along the way.

The jcmd utility ships with the JDK and lives in $JAVA_HOME/bin. It sends diagnostic command requests to a running JVM, making it a go-to tool for controlling Java Flight Recordings, troubleshooting, and diagnosing both the JVM and the Java applications running inside it.

Two conditions must be met before you can use it:

  1. You must run it on the same machine where the JVM is running.
  2. Only the user who owns the JVM process can connect to it.

The syntax is straightforward:

jcmd $PID $ARGUMENT

Examples

Example 1: Class histogram

Lists classes taking the most memory at the top, in descending order.

/usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.class_histogram > /tmp/22421_ClassHistogram.txt

Example 2: Generate a heap dump

/usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.heap_dump /tmp/test123.hprof

Example 3: Trigger a garbage collection cycle

/usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.run

Example 4: Generate a thread dump

/usr/jdk64/jdk1.8.0_60/bin/jcmd $PID Thread.print

Example 5: List JVM properties

/usr/jdk64/jdk1.8.0_60/bin/jcmd $PID VM.system_properties

Example 6: Show command-line options and CLASSPATH

/usr/jdk64/jdk1.8.0_60/bin/jcmd $PID VM.command_line
Important

To use certain diagnostic features of jcmd, the -XX:+UnlockDiagnosticVMOptions JVM option must be enabled.

When to Collect Thread Dumps

A common scenario: the JVM process is taking a lot of time processing a request, or it is stuck, slow, or completely hung. In that situation, collecting thread dumps reveals what each thread is actually doing. Sometimes a few threads are locked into CPU-intensive operations that are dragging down response times. Collect the thread dump along with CPU data from the top command.

A few things to keep in mind:

  1. Collect the thread dump when you see the issue (slowness, stuck, hung, etc.).
  2. A single thread dump is rarely useful on its own. Collect at least 5-6 dumps at roughly 10-second intervals, giving you a snapshot over about a minute.
  3. If you suspect specific threads are consuming high CPU, capture the thread dump and the top output at nearly the same time so you can correlate them.

To make this easier, the script below automates the collection. You can also grab it from the MiddlewareMagicDemos repository.

Script to Get Dumps

#!/bin/sh
# Takes the JavaApp PID as an argument.
# Make sure you set JAVA_HOME.
# Creates thread dumps a specified number of times (LOOP) at a set INTERVAL.
# Thread dumps are written to "jcmd_threaddump.out" in the directory where the script is executed.
#
#   Usage:
#          sudo - $user_Who_Owns_The_JavaProcess
#          ./threaddump_cpu_with_cmd.sh <JAVA_APP_PID>
#
#   Example:
#   NameNode PID is "5752" and it is started by user "hdfs", then run:
#
#   su -l hdfs -c "/tmp/threaddump_cpu_with_cmd.sh 5752"
#
#
# Number of times to collect data (total number of thread dumps).
LOOP=10

# Interval in seconds between data points.
INTERVAL=10

# Where to generate the threaddump and top output files.
WHERE_TO_GENERATE_OUTPUT_FILES="/tmp/dump"

# Setting the Java Home. USERS MUST SET THE JAVA_HOME before running this script as follows:
JAVA_HOME=/usr/jdk64/jdk1.8.0_60

echo "Writing CPU data log files to Directory: $WHERE_TO_GENERATE_OUTPUT_FILES"

for ((i=1; i <= $LOOP; i++))
do
    #$JAVA_HOME/bin/jstack -l $1 >> jstack_threaddump.out
    $JAVA_HOME/bin/jcmd $1 Thread.print >> $WHERE_TO_GENERATE_OUTPUT_FILES/jcmd_threaddump.out
    _now=$(date)
    echo "${_now}" >> $WHERE_TO_GENERATE_OUTPUT_FILES/top_highcpu.out
    top -b -n 1 -H -p $1 >> $WHERE_TO_GENERATE_OUTPUT_FILES/top_highcpu.out
    echo "Collected 'top' output and Thread Dump #" $i
    if [ $i -lt $LOOP ]; then
        echo "Sleeping for $INTERVAL seconds."
        sleep $INTERVAL
    fi
done

echo "Get the file 'jcmd_threaddump.out' and 'top_highcpu.out' for analysis."

One-Liner to Get Dumps

for i in $(seq 6); do top -b -H -p <PID> -n 1 > ./dump/cpu.$(date +%s).txt; <path_to_jstack> -l <PID> > ./dump/thread_dump.$(date +%s).txt; sleep 10; done

Inside a Kubernetes pod:

kubectl exec -ti container-1 -n namespace -- bash -c "<path_to_jstack> -l <pid>"

Inside a Docker container:

docker exec -ti container-1 /bin/bash -c "<path_to_jstack> -l <pid>"

How to Analyze the Thread Dump Data

For a deeper walkthrough, see the High CPU Utilization Finding Cause? post.

Common Errors with jcmd

You may hit this error when running jcmd against a process you do not own:

[root@c6401 keys]# /usr/jdk64/jdk1.8.0_60/bin/jcmd 5752 help
5752:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
 at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
 at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
 at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
 at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:147)
 at sun.tools.jcmd.JCmd.main(JCmd.java:131)

This happens because jcmd only allows connections to JVM processes owned by the current user. In the example above, the NameNode process is owned by hdfs, but the command was run as root. The root user does not own the process, so the attach fails.

Processes owned by hdfs:

# su -l hdfs -c "/usr/jdk64/jdk1.8.0_60/bin/jcmd -l"
5752 org.apache.hadoop.hdfs.server.namenode.NameNode
5546 org.apache.hadoop.hdfs.tools.DFSZKFailoverController
5340 org.apache.hadoop.hdfs.server.datanode.DataNode
4991 org.apache.hadoop.hdfs.qjournal.server.JournalNode

Processes owned by root:

[root@c6401 keys]# /usr/jdk64/jdk1.8.0_60/bin/jcmd -l
1893 com.hortonworks.support.tools.server.SupportToolServer
6470 com.hortonworks.smartsense.activity.ActivityAnalyzerFacade
16774 org.apache.ambari.server.controller.AmbariServer
29100 sun.tools.jcmd.JCmd -l
6687 org.apache.zeppelin.server.ZeppelinServer

References