Script to collect thread dump (Jstack)

Jstack is a command line tool that helps to capture the thread dump of the java process. Using the thread dump would help us to identify the number of threads used by the process and in which state that is residing currently.

Below are the thread state for reference

• RUNNABLE
• BLOCKED
• WAITING
• TIMED_WAITING
• TERMINATED

Command:

jstack[option] pid

Who can run the command?

jstack utility has to be run as the user, Who owns the process

In the below Datanode process, We can identify the process id and the owner

process ID = 16121

Owner = hdfs

hdfs       16121   16113  0 Nov12 ?        00:04:58 /usr/java/jdk1.8.0_141/bin/java -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/HDFS -Dhadoop.home.dir=/tmp/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode

To take a thread dump on the above process, We need to switch as hdfs user and run the command

sudo -u hdfs jstack 16121

The above command will give you the thread info as below.

   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d60841a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

Running the command as a different user or root user will provide you with inappropriate/Wrong output as below.

Thread 654947: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)

How many thread dumps do we need to collect?

It is essential to collect thread dump multiple times, To identify the thread activities and the state changes. If we are seeing a thread that has been blocked or waiting for more seconds, Then it shows there is some trouble in the process.

Now, How to automate the thread dump collection using a script

Let’s say, If we need to collect thread dump till the server crashes with an interval of 5 sec

Below the script, It will run the thread dump every 5 sec with an infinite loop. We can stop the script once you are done collecting the thread dump.

do 
sudo -u <user> /usr/java/bin/jstack >> /<Path>$(hostname).jstack.$i; 
sleep 5s; 
done

The above scripts can be configured based on our needs. We can change the sleep time to either 10 sec or 15 sec.

NOTE: Make sure you have Java & Jstack utility available to run the above commands

Please start a discussion if you have different ideas for collecting Jstack