Jstack is a command line tool that helps to capture the thread dump of the java process. Using the thread dump would help us to identify the number of threads used by the process and in which state that is residing currently.
Below are the thread state for reference
Who can run the command?
jstack utility has to be run as the user, Who owns the process
In the below Datanode process, We can identify the process id and the owner
process ID = 16121
Owner = hdfs
hdfs 16121 16113 0 Nov12 ? 00:04:58 /usr/java/jdk1.8.0_141/bin/java -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/HDFS -Dhadoop.home.dir=/tmp/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode
To take a thread dump on the above process, We need to switch as hdfs user and run the command
sudo -u hdfs jstack 16121
The above command will give you the thread info as below.
java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000d60841a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
Running the command as a different user or root user will provide you with inappropriate/Wrong output as below.
Thread 654947: (state = IN_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
How many thread dumps do we need to collect?
It is essential to collect thread dump multiple times, To identify the thread activities and the state changes. If we are seeing a thread that has been blocked or waiting for more seconds, Then it shows there is some trouble in the process.
Now, How to automate the thread dump collection using a script
Let’s say, If we need to collect thread dump till the server crashes with an interval of 5 sec
Below the script, It will run the thread dump every 5 sec with an infinite loop. We can stop the script once you are done collecting the thread dump.
do sudo -u <user> /usr/java/bin/jstack >> /<Path>$(hostname).jstack.$i; sleep 5s; done
The above scripts can be configured based on our needs. We can change the sleep time to either 10 sec or 15 sec.
NOTE: Make sure you have Java & Jstack utility available to run the above commands
Please start a discussion if you have different ideas for collecting Jstack