Commands like jps, jstat, jstack, etc show "<pid> not found"

Question

I am running Tomcat 9 on an Ubuntu 20.04 OS using OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode). When I try to gather diagnostics info using jstat, jstack, etc, I see PID not found. jps also cannot identify the Tomcat process id.

I have checked several posts like one, two, three, four, etc, but none of the answers given in these helped me to solve my problem!

Even though I am passing the username with which the Tomcat process is running, jstat cannot find that process: sudo -u tomcat jstat -gc 476174 5000

In case it matters:

1. I can see that the Tomcat process is started with -Djava.io.tmpdir=/tmp This folder is owned by root user but has full permissions(777) enabled.
2. When the Tomcat process is started, I can see a folder with name systemd-private-e6d8b5dc224848f8a64a3e943ac2e9c4-tomcat9.service-UH5knj (the last few chars after service- change every time the process is restarted) getting created with owner as root (probably because I start tomcat service using sudo service tomcat9 start) and this folder has permission of rwx------.

Any hints on how to solve this issue?

Thanks, Shobhana

Answer 1

In Ubuntu 20 the Tomcat 9 service runs in a private tmp space by default and because of that the java tools cannot find its pid.

This private tmp setting is defined in /lib/systemd/system/tomcat9.service as PrivateTmp. It is set to "yes" by default. You could edit that file directly but the recommended way to make changes is to create an override file using this command:

sudo systemctl edit tomcat9

Add the following content to the file and save:

[Service]
PrivateTmp=no
ReadWritePaths=/tmp/

It will create this file with your content: /etc/systemd/system/tomcat9.service.d/override.conf

Then you need to reload the systemd daemon and restart Tomcat:

sudo systemctl daemon-reload
sudo systemctl restart tomcat9

You should now be able to call the java tools using sudo directly, e.g. sudo jstat.... For some reason it does not work when I try to run it as the tomcat user with sudo -u tomcat jstat...

Answer 2

All these tools (jstack, jmap, jstat...) rely on the communication with the target JVM through /tmp directory.

Apparently Tomcat runs in a different mount namespace, so that its /tmp directory is not the same as /tmp of the current shell. To verify this, run

readlink /proc/$$/ns/mnt
readlink /proc/<java_pid>/ns/mnt

and check if inode numbers are different.

Unforunately, JDK 8 tools do not support Linux namespaces. You may try nsenter to launch these tools under Tomcat's namespace.

Or simply use JDK 11 (or newer) tools, which support containers (and thus namespaces) out of the box. They also work with applications running under JDK 8, e.g. jstack from JDK 11 can dump threads of a JDK 8 process.

Alternatively, there is a small jattach utility which can be used in place of jstack, jmap, and jcmd. It also supports containers out of the box. It does not require installation - there is just a single standalone binary.