Java 8 - Unable to attach to VM after process has been up for a few weeks

524 views Asked by At

I have a production environment (RHEL 7, Java 1.8.0_102-b14) with a number of JBoss (EAP 7), Tomcat( 8 ) and standalone processes running. All of our processes have been running since our last release 18 days ago. I'm unable to connect to the JBoss and standalone processes using the Attach API. jstack or simple attach below fails. All of these methods work fine with my Tomcat processes or with any new processes I start.

This issue isn't affecting anything the running processes, but our monitoring software no longer functions.

public class Attach{
    public static void main( String [] args ) throws Exception {
        com.sun.tools.attach.VirtualMachine vm = com.sun.tools.attach.VirtualMachine.attach( args[ 0 ] );
        System.out.println( vm.getSystemProperties() );
    }
}

My tomcat processes all respond ok and if I launch any new JBoss or standalone processes, I can also connect to them. The set of libraries loaded by the process don't seem to make any difference. The thread count of my JBoss processes is ~2k, the clientprocesses run with ~490, the tomcat processes with ~270.

Here is the jstack error:

26346: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

I 'can' connect using JMC from my local machine, but it takes a few minutes to respond and load the view. If I use the '-F' option with jstack, it produces the following output before hanging

Attaching to process ID 26346, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.102-b14
Deadlock Detection:

No deadlocks found.

I've left this running and it hasn't attached yet but I'll update the post if it does.

I have a release planned for this weekend and will be able to restart all the processes which I suspect will fix it, but it's not a long term solution. In the next release I'll try adding the '-XX:+StartAttachListener' flag to all our processes to see if that helps but it may be difficult to reproduce.

Update #1: Running 'kill -3' does dump the running threads and I can see the AttachListener is running on a process which doesn't respond:

"Attach Listener" #767 daemon prio=9 os_prio=0 tid=0x00002b88a0008000 nid=0x27d0 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

I don't see any deadlocked threads.

Update #2

There is a difference I hadn't considered between the JBoss,Standalone,Tomcat processes. JBoss EAP 7 exposes a JMX connection over its management port. You can connect to it by running JMC using the jboss-client.jar on the bootclasspath. For Tomcat, I explicitly expose jmx with the following arguments so that I can manage it remotely. Could it be possible that having jmxremote enabled is somehow keeping the local JMX attach listener in a responsive state?

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=12345
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 
0

There are 0 answers