VirtualMachine attach throws com.sun.tools.attach.AgentLoadException: 0 when using Open JDK 10.0.2

Question

I am facing an issue when using the com.sun.tools.attach.VirtualMachineJava API. My target application(Tomcat) is running using Oracle Hot Spot version 1.7.0_80. I am trying to connect that tomcat via dynamic attach from another Java program (on the same machine) which is using Open JDK 10.0.2. I am using the below code

...............
String agentPath = Agent.class.getProtectionDomain().getCodeSource().getLocation().getPath();
agentPath = convertFileSeparators(agentPath, File.separatorChar);
String agentInstallDir = agentPath.substring(0, agentPath.lastIndexOf(File.separator));
System.out.println("My Java agent installed on the location :"+agentPath);
StringBuilder sb = new StringBuilder();
sb.append("MY_RELIC_HOME").append("=").append(agentInstallDir);
if (agentArgs != null) {
     sb.append(",").append(agentArgs);
}
com.sun.tools.attach.VirtualMachine virtualMachine =  com.sun.tools.attach.VirtualMachine.attach(vmID);
virtualMachine.loadAgent(agentPath, sb.toString());
.........

I am able to attach successfully to the target application , but getting the below exception in my attach program (running open JDK 10.0.2) after attach.

com.sun.tools.attach.AgentLoadException: 0
    at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:104)
    at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:115)
    at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:139)
    at com.eg.agent.Agent.main(Agent.java:582)

The code on the line no 582 is virtualMachine.loadAgent(agentPath, sb.toString());.

If i run my attach program using Java 7 or 8, there is no exception and attach is success.

Why com.sun.tools.attach.AgentLoadException: 0 exception occurs when using JDK 10 ?

Answer 1

Found the issue , After looking the source code of sun.tools.attach.HotSpotVirtualMachine on both version of java 10 & 8. The Method loadAgentLibrary(...) is completely different between 10 & 8.

Below code is from 1.8 which shows that the value of the result is zero means VMAttach is success.

private void loadAgentLibrary(String agentLibrary, boolean isAbsolute, String options)
    throws AgentLoadException, AgentInitializationException, IOException
{
    InputStream in = execute("load",
                             agentLibrary,
                             isAbsolute ? "true" : "false",
                             options);
    try {
        int result = readInt(in);
        if (result != 0) {
            throw new AgentInitializationException("Agent_OnAttach failed", result);
        }
    } finally {
        in.close();

    }
}

Below code is from java 10 in VM Attach is success when the value of the result is "return code: 0" (but in 1.8 it is simply 0).

private void loadAgentLibrary(String agentLibrary, boolean isAbsolute, String options) 
    throws AgentLoadException, AgentInitializationException, IOException 
{ 
    String msgPrefix = "return code: "; 
    InputStream in = execute("load", 
                             agentLibrary, 
                             isAbsolute ? "true" : "false", 
                             options); 
    try (BufferedReader reader = new BufferedReader(new InputStreamReader(in))) { 
        String result = reader.readLine(); 
        if (result == null) { 
            throw new AgentLoadException("Target VM did not respond"); 
        } else if (result.startsWith(msgPrefix)) { 
            int retCode = Integer.parseInt(result.substring(msgPrefix.length())); 
            if (retCode != 0) { 
                throw new AgentInitializationException("Agent_OnAttach failed", retCode); 
            } 
        } else { 
            throw new AgentLoadException(result); 
        } 
    } 
} 

In my case Java 7/8 code (Target Application) returns the result as 0(which means VM Attach is success) to Java 10 code (VM Attach code), which expects the result should be "return code: 0"