We're using WebSphere AppServer 8.5 for Web processing, but at a specific point in time, such as 8 a.m., we get thread deadlocks and eventually the thread hangs. One hour later, the AppServer is running slowly. One hour later, however, the AppServer recovers. After checking the scheduled tasks of the operating system, it is found that no task is executed at 8:00 a.m.
1CIJAVAVERSION JRE 1.7.0 Windows 7 x86-32 build (pwi3270sr9-20150417_01(SR9) )
1CIVMVERSION VM build R26_Java726_SR9_20150406_1443_B242981
1CIJITVERSION tr.r11_20150401_88894
1CIGCVERSION GC - R26_Java726_SR9_20150406_1443_B242981
1CIJITMODES JIT enabled, AOT enabled, FSD disabled, HCR disabled
1CIRUNNINGAS Running as a standalone JVM
1CIPROCESSID Process ID: 10332 (0x285C)
1CICMDLINE C:\IBMRPT_871\SDP\jdk\bin\java.exe -DrptNextgenDebug -Drptserver.rootDiscoveryUrl=http://9.84.122.22:7081/deployment/ -Drptagent.agentName=localhost -Drptagent.engineName=localhost -Drptagent.scheduleName=sched2 -Drptagent.rptDeployDir=file:/C:/Users/IBM_ADMIN/IBM/RPTCitrixTest871/deployment_root/gdaronde/A1E515CFE7BA0690C143E56139636135/ -Drptagent.secure=false -Drptagent.securePort=7444 -Drptagent.installBase=C:/IBMRPT_871/SDP/Majordomo/ -Xmx1200m -DrptLocale=fr_FR -Drptserver.domoHeavyClass=com.ibm.rational.test.lt.kernel.runner.impl.RPTNextgenRunner -Drptserver.domoLiteUrl=file:/C:/IBMRPT_871/SDP/Majordomo/lib/ -Drptserver.domoLiteClass=com.ibm.rational.test.lt.nextgen.Domo -cp C:\IBMRPT_871\SDP\Majordomo\lib\boot.jar com.ibm.rational.test.lt.boot.DomoBooter
1CIJAVAHOMEDIR Java Home Dir: C:\IBMRPT_871\SDP\jdk\jre
1CIJAVADLLDIR Java DLL Dir: C:\IBMRPT_871\SDP\jdk\jre\bin
1CISYSCP Sys Classpath: C:\IBMRPT_871\SDP\jdk\jre\bin\default\jclSC170\vm.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\se-service.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\math.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\jlm.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmorb.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmorbapi.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmcfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmpkcs.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmcertpathfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjgssfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjssefw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmsaslfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjcefw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjgssprovider.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjsseprovider2.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmcertpathprovider.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\xmldsigfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\xml.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\charsets.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\resources.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\rt.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmgpu.jar;
1CIUSERARGS UserArgs:
2CIUSERARG -Xoptionsfile=C:\IBMRPT_871\SDP\jdk\jre\bin\default\options.default
0MEMUSER
1MEMUSER JRE: 1,320,399,028 bytes / 2770 allocations
1MEMUSER |
2MEMUSER +--VM: 1,315,550,900 bytes / 2282 allocations
2MEMUSER | |
3MEMUSER | +--Classes: 16,393,128 bytes / 508 allocations
2MEMUSER | |
3MEMUSER | +--Memory Manager (GC): 1,284,062,768 bytes / 446 allocations
3MEMUSER | | |
4MEMUSER | | +--Java Heap: 1,258,291,200 bytes / 1 allocation
3MEMUSER | | |
4MEMUSER | | +--Other: 25,771,568 bytes / 445 allocations
2MEMUSER | |
3MEMUSER | +--Threads: 13,546,748 bytes / 236 allocations
3MEMUSER | | |
4MEMUSER | | +--Java Stack: 475,728 bytes / 50 allocations
3MEMUSER | | |
4MEMUSER | | +--Native Stack: 12,746,752 bytes / 50 allocations
3MEMUSER | | |
4MEMUSER | | +--Other: 324,268 bytes / 136 allocations
2MEMUSER | |
3MEMUSER | +--Trace: 444,968 bytes / 353 allocations
2MEMUSER | |
3MEMUSER | +--JVMTI: 17,328 bytes / 13 allocations
2MEMUSER | |
3MEMUSER | +--JNI: 70,504 bytes / 290 allocations
2MEMUSER | |
3MEMUSER | +--Port Library: 9,632 bytes / 74 allocations
2MEMUSER | |
3MEMUSER | +--Other: 1,005,824 bytes / 362 allocations
1MEMUSER |
2MEMUSER +--JIT: 3,880,280 bytes / 225 allocations
2MEMUSER | |
3MEMUSER | +--JIT Code Cache: 1,048,576 bytes / 2 allocations
2MEMUSER | |
3MEMUSER | +--JIT Data Cache: 524,336 bytes / 1 allocation
2MEMUSER | |
3MEMUSER | +--Other: 2,307,368 bytes / 222 allocations
1MEMUSER |
2MEMUSER +--Class Libraries: 967,848 bytes / 263 allocations
2MEMUSER | |
3MEMUSER | +--Harmony Class Libraries: 1,024 bytes / 1 allocation
2MEMUSER | |
3MEMUSER | +--VM Class Libraries: 966,824 bytes / 262 allocations
3MEMUSER | | |
4MEMUSER | | +--sun.misc.Unsafe: 72,360 bytes / 10 allocations
4MEMUSER | | | |
5MEMUSER | | | +--Direct Byte Buffers: 66,040 bytes / 8 allocations
4MEMUSER | | | |
5MEMUSER | | | +--Other: 6,320 bytes / 2 allocations
3MEMUSER | | |
4MEMUSER | | +--Other: 894,464 bytes / 252 allocations
3XMTHREADINFO "Brother-7" J9VMThread:0x518FDD00, j9thread_t:0x527F7634, java/lang/Thread:0x030B1D80, state:P, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x32, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0xA44, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x01020000)
3XMCPUTIME CPU usage total: 0.0 secs, user: 0.0 secs, system: 0.0 secs
3XMTHREADBLOCK Parked on: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x02E83C68 Owned by: <unknown>
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at sun/misc/Unsafe.park(Native Method)
4XESTACKTRACE at java/util/concurrent/locks/LockSupport.park(LockSupport.java:198)
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2055)
4XESTACKTRACE at java/util/concurrent/LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:501)
4XESTACKTRACE at java/util/concurrent/LinkedBlockingDeque.take(LinkedBlockingDeque.java:690)
4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1080)
4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
4XESTACKTRACE at java/lang/Thread.run(Thread.java:798)
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK NtWaitForSingleObject+0x15 (0x77A0F8CD [ntdll+0x1f8cd])
4XENATIVESTACK WaitForSingleObjectEx+0x43 (0x75491194 [kernel32+0x11194])
4XENATIVESTACK WaitForSingleObject+0x12 (0x75491148 [kernel32+0x11148])
4XENATIVESTACK j9thread_park+0xba (j9thread.c:2631, 0x73B71C8A [J9THR26+0x1c8a])
4XENATIVESTACK jclCallThreadPark+0x198 (threadpark.c:82, 0x666C2F88 [jclse7b_26+0x42f88])
4XENATIVESTACK sun_misc_Unsafe_park+0x5a (smunsafe.asm:14351, 0x666B760A [jclse7b_26+0x3760a])
4XENATIVESTACK javaProtectedThreadProc+0x9d (vmthread.c:1881, 0x66831DBD [j9vm26+0x51dbd])
4XENATIVESTACK j9sig_protect+0x44 (j9signal.c:150, 0x6BB9F8B4 [J9PRT26+0xf8b4])
4XENATIVESTACK javaThreadProc+0x39 (vmthread.c:298, 0x66832739 [j9vm26+0x52739])
4XENATIVESTACK thread_wrapper+0xda (j9thread.c:1154, 0x73B747AA [J9THR26+0x47aa])
4XENATIVESTACK _endthread+0x48 (0x67E3C55C [msvcr100+0x5c55c])
4XENATIVESTACK _endthread+0xe8 (0x67E3C5FC [msvcr100+0x5c5fc])
4XENATIVESTACK BaseThreadInitThunk+0x12 (0x7549337A [kernel32+0x1337a])
4XENATIVESTACK RtlInitializeExceptionChain+0x63 (0x77A292B2 [ntdll+0x392b2])
4XENATIVESTACK RtlInitializeExceptionChain+0x36 (0x77A29285 [ntdll+0x39285])
We analyzed the thread information and found that there may be a GC conflict between WebSphere AppServer 8.5 and JDK 1.7, which will be resolved after the server is restarted.
We found the same problem. We used Websphere 8.2, at a specific point in time, the Websphere was hung all threads and made a deadlock in JVM. So, we create a case on IBM Support and they said that the Java 1.7 Full GC and main thread of WebSphere conflict at some point in time.
We already upgraded to WebSphere 9.0.5 and JDK 1.8. The problem has been resolved.
IBM Docs: https://www.ibm.com/docs/en/ztpf/1.1.0.15?topic=threads-blocked-thread-information