404 Handler Hanging on ColdFusion 10, works perfect on ColdFusion 8

Question

I'm having an odd intermittent issue with the 404 handler pages on our ColdFusion 10 server. Here's the backstory:

1. We have been running 2 load-balanced ColdFusion 8 servers for ages without issue.

2. We have upgraded one of these two servers to ColdFusion 10 in order to soft-launch our ColdFusion 10 upgrade.

3. I've installed Update 8 on the CF10 server, and have removed and re-added the connectors.

4. Both servers are running Windows 2008 R2.

I've read several forum posts stating issues with people getting connection reset issues with their 404 pages, but I'm not seeing that particular issue. What I am seeing is a hung thread in FusionReactor for the 404.cfm file. Here's an example of the stacktrace taken from one of these hung threads:

Thread Stack Trace
Trace Time:   07:02:14.638 04-Mar-2013
Request ID:   179934
Script Name:  http://example.com/404.cfm?404;http://example.com:80/somemissingfile
Started:      06:07:17.581 04-Mar-2013
Exec Time:    3297057ms
Memory Used:  (32%)1,669,600KB
Memory Free:  3,539,295KB
Thread ID:    0x53de (21470)
Thread Name:  ajp-bio-8012-exec-86
Priority:     5
Hashcode:     1636847260
State:        RUNNABLE

"ajp-bio-8012-exec-86" daemon prio=5 runnable

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:319)
org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:445)
org.apache.coyote.ajp.AjpProcessor.receive(AjpProcessor.java:341)
org.apache.coyote.ajp.AbstractAjpProcessor.finish(AbstractAjpProcessor.java:1032)
org.apache.coyote.ajp.AbstractAjpProcessor.action(AbstractAjpProcessor.java:386)
org.apache.coyote.Response.action(Response.java:170)
org.apache.coyote.Response.finish(Response.java:276)
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:288)
org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:108)
com.intergral.fusionreactor.plugin.amf.InterposerOutputStream.close(InterposerOutputStream.java:104)
com.intergral.fusionreactor.trackedstream.TrackedStream.close(TrackedStream.java:113)
com.intergral.fusionreactor.filter.softkill.SoftKillResponseStream.close(SoftKillResponseStream.java:146)
com.intergral.fusionreactor.filter.FusionReactorResponseWrapper.finish(FusionReactorResponseWrapper.java:192)
com.intergral.fusionreactor.core.FusionReactor.finish(FusionReactor.java:683)
com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doHttpServletRequest(FusionReactorCoreFilter.java:575)
com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorCoreFilter.java:337)
com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:246)
com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java:121)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
java.lang.Thread.run(Thread.java:662)

I've tried manually killing these threads through FusionReactor, but they won't die. The only way to clear the hung threads is to restart the ColdFusion service (which I've had to do at least once a day since I upgraded to ColdFusion 10.

The 404 doesn't do anything special other than run our standard layout custom tag and display a page not found notification.

Looking at the thread dump it appears that ColdFusion has finished doing the processing and completed filling the buffer with output, but it's not finishing sending to the browser?

org.apache.coyote.ajp.AbstractAjpProcessor.finish(AbstractAjpProcessor.java:1032)
org.apache.coyote.ajp.AbstractAjpProcessor.action(AbstractAjpProcessor.java:386)
org.apache.coyote.Response.action(Response.java:170)
org.apache.coyote.Response.finish(Response.java:276)
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:288)
org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:108)

There's lots of stuff there about closing and finishing, but the steps following it have more reads and responses. So I'm not sure what might be happening here, but it's definitely stalling the completion of our ColdFusion 10 migration. I need to discover exactly what's causing these hung threads.

Answer 1

What you are experiencing is the same bug I encountered in January. I've logged the bug in Adobe's bugbase:

https://bugbase.adobe.com/index.cfm?event=bug&id=3494728

Their engineering team has acknowledged the issue, but no ETA on an actual fix. Adobe's connector for IIS in CF10 has been quite troublesome for us. You can set a timeout to kill these hanging threads as listed in my workaround, which will at least keep you from having to reboot the server daily. Good luck!