I am using spray client i.e. akka within a Tomcat webApp.
While I do properly destroy the actor system on tomcat shutdown via spring, i'm nevertheless still facing this threading issue.
I found that this issue was already reported few years ago Tomcat complaints when re-deploying servlet with akka inside, but could not find the solution for it
I'm using the default spray-client configuration.
here is the error that i get when i stop tomcat. (Sorry for the amount of information but i try to give as much clue as possible)
10-Jun-2015 12:54:22.516 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [PoolPartyConnector-ActorSystem-scheduler-1] but has failed to stop it. This is very likely to create a memory leak.
Stack trace of thread: java.lang.Thread.sleep(Native Method) akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226) akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405) akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) java.lang.Thread.run(Thread.java:745)
10-Jun-2015 12:54:22.516 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [PoolPartyConnector-ActorSystem-akka.actor.default-dispatcher-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
10-Jun-2015 12:54:22.516 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [PoolPartyConnector-ActorSystem-akka.actor.default-dispatcher-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: akka.actor.dungeon.DeathWatch$class.watchingContains(DeathWatch.scala:73) akka.actor.dungeon.DeathWatch$class.watchedActorTerminated(DeathWatch.scala:54) akka.actor.ActorCell.watchedActorTerminated(ActorCell.scala:369) akka.actor.ActorCell.invokeAll$1(ActorCell.scala:455) akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:241) akka.dispatch.Mailbox.run(Mailbox.scala:220) akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
10-Jun-2015 12:54:22.517 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [PoolPartyConnector-ActorSystem-akka.actor.default-dispatcher-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Another part of the trace says:
10-Jun-2015 12:54:22.571 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [oai] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@14483049]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@752622d4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
10-Jun-2015 12:54:22.571 SEVERE [localhost-startStop-1]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [oai] created a ThreadLocal with key of type [scala.concurrent.forkjoin.ThreadLocalRandom$1] (value [scala.concurrent.forkjoin.ThreadLocalRandom$1@20399e41]) and a value of type [scala.concurrent.forkjoin.ThreadLocalRandom] (value [scala.concurrent.forkjoin.ThreadLocalRandom@1286565]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
10-Jun-2015 12:54:22.571 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [oai] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@64b67887]) and a value of type [org.apache.tomcat.util.log.SystemLogHandler] (value [org.apache.tomcat.util.log.SystemLogHandler@523e426f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
10-Jun-2015 12:54:22.571 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [oai] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@14483049]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@20c2d647]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Finally toward the end i get the following
10-Jun-2015 12:54:22.574 INFO [PoolPartyConnector-ActorSystem-akka.actor.default-dispatcher-2] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [scala.util.control.NonFatal$]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [scala.util.control.NonFatal$]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1327) at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1196) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157) at akka.actor.ActorSystemImpl$$anon$2.uncaughtException(ActorSystem.scala:511) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
PS:
This is causing an issue in my Intellij. Indeed, i can still start and stop from the terminal tomcat. However, from within Intellij I get the following on restart:
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 1099; nested exception is: java.net.BindException: Address already in use
I have to kill the process on 1099 every time to restart from within intellij.