For the past few days, we have been noticing a registry refresh failure happening every day on some of our production servers.There are some 3-4 different kind of exceptions thrown every day. giving some examples :
[11/19/14 5:58:14:264 GMT] 00000135 RemoteExcepti E CNTR0019E: EJB threw an unexpected (non-declared) exception during invocation of method "findAll". Exception data: com.ibm.websphere.csi.CSITransactionRolledbackException: Transaction rolled back; nested exception is:
javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1376)
at com.ibm.ejs.csi.TransactionControlImpl.preInvoke(TransactionControlImpl.java:274)
at com.ibm.ejs.csi.UOWControlImpl.preInvoke(UOWControlImpl.java:254)
at com.ibm.ejs.container.EJSContainer.preInvokeActivate(EJSContainer.java:3897)
at com.ibm.ejs.container.EJSContainer.preInvoke(EJSContainer.java:3269)
at com.ibm.commerce.common.objects.EJSRemoteCMPLanguageHome_c0ee4bdc.findAll(EJSRemoteCMPLanguageHome_c0ee4bdc.java:63)
at com.ibm.commerce.common.objects._LanguageHome_Stub.findAll(_LanguageHome_Stub.java:68)
at com.ibm.commerce.common.objects.LanguageAccessBean.findAll(LanguageAccessBean.java:67)
at com.ibm.commerce.common.objsrc.LanguageCache$MyFindAll.generateCacheableData(LanguageCache.java:209)
at com.ibm.commerce.datatype.AbstractFinderResult.invokeFinder(AbstractFinderResult.java:602)
at com.ibm.commerce.datatype.CacheableFinderResult.invokeFinder(CacheableFinderResult.java:195)
at com.ibm.commerce.datatype.AbstractEntityAccessBeanFinderResult.invokeFinder(AbstractEntityAccessBeanFinderResult.java:278)
at com.ibm.commerce.dynacache.commands.TransactionCacheableCommandHelper.executeFromTransactionCache(TransactionCacheableCommandHelper.java:143)
at com.ibm.commerce.dynacache.commands.AbstractFinderResultCache.myInvokeFinder(AbstractFinderResultCache.java:1237)
at com.ibm.commerce.dynacache.commands.AbstractFinderResultCache.invokeFinder(AbstractFinderResultCache.java:1102)
at com.ibm.commerce.common.objsrc.LanguageCache.invokeFinder(LanguageCache.java:586)
at com.ibm.commerce.common.objsrc.LanguageCache.myFindAllCollection(LanguageCache.java:299)
at com.ibm.commerce.common.objsrc.LanguageCache.findAll(LanguageCache.java:264)
at com.ibm.commerce.registry.LanguageRegistry.retrieve(LanguageRegistry.java:204)
at com.ibm.commerce.registry.LanguageRegistry.refresh(LanguageRegistry.java:177)
at com.ibm.commerce.registry.RegistryManager.refreshAll(RegistryManager.java:219)
at com.ibm.commerce.scheduler.commands.RefreshLocalRegistryCmdImpl.refreshRegistry(RefreshLocalRegistryCmdImpl.java:182)
at com.ibm.commerce.scheduler.commands.RefreshLocalRegistryCmdImpl.performExecute(RefreshLocalRegistryCmdImpl.java:136)
at com.ibm.commerce.command.ECCommandTarget.executeCommand(ECCommandTarget.java:157)
at com.ibm.ws.cache.command.CommandCache.executeCommand(CommandCache.java:332)
at com.ibm.websphere.command.CacheableCommandImpl.execute(CacheableCommandImpl.java:166)
at com.ibm.commerce.command.AbstractECTargetableCommand.execute(AbstractECTargetableCommand.java:234)
at com.ibm.commerce.component.BaseComponentImpl.executeCommand(BaseComponentImpl.java:202)
at com.ibm.commerce.component.WebAdapterComponentImpl.executeCommand(WebAdapterComponentImpl.java:46)
at com.ibm.commerce.component.objimpl.WebAdapterServiceBeanBase.executeCommand(WebAdapterServiceBeanBase.java:58)
at com.ibm.commerce.component.objects.EJSLocalStatelessWebAdapterService_ce749a4a.executeCommand(EJSLocalStatelessWebAdapterService_ce749a4a.java:31)
at com.ibm.commerce.component.objects.WebAdapterServiceAccessBean.executeCommand(WebAdapterServiceAccessBean.java:160)
at com.ibm.commerce.scheduler.SchedulerJob.process(SchedulerJob.java:520)
at com.ibm.commerce.scheduler.SchedulerSecurityPriviledgedAction.run(SchedulerSecurityPriviledgedAction.java:59)
at com.ibm.commerce.scheduler.SchedulerJob.run(SchedulerJob.java:1068)
at com.ibm.commerce.threadmanagement.internal.BaseWork.run(BaseWork.java:139)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:264)
at java.security.AccessController.doPrivileged(AccessController.java:228)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1137)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
Caused by: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
at com.ibm.ws.tx.jta.TranManagerImpl.completeTxTimeout(TranManagerImpl.java:527)
at com.ibm.ws.tx.jta.TranManagerSet.completeTxTimeout(TranManagerSet.java:639)
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1368)
... 41 more
Another example :
[11/12/14 5:55:06:024 GMT] FFDC Exception:com.ibm.db2.jcc.am.SqlDataException SourceId:com.ibm.ejs.container.ContainerManagedBeanO.store ProbeId:472 Reporter:com.ibm.ejs.container.ContainerManagedBeanO@12ab12ab
com.ibm.db2.jcc.am.SqlDataException: DB2 SQL Error: SQLCODE=-302, SQLSTATE=22001, SQLERRMC=null, DRIVER=4.11.77
at com.ibm.db2.jcc.am.gd.a(gd.java:668)
at com.ibm.db2.jcc.am.gd.a(gd.java:60)
at com.ibm.db2.jcc.am.gd.a(gd.java:127)
at com.ibm.db2.jcc.am.jn.b(jn.java:2230)
at com.ibm.db2.jcc.am.jn.c(jn.java:2213)
at com.ibm.db2.jcc.t4.cb.k(cb.java:369)
at com.ibm.db2.jcc.t4.cb.a(cb.java:61)
at com.ibm.db2.jcc.t4.q.a(q.java:50)
at com.ibm.db2.jcc.t4.sb.b(sb.java:226)
at com.ibm.db2.jcc.am.kn.oc(kn.java:2930)
at com.ibm.db2.jcc.am.kn.b(kn.java:3876)
at com.ibm.db2.jcc.am.kn.b(kn.java:4047)
at com.ibm.db2.jcc.am.kn.gc(kn.java:743)
at com.ibm.db2.jcc.am.kn.executeUpdate(kn.java:722)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteUpdate(WSJdbcPreparedStatement.java:1152)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeUpdate(WSJdbcPreparedStatement.java:793)
at com.ibm.commerce.base.objects.EJSJDBCPersisterCMPSchedulerBroadcastBean_ccbb593d.store(EJSJDBCPersisterCMPSchedulerBroadcastBean_ccbb593d.java:236)
at com.ibm.ejs.container.ContainerManagedBeanO.store(ContainerManagedBeanO.java:735)
at com.ibm.ejs.container.EntityBeanO.beforeCompletion(EntityBeanO.java:1237)
at com.ibm.ejs.container.ContainerTx.beforeCompletion(ContainerTx.java:721)
at com.ibm.ejs.util.tran.SyncDriver.beforeCompletion(SyncDriver.java:280)
at com.ibm.tx.jta.RegisteredSyncs.coreDistributeBefore(RegisteredSyncs.java:289)
at com.ibm.ws.tx.jta.RegisteredSyncs.distributeBefore(RegisteredSyncs.java:150)
at com.ibm.ws.tx.jta.TransactionImpl.prePrepare(TransactionImpl.java:2323)
at com.ibm.ws.tx.jta.TransactionImpl.stage1CommitProcessing(TransactionImpl.java:541)
at com.ibm.tx.jta.TransactionImpl.processCommit(TransactionImpl.java:986)
at com.ibm.tx.jta.TransactionImpl.commit(TransactionImpl.java:920)
at com.ibm.ws.tx.jta.TranManagerImpl.commit(TranManagerImpl.java:436)
at com.ibm.tx.jta.TranManagerSet.commit(TranManagerSet.java:161)
at com.ibm.ws.tx.jta.UserTransactionImpl.commit(UserTransactionImpl.java:293)
at com.ibm.commerce.server.TransactionManager.commit(TransactionManager.java:183)
at com.ibm.commerce.server.TransactionManager.commit(TransactionManager.java:241)
at com.ibm.commerce.scheduler.commands.RefreshLocalRegistryCmdImpl.updateState(RefreshLocalRegistryCmdImpl.java:352)
at com.ibm.commerce.scheduler.commands.RefreshLocalRegistryCmdImpl.refreshRegistry(RefreshLocalRegistryCmdImpl.java:198)
at com.ibm.commerce.scheduler.commands.RefreshLocalRegistryCmdImpl.performExecute(RefreshLocalRegistryCmdImpl.java:136)
one more:
11/13/14 5:55:01:603 GMT] 00000008 TimeoutManage I WTRN0006W: Transaction 00000149A7B3897F00000001001F589FFF2F876A1300AAE3A7ED20831067E29B8BFE6E2500000149A7B3897F00000001001F589FFF2F876A1300AAE3A7ED20831067E29B8BFE6E2500000001 has timed out after 300 seconds.
[11/13/14 5:55:01:605 GMT] 00000008 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WorkManager.broadcast : 83,5,reserved: WC_arglp]. The stack trace of this thread when the timeout occurred was:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:140)
com.ibm.db2.jcc.t4.z.b(z.java:199)
com.ibm.db2.jcc.t4.z.c(z.java:259)
com.ibm.db2.jcc.t4.z.c(z.java:372)
com.ibm.db2.jcc.t4.z.v(z.java:1147)
com.ibm.db2.jcc.t4.cb.c(cb.java:53)
com.ibm.db2.jcc.t4.q.c(q.java:44)
com.ibm.db2.jcc.t4.rb.j(rb.java:147)
com.ibm.db2.jcc.am.jn.kb(jn.java:2047)
com.ibm.db2.jcc.am.kn.b(kn.java:3900)
com.ibm.db2.jcc.am.kn.fc(kn.java:702)
com.ibm.db2.jcc.am.kn.executeQuery(kn.java:672)
com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteQuery(WSJdbcPreparedStatement.java:1082)
com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeQuery(WSJdbcPreparedStatement.java:703)
com.ibm.commerce.catalog.objects.EJSJDBCPersisterCMPCatalogGroupDescriptionBean_6f241499.load(EJSJDBCPersisterCMPCatalogGroupDescriptionBean_6f241499.java:166)
com.ibm.ejs.container.ContainerManagedBeanO.load(ContainerManagedBeanO.java:1035)
com.ibm.ejs.container.ContainerManagedBeanO.loadForEnlist(ContainerManagedBeanO.java:992)
com.ibm.ejs.container.EntityBeanO.enlist(EntityBeanO.java:785)
com.ibm.ejs.container.activator.OptCEntityActivationStrategy.atActivate(OptCEntityActivationStrategy.java:168)
com.ibm.ejs.container.activator.Activator.activateBean(Activator.java:599)
com.ibm.ejs.container.EJSHome.activateBean_Common(EJSHome.java:2052)
com.ibm.ejs.container.EJSHome.activateBean(EJSHome.java:1890)
com.ibm.commerce.catalog.objects.EJSJDBCPersisterCMPCatalogGroupDescriptionBean_6f241499.findByPrimaryKey(EJSJDBCPersisterCMPCatalogGroupDescriptionBean_6f241499.java:423)
com.ibm.commerce.catalog.objects.EJSCMPCatalogGroupDescriptionHomeBean_6f241499.findByPrimaryKey(EJSCMPCatalogGroupDescriptionHomeBean_6f241499.java:31)
com.ibm.commerce.catalog.objects.EJSRemoteCMPCatalogGroupDescriptionHome_6f241499.findByPrimaryKey(EJSRemoteCMPCatalogGroupDescriptionHome_6f241499.java:112)
com.ibm.commerce.catalog.objects._CatalogGroupDescriptionHome_Stub.findByPrimaryKey(_CatalogGroupDescriptionHome_Stub.java:236)
com.ibm.commerce.catalog.objects.CatalogGroupDescriptionAccessBean.instantiateEJBByPrimaryKey(CatalogGroupDescriptionAccessBean.java:239)
com.ibm.commerce.catalog.beansrc.CompactCatalogGroupDescriptionAccessBean.instantiateEJBByPrimaryKey(CompactCatalogGroupDescriptionAccessBean.java:241)
com.ibm.ivj.ejb.runtime.AbstractEntityAccessBean._instantiate(AbstractEntityAccessBean.java:173)
com.ibm.ivj.ejb.runtime.AbstractEntityAccessBean.refreshCopyHelper(AbstractEntityAccessBean.java:309)
com.ibm.commerce.catalog.objects.CatalogGroupDescriptionAccessBean.refreshCopyHelper(CatalogGroupDescriptionAccessBean.java:271)
com.ibm.commerce.catalog.objsrc.CachedCatalogGroupDescriptionAccessBean.refreshCopyHelper(CachedCatalogGroupDescriptionAccessBean.java:243)
com.ibm.commerce.datatype.AbstractEntityAccessBeanFinderResult.invokeFinder(AbstractEntityAccessBeanFinderResult.java:317)
com.ibm.commerce.dynacache.commands.AbstractFinderResultCache.myInvokeFinder(AbstractFinderResultCache.java:1411)
com.ibm.commerce.dynacache.commands.AbstractFinderResultCache.invokeFinder(AbstractFinderResultCache.java:1102)
com.ibm.commerce.catalog.objsrc.CatalogGroupDescriptionCache.invokeFinder(CatalogGroupDescriptionCache.java:639)
com.ibm.commerce.catalog.objsrc.CatalogGroupDescriptionCache.invokeSingleCatalogGroupDescriptionAccessBeanFinder(CatalogGroupDescriptionCache.java:610)
com.ibm.commerce.catalog.objsrc.CatalogGroupDescriptionCache.findByPrimaryKey(CatalogGroupDescriptionCache.java:229)
com.ibm.commerce.catalog.objsrc.CatalogGroupDescriptionCache.findByPrimaryKey(CatalogGroupDescriptionCache.java:182)
com.ibm.commerce.catalog.objimpl.CatalogGroupBeanBase.getDescription(CatalogGroupBeanBase.java:578)
com.ibm.commerce.catalog.objimpl.CatalogGroupBeanBase.getFallbackDescription(CatalogGroupBeanBase.java:641)
com.ibm.commerce.base.objects.ECEntityBean.getFallbackDescription(ECEntityBean.java:276)
com.ibm.commerce.catalog.objimpl.CatalogGroupBeanBase.getDescription(CatalogGroupBeanBase.java:603)
com.ibm.commerce.catalog.objects.EJSRemoteCMPCatalogGroup_8e5bb4d3.getDescription(EJSRemoteCMPCatalogGroup_8e5bb4d3.java:1043)
com.ibm.commerce.catalog.objects._CatalogGroup_Stub.getDescription(_CatalogGroup_Stub.java:872)
com.ibm.commerce.catalog.objects.CatalogGroupAccessBean.getDescription(CatalogGroupAccessBean.java:890)
com.ibm.commerce.catalog.objsrc.CachedCatalogGroupAccessBean.getDescription(CachedCatalogGroupAccessBean.java:298)
com.ibm.commerce.catalog.helper.SearchNavigationCache$MyFindCategoryNameAndParent.generateCacheableData(SearchNavigationCache.java:125)
com.ibm.commerce.datatype.AbstractFinderResult.invokeFinder(AbstractFinderResult.java:602)
com.ibm.commerce.datatype.CacheableFinderResult.invokeFinder(CacheableFinderResult.java:195)
com.ibm.commerce.dynacache.commands.AbstractFinderResultCache.myInvokeFinder(AbstractFinderResultCache.java:1411)
com.ibm.commerce.dynacache.commands.AbstractFinderResultCache.invokeFinder(AbstractFinderResultCache.java:1102)
com.ibm.commerce.dynacache.commands.AbstractDistributedMapCache$Cache.getOrPut(AbstractDistributedMapCache.java
We have been getting these exceptions repeatedly.
Can anyone suggest why is this happening and how to resolve this?
Saw today that IBM has created an APAR for an registry update issue.
https://www-304.ibm.com/connections/blogs/wcs/entry/ensure_you_have_jr48163?lang=en
------------- old below --------------
We usually don't refresh registries by scheduling because configuration changes could be replicated in an unpredictable way. But if you run this scheduled command it's required to be done with an application type of BROADCAST.
http://www-01.ibm.com/support/knowledgecenter/SSZLC2_7.0.0/com.ibm.commerce.admin.doc/concepts/cjsjobs.htm