Access Remote EJB in one Websphere profile from another WebSphere profile

Question

In short: I make a remote JNDI call for a stateless bean from one Websphere server to another (different profiles in different JVMs on the same machine) but after the call is done, the InitialContext gets changed and I can no longer access my local beans.

The situation is as follows:
A Websphere server Server1 has a complex application with many beans and also needs to access a stateless bean (BeanX) on Websphere Server2 that runs another application.
I have managed to access BeanX on Server2 using either of the two methods:

1. Connection via code:

   private static Context;
   static {
     Hashtable env = new Hashtable();
     env.put(Context.INITIAL_CONTEXT_FACTORY, com.ibm.websphere.naming.WsnInitialContextFactory");
     env.put(Context.PROVIDER_URL, "iiop://localhost:9101");
     try {
       ctx = NamingManager.getInitialContext(env);
       mgr = (RemoteBeanXManager) PortableRemoteObject.narrow(ctx.lookup("BeanXManagerBean/remote"), RemoteBeanXManager.class);
     } catch (NamingException e) {//logging
     }
   }
   

2. Connection via Webshere configuration Configured in Server1: Environment -> Naming -> Name Space binding -> New ... Indirect And introduced the required values, corresponding to the exposed bean from Server2.
   Provider URL: corbaloc:iiop:localhost:9101
   Initial context factory name: com.ibm.websphere.naming.WsnInitialContextFactory
   And then proceeded to obtain the bean in the code as if it was in my local context.

I removed security for RMI/IIOP for Outbound in Server1 and for Inbound in Server2, so that the servers can communicate.
The bean is exposed by Server2, the remote interface RemoteBeanXManager exists in a common package for both applications and the call goes through correctly.

The problem appears after the call, when subsequent EJB injections in Server1 for its local beans return a com.ibm.ejs.container.util.ExceptionUtil.NoSuchEJBException

Basically, at the next @EJB annotation encountered, Server1 will throw:

com.ibm.ejs.container.EJBNotFoundException: EJB named SomeOtherBean not present in application My Enterprise Application. at com.ibm.ejs.container.HomeOfHomes.resolveEJBLink(HomeOfHomes.java:751)

What I managed to understand as I investigate, is that the InitialContext on Server1 is changed by the connection to Server2, and cannot 'reset' back after the call through the remote interface. I tried re-obtaining the InitialContext with the default settings after the remote manager is obtained, but to no avail.
I tried to add the optional configuration parameters:
com.ibm.websphere.naming.jndicache.cachename providerURL
com.ibm.websphere.naming.jndicache.cacheobject none

Still the same exception.

If you have any ideas on how to isolate/cache the context on Server1 so that it can resume normal operation after a RMI-IIOP call, please let me know

Updated to add:
When I comment out the code that performs the remote call, the application starts correctly, and all @EJB injections work.

In Server2, at the call of the method from Server1 - no exception, just these messages:

    [11/27/18 9:28:13:525 EET] 00000139 BeanXManagerB I com.my.app.manager.impl.BeanXManagerBean performTaskForServer1 End     
    [11/27/18 9:28:13:883 EET] 00000139 RegisteredRes E   WTRN0064E: An illegal attempt to commit a one phase capable resource in a subordinate transaction branch has occurred.    
    [11/27/18 9:28:13:920 EET] 00000139 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile2\logs\ffdc\Server2_db3b4f88_18.11.27_09.28.13.8878526370681854823401.txt com.ibm.tx.jta.TransactionImpl.prepareResources 467    
    [11/27/18 9:28:13:932 EET] 00000139 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile2\logs\ffdc\Server2_db3b4f88_18.11.27_09.28.13.9214065828085434219239.txt com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505    

The complete stacktrace of Server1's error is:
[I thought it would be too long to post all the incident reports. ]

    [11/27/18 9:28:13:946 EET] 0000011a DMAdapter     I com.ibm.ws.ffdc.impl.DMAdapter getAnalysisEngine FFDC1009I: Analysis Engine using data base: D:\programs\IBM\WebSphere\AppServer\properties\logbr\ffdc\adv\ffdcdb.xml    
    [11/27/18 9:28:13:967 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9411059231941807719848.txt com.ibm.tx.jta.impl.RegisteredResources.prepareResource 1216    
    [11/27/18 9:28:13:991 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9683456920548561326798.txt com.ibm.tx.jta.TransactionImpl.prepareResources 467    
    [11/27/18 9:28:14:014 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.13.9913497264494769300154.txt com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505    
    [11/27/18 9:28:14:032 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0285054752089388843670.txt com.ibm.ejs.csi.TranStrategy.commit 294    
    [11/27/18 9:28:14:039 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.036797394156202410033.txt com.ibm.ejs.container.EJSHome.createBeanO 1047    
    [11/27/18 9:28:14:044 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0403016172875256112876.txt com.ibm.ejs.csi.EJBApplicationMetaData.createStartupBeans 6921    

    [11/27/18 9:28:14:044 EET] 0000011a EJBApplicatio E   CNTR0190E: The StarterBean startup singleton session bean in the app1.jar module failed initialization with exception:    
    javax.ejb.NoSuchEJBException: An error occurred during initialization of singleton session bean MY Enterprise Application#app1.jar#StarterBean, resulting in the discarding of the singleton instance.; nested exception is: javax.ejb.EJBTransactionRolledbackException: nested exception is: javax.transaction.RollbackException    
        at com.ibm.ejs.container.util.ExceptionUtil.NoSuchEJBException(ExceptionUtil.java:540)    
        at com.ibm.ejs.container.EJSHome.createSingletonBeanO(EJSHome.java:3752)    
        at com.ibm.ejs.csi.EJBApplicationMetaData.createStartupBeans(EJBApplicationMetaData.java:959)    
        at com.ibm.ejs.csi.EJBApplicationMetaData.startedModule(EJBApplicationMetaData.java:680)    
        at com.ibm.ws.runtime.component.EJBContainerImpl.stateChanged(EJBContainerImpl.java:4525)    
        at com.ibm.ws.runtime.component.ApplicationMgrImpl$ComparableDeployedObjectListener.stateChanged(ApplicationMgrImpl.java:2652)    
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.stateChanged(ApplicationMgrImpl.java:1178)    
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectEvent(DeployedApplicationImpl.java:1558)    
        at com.ibm.ws.runtime.component.DeployedModuleImpl.setState(DeployedModuleImpl.java:252)    
        at com.ibm.ws.runtime.component.DeployedModuleImpl.setState(DeployedModuleImpl.java:248)    
        at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707)    
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1150)    
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:800)    
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1450)        
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2311)    
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)    
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)    
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)    
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)    
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)    
        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5574)    
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5700)    
        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)    
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)    
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)    
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1340)    
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)    
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)    
        at java.lang.reflect.Method.invoke(Method.java:508)    
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:83)    
        at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)    
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)    
        at java.lang.reflect.Method.invoke(Method.java:508)    
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:287)    
        at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1263)    
        at java.security.AccessController.doPrivileged(AccessController.java:666)    
        at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)    
        at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1257)    
        at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1096)    
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)    
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:813)    
        at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1350)    
        at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)    
        at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1243)    
        at com.ibm.ws.management.application.sync.StartDeploymentTask.startDeployment(StartDeploymentTask.java:249)    
        at com.ibm.ws.management.application.sync.StartDeploymentTask.fullAppUpdate(StartDeploymentTask.java:121)    
        at com.ibm.ws.management.application.sync.StartDeploymentTask.performTask(StartDeploymentTask.java:109)    
        at com.ibm.ws.management.application.sync.AppBinaryProcessor$ExpandApp.expand(AppBinaryProcessor.java:1770)    
        at com.ibm.ws.management.application.sync.AppBinaryProcessor.postProcessSynchronousExt(AppBinaryProcessor.java:811)    
        at com.ibm.ws.management.bla.sync.BLABinaryProcessor.postProcess(BLABinaryProcessor.java:599)    
        at com.ibm.ws.management.bla.sync.BLABinaryProcessor.onChangeCompletion(BLABinaryProcessor.java:476)    
        at com.ibm.ws.management.bla.sync.BinaryProcessorWrapper.onChangeCompletion(BinaryProcessorWrapper.java:109)    
        at com.ibm.ws.management.repository.FileRepository.postNotify(FileRepository.java:1938)    
        at com.ibm.ws.management.repository.FileRepository.update(FileRepository.java:1442)    
        at com.ibm.ws.management.repository.client.LocalConfigRepositoryClient.update(LocalConfigRepositoryClient.java:189)    
        at com.ibm.ws.sm.workspace.impl.WorkSpaceMasterRepositoryAdapter.update(WorkSpaceMasterRepositoryAdapter.java:667)    
        at com.ibm.ws.sm.workspace.impl.RepositoryContextImpl.update(RepositoryContextImpl.java:1998)    
        at com.ibm.ws.sm.workspace.impl.RepositoryContextImpl.synch(RepositoryContextImpl.java:1946)    
        at com.ibm.ws.sm.workspace.impl.WorkSpaceImpl.synch(WorkSpaceImpl.java:549)    
        at com.ibm.ws.console.core.action.SyncWorkSpaceAction$1.run(SyncWorkSpaceAction.java:284)    
        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5574)    
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5700)    
        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)    
        at com.ibm.ws.console.core.action.SyncWorkSpaceAction.execute(SyncWorkSpaceAction.java:288)    
        at org.apache.struts.action.RequestProcessor.processActionPerform(Unknown Source)    
        at org.apache.struts.action.RequestProcessor.process(Unknown Source)    
        at org.apache.struts.action.ActionServlet.process(Unknown Source)    
        at org.apache.struts.action.ActionServlet.doGet(Unknown Source)    
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)    
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)    
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)    
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)    
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)    
        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)    
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)    
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)    
        at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistance(WSCUrlFilter.java:971)    
        at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:518)    
        at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:339)    
        at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)    
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)    
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:969)    
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1109)    
        at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4217)    
        at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(WebAppImpl.java:2208)    
        at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)    
        at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1030)    
        at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)    
        at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)    
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)    
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)    
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)    
        at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)    
        at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)    
        at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)    
        at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)    
        at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)    
        at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)    
        at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)    
        at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)    
        at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)    
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)    
    Caused by: javax.ejb.EJBTransactionRolledbackException: nested exception is: javax.transaction.RollbackException    
        at com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapCSIException(BusinessExceptionMappingStrategy.java:152)    
        at com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapCSITransactionRolledBackException(BusinessExceptionMappingStrategy.java:613)    
        at com.ibm.ejs.container.EJSDeployedSupport.mapCSITransactionRolledBackException(EJSDeployedSupport.java:609)    
        at com.ibm.ejs.container.EJSContainer.postInvokeRolledbackException(EJSContainer.java:4392)    
        at com.ibm.ejs.container.EJSContainer.postInvokeForLifecycleInterceptors(EJSContainer.java:4584)    
        at com.ibm.ejs.container.SingletonBeanO.callTransactionalLifecycleInterceptors(SingletonBeanO.java:248)    
        at com.ibm.ejs.container.SingletonBeanO.initialize(SingletonBeanO.java:330)    
        at com.ibm.ejs.container.BeanOFactory.create(BeanOFactory.java:105)    
        at com.ibm.ejs.container.EJSHome.createSingletonBeanO(EJSHome.java:3738)    
        ... 101 more    
    Caused by: javax.transaction.RollbackException    
        at com.ibm.tx.jta.impl.TransactionImpl.stage3CommitProcessing(TransactionImpl.java:1279)    
        at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:1053)    
        at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:974)    
        at com.ibm.ws.tx.jta.TranManagerImpl.commit(TranManagerImpl.java:439)    
        at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:191)    
        at com.ibm.ejs.csi.TranStrategy.commit(TranStrategy.java:866)    
        at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:188)    
        at com.ibm.ejs.csi.RequiresNew.postInvoke(RequiresNew.java:118)    
        at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:482)    
        at com.ibm.ejs.container.EJSContainer.postInvokeForLifecycleInterceptors(EJSContainer.java:4576)    
        ... 105 more    
    [11/27/18 9:28:14:051 EET] 0000011a FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\programs\IBM\WebSphere\AppServer\profiles\profile1\logs\ffdc\Server1_ede9c292_18.11.27_09.28.14.0481845564063299454983.txt com.ibm.ws.runtime.component.EJBContainerImpl.stateChanged 6637    

Answer 1

HomeOfHomes.resolveEJBLink is normally used for @EJB annotations that are resolved locally; an InitialContext is not used, nor is a JNDI lookup performed. The code processing the @EJB skips JNDI entirely and locates the EJB directly in the EJB Container. The fact the failure occurs after a remote lookup could just be a coincidence.

I recommend looking at the beanName parameter of the @EJB and verifying that the beanName does in fact match a bean name in the same application. If the beanName parameter is of the format myModule.jar#SomeOtherBean or myModule/SomeOtherBean, then make sure the module name is correct as well. The names myModule and SomeOtherBean must match the names on one of the CNTR0167I messages in SystemOut.log, in the same application.

For WebSphere, resolving an EJB based on the beanName parameter only works within the same application. If the @EJB is referencing an EJB in a different application, even if that application is on the same server process, then you must provide a binding that identifies the JNDI name of the referenced EJB. The binding would be specified in one of the files, ibm-ejb-jar-bnd.xml/xmi or ibm-web-bnd.xml/xmi. A binding may also be provided during application deployment (and the deployment would add it to the appropriate *bnd.xml file).

If the beanName parameter does appear to be correct, and a JNDI lookup is occurring for the @EJB annotation, then you'll need to provide the full exception stacks for both the client and server side of the lookup, to better understand how the @EJB processes was re-directed to the wrong server. If this is occurring, then the EJBNotFoundException should appear in the Server2 logs, and then flow back to Server1, where it would be reported again; so exception stacks from both server1 and server2 would be needed to identify how the lookup was incorrectly routed.