glassfish throwing org.postgresql.xa.PGXAException

Question

I am not really familiar with J2EE so I might make some mistakes while explaining the error. Please bear with me.

I am trying to run a query over my java enterprise application but glassfish throws the following exception:

[#|2014-12-05T15:31:00.412+0200|WARNING|glassfishv3.0|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=86;_ThreadName=Thread-1;|JTS5031: Exception [java.lang.RuntimeException: org.postgresql.xa.PGXAException: Error preparing transaction] on Resource [prepare] operation.|#]

[#|2014-12-05T15:31:00.413+0200|SEVERE|glassfishv3.0|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=86;_ThreadName=Thread-1;|JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.|#]

[#|2014-12-05T15:31:00.439+0200|WARNING|glassfishv3.0|javax.enterprise.system.container.ejb.com.sun.ejb.containers|_ThreadID=86;_ThreadName=Thread-1;|A system exception occurred during an invocation on EJB OFReportTimeoutService method public void com.companyname.appname.service.OFReportTimeoutService.ofTimeout()
javax.ejb.EJBException: Unable to complete container-managed transaction.
    at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:4962)
    at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4716)
    at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1941)
    at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1892)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:198)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:84)
    at com.sun.proxy.$Proxy307.ofTimeout(Unknown Source)
    at com.companyname.appname.service.__EJB31_Generated__OFReportTimeoutService__Intf____Bean__.ofTimeout(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.companyname.appname.servlet.GFv3EJBInvokerJob.execute(GFv3EJBInvokerJob.java:88)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.  vmcid: 0x0  minor code: 0  completed: No
    at com.sun.jts.jta.TransactionManagerImpl.commit(TransactionManagerImpl.java:330)
    at com.sun.enterprise.transaction.jts.JavaEETransactionManagerJTSDelegate.commitDistributedTransaction(JavaEETransactionManagerJTSDelegate.java:169)
    at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:843)
    at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:4951)
    ... 14 more
|#]

Another system with the same versions of glassfish, postgresql, and web application does not throw any exception. They both have the same domain.xml and postgresql.conf files.

I have changed max_prepared_transactions from 100 to 10000 and shared_buffers from 32MB to 320MB but it did not work.

Any ideas?

EDIT: ofTimeout code added.

@Startup
@Singleton(mappedName="OFReportTimeoutService")
public class OFReportTimeoutService {

    @EJB
    protected QueueManagerService queueManagerService;

    @EJB
    protected AnalyzerService analyzerService;

    @EJB
    protected  ErrorReportJpaController errorReportJpaController;

    @EJB
    protected ReportJpaController reportJpaController;

    protected Boolean isProcessing = Boolean.FALSE;

    private Boolean exceptionPresent = Boolean.FALSE;
    private String errorText = "error";
    private Integer reportId = 0;

    //@Schedule(second="*/10", minute="*", hour="*", persistent=false)
    public void ofTimeout() {
        //System.out.println("STATE : " + "OfReportTimeOutService is running...") ;
        if(exceptionPresent) {
            ErrorReport errorReport = errorReportJpaController.create();
            errorReport.setErrorDate(new Date());
            errorReport.setErrorText(errorText);
            errorReport.setReport(reportJpaController.find(reportId));
            errorReportJpaController.persist(errorReport);
            exceptionPresent =  Boolean.FALSE;
            if (queueManagerService.getLastReport() != null)
                queueManagerService.resetLastReport();
            isProcessing = Boolean.FALSE;
        }

        if (isProcessing)
            return;

        if (queueManagerService.reportQueueSize() > 0)
            isProcessing = Boolean.TRUE;
        else {
            //System.out.println("STATE : " + "queueManagerService.reportQueueSize == 0 !!!") ;
            return;
        }        

        while (queueManagerService.reportQueueSize() > 0)
            try {
                Report report = queueManagerService.pullReport();
                reportId = report.getId();
                if ( reportId != null )
                    System.out.println("STATE : " + "reportId var") ;
                analyzerService.process(report);
            } catch (ReportJPAException rex) {
                Logger.getLogger(OFReportTimeoutService.class.getName()).log(Level.SEVERE, null, rex);
                exceptionPresent = Boolean.TRUE;
                errorText = rex.toString();
                break;
            } catch (RuntimeException rex) {
                Logger.getLogger(OFReportTimeoutService.class.getName()).log(Level.SEVERE, null, rex);
                exceptionPresent = Boolean.TRUE;
                errorText = rex.toString();
                break;
            } catch (Exception ex) {
                Logger.getLogger(OFReportTimeoutService.class.getName()).log(Level.SEVERE, null, ex);
                exceptionPresent = Boolean.TRUE;
                errorText = ex.toString();
                break;
            }

        if(exceptionPresent) {
            return;
        }

        isProcessing = Boolean.FALSE;
        queueManagerService.resetLastReport();


    }

    public Boolean isProcessing() {
        return isProcessing;
    }

    public void setProcessing(Boolean isProcessing) {
        this.isProcessing = isProcessing;
    }

}

Answer 1

I have solved the problem. We are using bucardo for replication of our postgresql database.This was the cause of my problem. In postgresql logs I saw an error log like this:

ERROR:  cannot PREPARE a transaction that has executed LISTEN, UNLISTEN or NOTIFY

In this blog post the cause of the problem has been explained:

The problem is that the Postgres LISTEN/NOTIFY system cannot be used with prepared transactions. Bucardo uses a trigger on the source tables that issues a NOTIFY to let the main Bucardo daemon know that something has changed and needs to be replicated. However, their application was issuing a PREPARE TRANSACTION as an occasional part of its work. Thus, they would update the table, which would fire the trigger, which would send the NOTIFY. Then the application would issue the PREPARE TRANSACTION which produced the error given above. Bucardo is setup to deal with this situation; rather than using notify triggers, the Bucardo daemon can be set to look for any changes at a set interval. The steps to change Bucardo's behavior for a given sync is simply:

The solution on the blog post did not work for us. We could afford not replicating the database causing the error. So we removed the replication with the following commands:

[root@Baskan config]# bucardo deactivate synclrms
Deactivating sync synclrms

[root@Baskan config]# bucardo purge synclrms
Purging name synclrms

[root@Baskan config]# bucardo remove sync synclrms
Removed sync "synclrms"
Note: table triggers (if any) are not automatically removed!

As the triggers are not removed automatically, they should be removed manually: In our case there were three triggers. Their names were: bucardo_delta, bucardo_kick_synclrms, bucardo_note_trunc_synclrms bucardo_note_trunc_synclrms

To drop a trigger the following command is used:

drop TRIGGER trigger_name on table_name;

Just in case there may be other triggers on a table placed by bucardo, you can use the following command in postgresql to see all the triggers on a table:

\dS table_name;

After these steps, the system started working normally without throwing any exceptions.