Apache Spark : Spring Transaction Not commiting

441 views Asked by At

I am using Spark and Spring JDBC Template class for commiting in DB. DB is MS SQL Server. From Apache spark, mapToParition sends data in batches to DB via Springs batchUpdate in a transaction. The transaction gets completed but data doesn't get written to DB.

Does anybody know whats wrong here ?

          .mapToPartition(....) {
             call(..) {
               DefaultTransactionDefinition paramTransactionDefinition = new        
               DefaultTransactionDefinition();

               TransactionStatus status =  
               transactionManager.getTransaction(paramTransactionDefinition );
               .....
               ..... code for sql.......
               .....
               _jdbcTemplate.batchUpdate(finalSql, jdbcArgs);
               transactionManager.commit(status);
             } // call ends
          }//mapToPartition ends

Here status.isCompleted() returns true.

This code works perfectly fine if I run in local[*] mode of apache spark, but when I run same code in Apache Spark's clustered/Distributed mode with 3 workers then it doesn't write any data to DB.

0

There are 0 answers