I'm working on a Java Spring Boot application that needs to perform calculations based on a substantial amount of master data, approximately 9 million entries. Ensuring the master data corresponds to a specific state as a unit is critical. We can't have one calculation working with one version of the master data while another process operates on an older state of the same data.
The straightforward approach would be treating the process like a large transaction, but due to the size of the dataset, keeping the database connection open for that long isn't feasible (I'm currently using an Oracle 12 database and hibernate as an ORM).
One option is functional locking, where all business data is locked until the batch process completes. However, this would disrupt user workflows. Another idea is maintaining multiple versions of the data, ensuring each user session works with a consistent version (possibly using Hibernate Envers).
What other solutions might address this issue? Have you encountered similar challenges before? I haven't committed to any batch framework yet, but I'm considering Spring Batch or Quartz.