We are going to read a file from SFTP and put employees in database.
Following is XML Structure
<employees>
<employee></employee>
<employee></employee>
<employee></employee>
<employees>
The strategy i think is:
- Pick XML file from SFTP ->
- Fetch Employee No. from XML -> (24000 Employees)
- Fetch Data from "System 1 and System 2" on base of employee no. in xml file all together in memory ->
- Split Employee from XML ->
- Assign Data from "System 1 and System 2" to Each employee Exchange ->
- Put Each employee XML to JMS Queue (Read in clustered envionment)
Is there any other best strategy to handle this scenario in Apache Camel in abve route or route 2 (Jms Queue)?
Note: As it is clustered environment; we can't fetch data from System1 & System2 in servers itself then we need to keep tracking by assigning batch id to one file contents. You can think this scenario in other integrations too
The above steps can be improved by
If step 3 (fetch data from "System 1 and System 2") can be done concurrently, launch multiple application instance with idempotent (leverage external cache or DB) might help for lots of files.