- Step 1- This is X job that creates the (b) job.dat file
- Step 2- This is an SSIS package that splits the output dat file into 4 different files to send to Destination
- Step 3-Moves the four files from the workarea to another location where MOVEIT can pick them up from
***Step two is not restartable
***There is no reversing out if any of the step fails
Note: what if i add exception handler or should I add condional split... any other ideas ?
Batch Persistence
One thing you can do for starters is to append the file(s) names with a timestamp that includes the date time of the last record processed (if timestamps do not apply then you can use a primary key incrementing value). The batch identifiers could also be stored in a database. If your SSIS package can smartly name the files in chronological sequence then third step can safely ignore files that is has already processed. Actually, you could do that at each step. This would give you the ability to start the whole process from scratch, if you must do it that way.
Ignorant and Hassle Free Dumping
Another suggestion would be do dump all data each day. If the files do not get super large then just dump all data for whatever it is you are dumping. This way each step would not have to maintain state and the process could start/stop at anytime.