How to split a file in spring batch based on no of lines?

Question

I have a file like this: abc.txt

1 fhdfjdkslk
2 jskslsss
3 iowiwjwm
4 jwkwlwo
5 uuwwwmwaa
6 sbnsbshj
7 yywuiwoiw
8 dhdhgsgs
9 sgsgsgs
10 uwuwqmksam

I want to split this file using Spring Batch in a configurable way and read the file in a multi-threaded way.

For example if we split by 4 lines.

Thread 1 will process

1 fhdfjdkslk
2 jskslsss
3 iowiwjwm
4 jwkwlwo    

Thread 2 will process

5 uuwwwmwaa
6 sbnsbshj
7 yywuiwoiw
8 dhdhgsgs

Thread 3 will process

9 sgsgsgs
10 uwuwqmksam

Finally the output of all the 3 threads will be consolidated in the last step of the Spring Batch job.

I have been able to achieve this if each set contains only 1 line.

How can I achieve this if each set contains multiple lines in Spring Batch ?

Answer 1

You could write your own SystemCommandTasklet. This tasklet is used to execute a system command. You could use the split command to split the files. Subsequently you can use partitioner to execute the processing of separate files.

More information about this tasklet Here