WSO2 streaming integrator - read large files (one million of records) and write to another file

Question

I want to read one million records from a CSV file and write to another CSV file with WSO2 streaming integrator. The problem is the slow speed of reading and writing data, almost 10 per second. WSO2 says "Read a 100GB file with 10 million lines in just 20 mins" but I found no sample or guide to do that. can anyone help me?

Answer 1

Finally I found some example here https://ei.docs.wso2.com/en/7.2.0/streaming-integrator/ref/performance-analysis-results/ and with these example my speed is ok.

batch.size.max and buffer.size are useful parameters here.