I am uploading my database results as text file to S3 cluster. Now I need to split this file into multiple files in S3 based on column names combination. After that the individual files will be given to a service which will do some preprocessing on this individual files in parallel. After which they will be combined together as a single file and stored back to s3.
Any suggestions on how to do this?