Consider an S3 bucket with objects with keys like:
abc_1_epoch1.ext
abc_1_epoch2.ext
abc_2_epoch1.ext
xyz_1_epcoh1.ext
When i group by keys with prefix, then due to epoch it forms a lexicographical order. I want to delete all objects except the one which comes last in lexicographical order. So, the expected output in the bucket after cleanup task is:
abc_1_epoch2.ext
abc_2_epoch1.ext
xyz_1_epoch1.ext
As you can see grouping keys are abc_1, abc_2 and xyz_1. Point to note is that i have multi-million such objects in the bucket and hence I want a scalable solution.