Delete all objects in S3 bucket except one which comes last in lexicographical order group by a prefix

156 views Asked by At

Consider an S3 bucket with objects with keys like:

abc_1_epoch1.ext
abc_1_epoch2.ext
abc_2_epoch1.ext
xyz_1_epcoh1.ext

When i group by keys with prefix, then due to epoch it forms a lexicographical order. I want to delete all objects except the one which comes last in lexicographical order. So, the expected output in the bucket after cleanup task is:

abc_1_epoch2.ext
abc_2_epoch1.ext
xyz_1_epoch1.ext

As you can see grouping keys are abc_1, abc_2 and xyz_1. Point to note is that i have multi-million such objects in the bucket and hence I want a scalable solution.

0

There are 0 answers