How to read the file from s3 with cdc in Java?

Question

I want to read the file from s3. Moreover, if the new files is created in the bucket, then the app know the changes and ingest them as cdc likes.

How to achieve this using Java? Can it be possible since I don't get any information on the Internet.

Note that using the getObject or listObject every time to track the new event is not the one I want because it looks very slow. So How to achieve it?

And also, I don't want to use any other 3rd party infra such as kafka.

I want to use only java application to consume data from s3 as well as the new event.

Any idea will be very appreciated.

Thanks,

Answer 1

Take a look here: S3 Events
There are bunch of CDC events you can use for your app.
Here is what you need to do:

1. Create S3 bucket in AWS Console.
2. Amazon SQS, Amazon SNS, and Lambda are available for evets receiving.
   • I recommend using Amazon SQS to ensure event delivery to your application. Configure SQS
3. Enable notifications on that bucket. How to enable notifications
4. Use AWS SDK for accessing S3 and SQS from your application: Accessing S3 and Receiving Messages from SQS

All S3 event types you can find here Supported notification event types