Spring Integration: Only Ingest New Files Since Last Poll

Question

I'm using Spring Integration to poll for files from an S3 bucket. I cannot delete the files after processing and so I added the S3PersistentAcceptOnceFileListFilter. I have a lot of files and poll frequently which is causing unneeded stress on the FILE_SYNC_METADATA_STORE table.

I'm wondering if there is a FileListFilter implementation that filters for files that are new/updated since the last poll, rather than having to query the DB for each file.

Answer 1

There is no such a FileListFilter, but you definitely can implement your own, for example saving the latest lastmodified of the S3Object in the filter and compare with it on the next poll. This way you indeed not going to the FILE_SYNC_METADATA_STORE table configured for the S3PersistentAcceptOnceFileListFilter since you won't pass to that filter. Although with the custom lastmodified filter you might not need an S3PersistentAcceptOnceFileListFilter at all...