In Foundry, is it possible to extract the original file names from a concatenated import?

Question

I need to import billions of very small image files into Foundry. I understand Foundry won't handle that amount of files very well, so best practice would be to concatenate them and make the import incremental.

However, I need to be able to get the raw files and their names to be able to process them in my pipeline. How can I get the original files back from the concatenated file?

Answer 1

The original file names, sizes, and modified dates are included in the transaction metadata:

Assuming you'll need data across multiple transactions, you likely want to do this as incrementally as possible.