How can I use a DataFusion to perform ETL operations when I have multiple data files (.txt) to convert? Using functions within the Datafusion UI

Question

The tasks are outlined below. Multiple data files within Google Cloud Storage(GCS) are stored in partitions(/directory01/directory02/.../.text)

I am going to use datafusion to carry out ETL work and load it on the bigQuery table.

ETL operation was performed only when one data file was selected.

However I want to process all of the data file in Google Cloud storage at once, but there is too much data file to do manually.

Please let me know if there is a good way.

Thank you.

Answer 1

As @narendra commented, you can specify the directory as the path in GCS plugin properties/settings and make sure the path ends with a "/".

You can find documentation in the "Documentation" tab to the right of "Properties" tab when you open the plugin properties/settings.