How to upload huge number of xml files to ADLS or DBFS from UI or CLI in a faster way. Any suggestions would be really appreciated?

Question

I have around 50 k xml files which I wanted to upload to ADLS/DBFS. Can this can be uploaded as a zip file and work on zip file from ADLS/DBFS. As of now its taking so much of time for uploading into ADLS/DBFS. I use dbfs cp -r which is not effective in this case. Also trying to upload through Databricks UI. Same case when trying to upload through storage explorer to ADLS Looking forward to valid suggestions:)

Answer 1

You can use AzCopy or Azure Storage Explorer to upload to ADLS. They use multiple upload threads and may end up being faster than dbfs cp -r which uploads files one by one.