Filter out files in 3 levels hierarchy

71 views Asked by At

I'm using the Azure File Share API to retrieve files stored in a 3 levels hierarchy /<user>/<machine>/<datafile>:

+
|
+- albert
   |
   +--- pc1
   |    |
   |    +--- 20010101.dat
   |    |
   |    +--- 20010101.dat
  ...  ...
   |    |
   |    +--- 20231001.dat
   +--- pc2
+- james
+- john

I'm looking for files at a specific date, so iterating over users and machines folders:

foreach (var userFolder in rootFolder.GetSubdirectoriesClients())
{
    foreach (var machineFolder in userFolder.GetSubdirectoriesClients())
    {
        var dataFileName = $"{date:yyyyMMdd}.dat";

        var dataFileItem = machineFolder.GetFilesAndDirectories(new ShareDirectoryGetFilesAndDirectoriesOptions { Prefix = dataFileName }).SingleOrDefault();

        if (dataFileItem != null)
        {
            // Process file
        }

All is working fine for some time and then all of a sudden the call to machineFolder.GetFilesAndDirectories freezes and after 1min40s (100s) I get a System.Threading.Tasks.TaskCanceledException with message "The operation was canceled.".

But it is caught somewhere in between and does not bubble up to my code, and the process continues, managing to do more API calls, then freezes again.

Maybe I could avoid this issue by making less or different API calls.
Ideally I would get the whole list of files from the root in a single call, then prune it to keep only the files I want, but I've not seen such API entry-point.

(Note that this is already my second attempt, the first one being with:

var dataFile = machineFolder.GetFileClient(dataFileName);

if (dataFile.Exists())
{
    // ...
}

Same issue but on the call to Exists instead.)

0

There are 0 answers