Why does STD.FILE.LogicalFileList() results not completely correlate with STD.FILE.GetLogicalFileAttribute(x,y)?

Question

I execute the statement:

NrmFiles     := NOTHOR(STD.File.LogicalFileList());

This produces a dataset of all the normal files contained in my HPCC environment. It has the record layout:

    string lib_fileservices__fslogicalfilename := string{maxlength(255)};

    record__0 := RECORD
       lib_fileservices__fslogicalfilename name;
    END;
    FsLogicalFileInfoRecord := RECORD(record__0)
      boolean  superfile;
      integer8 size;
      integer8 rowcount;
      string19 modified;
      string   owner{maxlength(255)};
      string   cluster{maxlength(255)};
    END;

For my purpose, I need the workunit that created/modified each file, so I PROJECT that file which produces a dataset of all the normal files and the WUID appended:

NrmFiles             := dataset('~temp::allfilelist::normal', FsLogicalFileInfoRecord, THOR);
/*--------------------------------------------------------------------------------------------------
| retrieve the workunit ID for each filenme
| The CATCH function executes the action (set the value returned to 'NOTFND') if the 
| GetLogicalFileAttribute expression fails for any reason
|------------------------------------------------------------------------------------------------*/
FN_Retrieve_WUID(string pFname) := FUNCTION
   RETURN CATCH(STD.File.GetLogicalFileAttribute('~'+TRIM(pFname),'workunit'), 'NOTFND');
END;
dsNormFilesWUID      := PROJECT(  NrmFiles
                                , TRANSFORM(  {FsLogicalFileInfoRecord; string WUID;}
                                            , SELF.WUID := FN_Retrieve_WUID(LEFT.name);
                                              SELF      := LEFT;
                                          ));
output(dsNormFilesWUID, ,'~temp::allfilelist::normal::wuid', compressed, overwrite, expire(30));

PROBLEM It was failing to find some files when executing the GetLogicalFileAttribute. That's why I placed a CATCH() around the statement and it would return a 'NOTFND' instead of failing. BTW, I had to execute this on an hthor because if I executed on a multi-node THOR, it would fail with a "No access to Dali - this normally means a plugin call is being called from a thorslave". And wrapping the call in a NOTHOR() would get me a different failure: "NOTHOR expression 'fn_retrieve_wuid' appears to access a parent dataset - this may cause a dataset not active error"

QUESTION Does the STD.File.LogicalFileList() retrieve its information from the DFU (Distributed File Utility)? Does the STD.File.GetLogicalFileAttribute(logicalfilename, attrname) retrieve its information from the DALI (based on the file's metadata)?

I believe these statements are correct (based on the type of data returned). That does raise a concern however that the DFU and DALI can get out of sync. I mean on the same BWR / node I can get a list of files and when requesting an attribute based on a file name from that list, that file name might not be found.

Answer 1

The answer to your QUESTION is: Both functions get all their metadata results from Dali.

Your statement, "It was failing to find some files when executing the GetLogicalFileAttribute." is the only thing here that troubles me. I tried writing some test code:

IMPORT Std;
NrmFiles := STD.File.LogicalFileList();
NrmFiles;
NewRec := {NrmFiles,STRING20 WUID};
PROJECT(NrmFiles,
        TRANSFORM(NewRec,
                  SELF.WUID := STD.File.GetLogicalFileAttribute('~'+LEFT.name,'workunit'),
                  SELF := LEFT)); 

Because this job is only looking at DFU metadata, there's no reason to ever run it on a Thor, so it should always be run on hThor. My code runs perfectly on my test system, always finding all the logical files I have, whether they were OUTPUT-generated or sprayed.

I suggest that you submit a JIRA ticket (https://track.hpccsystems.com) to report this issue. Before you do that, you should also explore exactly which files are NOT getting picked up by the STD.File.GetLogicalFileAttribute() function to see if you can detect a reason why. It might be possible that the scope checking is preventing access, but probably not very likely.