Node.JS on IBMi / OS400: filename with accents non accessible

Question

I have a NodeJs (14 & 18) service which list and read files from the local filesystem, on IBMi or AS/400...

I use:

• await fsPromises.readdir(mydir) for to list all files of my directory
• await fsPromises.readFile(filename) for to read the content...

For filenames without accent, there is no problem... But, when there are some french accents, I have several problems:

• readdir(mydir) return name_��.png instead of name_éè.png ... But I resolve this first problem with the encoding parameter, and readdir(mydir, {encoding: 'binary'}) return the good name !

• calling await fsPromises.readFile('name_éè.png') or await fsPromises.readFile('name_éè.png', {encoding: 'binary'}) causes an access exception: name_��.png: ENOENT: no such file or directory

It seems that the "encoding" option only applies to the result, and not to the filename parameter. This problem does not seem to exist on a Windows machine, but seems specific to AS/400.

Context: Software version: tested with NodeJS v 14.21.3, and OS400 V7R3

Usually it is started via a CL which contains the command

CMD(QSH CMD(&CMD)) JOB(&JOB) JOBQ(QUSRNOMAX) CCSID(1147) SPLFACN(*DETACH) 

where &CMD is the path of the SH script ( node app.js &) and &JOB is the service's name...

I also tested it whithout the CCSID(1147) parameter, but I had the same problem... (finally, it doesn't mean anything because it's the system default...)

Any idea to resolve this ? Thanks

Answer 1

I finally solved my problem: it's a problem of character encoding difference between Node and the AS/400 job environment:

• in Node, all strings are utf8 encoded

• but OS400 is linked to the CCSID (1147 in my case) of the execution environment (of the job, of the user profile, etc.) and therefore the names of utf8 files must be converted as soon as they are transmitted to the "fs" primitives or "fsPromises"...

Concretely:

• I made a conversion function (because my Node service can also run on Windows or Linux machines,...)

   const os = require('os');
   
   let ENCODING_TYPE = 'utf8';
   if (os.type() == "OS400"){
     ENCODING_TYPE = 'binary'; // for CCSID 1147
   }
   
   function convertFilenameCharset(filename){
       if (os.type() != "OS400"){
           return filename;
       }
       const res =  Buffer.from(filename, 'binary'); // return the filename in a Buffer object !
       return res;
   }

• I use it in "fs" API calls...

    await fsPromises.unlink(convertFilenameCharset(fullfilepath));
    
    await fsPromises.rename(workfilepath, convertFilenameCharset(fullfilepath));
    
    let data = await fsPromises.readFile(convertFilenameCharset(fullFilePath));  

For read the list of elements of a directory, the result must also be re encoded:

let files = await fsPromises.readdir(convertFilenameCharset(abspath), {encoding: ENCODING_TYPE}); 

NOTE: it doesn't run for some API like await Jimp.read(convertFilenameCharset(filepath)) because they don't support Buffer parameter like the "fs" API: in those cases, I use the normalize-diacritics module to generate a new filename without accents, and I do a work copy of the file source...

Answer 2

Generally speaking, QSH is a bad choice for running PASE programs.

Think of QSH as a IBM native "Unix like" shell emulmation. The commands available are alias to actual *PGM objects.

When dealing with PASE tools and open-source packages specifically, a real Unix/Linux shell is preferred. For interactive usage, SSH from your client PC is preferred. If you absolutely have no choice to use 5250, then QP2TERM is a better choice than QSH.

For starting a batch job aka background service, the PASE shells QP2SHELL and QP2SHELL2 are possible options. Be sure to read and understand the notes for those commands.

However, the absolute best way to handle open-source services on the IBM i is via the open-source Service Commander

Lastly, if non of the above helps, you may want to reach out to other IBM i open source communities.

• IBM i Open Source Chat on Ryver (must first join at this link)
• IBM i Community Slack (must first join at this link)
• IBM Community pages for IBM i
• IBMiOSS LinkedIn group
• #IBMiOSS hashtag on Twitter
• Jesse Gorzinski’s personal Twitter