Upload multiple files to SFTP from Node

Question

I'm trying to upload multiple files to my SFTP server but not having any luck. I've seen this example:

how do I send (put) multiple files using nodejs ssh2-sftp-client?

But this does not seem a very efficient method as it opens and closes the connection to the server on every file upload. Just to give some context, my app could potentially try to upload many thousands of files. I am sending an array of filenames from the client to server and calling SFTP.get to get the file from a remote location, and SFTP.put that file stream back into another remote location. This is being done with the ssh2-sftp-client npm module.

I have attempted to upload multiple files using a few different methods which have failed.

Firstly:

for (var i=0; i<10; i++) {

      sftp.get('filepath/testFile.csv').then((stream) => {
            sftp.put(stream, 'newfilepath/testFile' + i + '.csv');
      })

}

This was just an attempt at fetching the same file and uploading it 10 times under a different file name each time. However, when I look at my server:

Only the last file was uploaded - When looking at the logs it actually just attempted to upload the testFile10.csv 10 times rather than testFile1.csv, testFile2.csv etc...

Secondly:

var files = ['testFile.csv', 'testFile2.csv']

    for (var i=0; i<2; i++) {

        sftp.get('filepath/' + files[i]).then((stream) => {
            sftp.put(stream, 'newfilepath/testFile' + i + '.csv');
        })

    }

This was an attempt at simply uploading 2 files. However, like with the last example, only the last file was uploaded - In this case testFile2.csv.

Any suggestions how I could get this to work? Is it even possible to PUT multiple files in a for loop? It doesn't seem like something that should be technically difficult I just don't understand why it is't even attempting to loop from 0 to 10 it's just trying to put testFile10 10 times.

This is quite a crucial technical hurdle which I need to overcome with a deadline so I would really appreciate any help you can give.

Many thanks in advance,

G

Answer 1

Because async callback runs when the promise is resolved, outside of the main execution order, when it runs, i is already equals with the array's length. Therefore it uploads only the last file.

So you can do it with promises, like this (I can't check it right now):

const files = ['testFile.csv', 'testFile2.csv'];

const promises = files.map(async file => {
  const stream = await sftp.get(`filepath/${file}`);
  sftp.put(stream, file);
});

Promise.all(promises)
  .then(results => {
    console.log('all files have been uploaded');
    // do something
  })
  .catch(e => {
    console.error(e);
    // handle error
  });