Convert .csv file into .xlsx using streams

37 views Asked by At

I have node js server with a Postgres db. Im using pg-copy-streams to generate csv files and aws-sdk to upload them to S3. In other words, I have a pipeline that looks like this

      const passthrough = createPassthrough();
      pipeline(
        hereGoesMyPgCopyStream,
        passthrough,
        (err) => {
          if (err) {
            console.log("There is an error");
          } else {
            console.log("pipeline successful");
          }
        },
      );

      const params = {
       Bucket: bucketName,
       Key: 'uploaded_file.csv',
       Body: passthrough
      };


      s3.upload(params, (err, data) => {
        if (err) {
         console.error("Error uploading file:", err);
        } else {
         console.log("File uploaded successfully. Location:", data.Location);
        }
      }); 

It's working fine but now I need to upload not csv but xlsx files. I would like to keep using streams so I was thinking of doing something like

      const passthrough = createPassthrough();
      pipeline(
        hereGoesMyPgCopyStream,
        csvToXlxsConverterStream,
        passthrough,
        (err) => {
          if (err) {
            console.log("There is an error");
          } else {
            console.log("pipeline successful");
          }
        },
      );

How can I implement the csvToXlxsConverterStream? All converting methods I've seen require the whole file. Is it even possible to do such conversion by chunks?

As I don't know much about the xlsx format I tried looking for libraries with this capability. The thing is, I'm not sure of how to do the convertion by chunks without resulting in a bad xlsx

0

There are 0 answers