Expected functionality of the code:
- receive CSV file
- write it to a temporary directory
- convert that file to JSON by using csvtojson library
- add prefix
data =
to the file - create JS file as a result, and write it to a different directory
The process works fine with short CSV files. When larger files are sent (CSV files containing more than 20.000 rows), result data is cut to approximately 1800-2500 rows. When I start the app locally and upload file for the first time, even when uploading large files, everything works fine. Everything is also fine if I change anything in readStream part of the code and server restarts, so I'm guessing that read stream needs to be reset somehow. I used this documentation to implement that part.
EDIT 03-09-2020: Once deployed to the server, app manages to handle even large CSV files
router.post("/", upload.single("productData"), (req, res, next) => {
const product = new Product({
_id: new mongoose.Types.ObjectId(),
productData: req.file.path,
});
product
.save()
.then(() => {
let nameOfNewDirectory = req.body.dirName;
let dir = `./someDirectory/${nameOfNewDirectory}`;
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir);
}
const readStream = fs.createReadStream(req.file.path);
const writeStream = fs.createWriteStream(`./${dir}/data.js`, {
flags: "w",
});
writeStream.write("data = ", () => {
console.log("prefix to a data.js is written");
});
readStream
.pipe(
CSVToJSON({
maxRowLength: 65535,
checkColumn: true,
downstreamFormat: "array",
ignoreEmpty: false,
delimiter: "auto",
})
)
.pipe(writeStream);
})
});