Due to backend filesize limit of 5.5MB of the API, I need to chunk large files (only .csv, .xls,.xlsx) when uploaded into chunks of <= 5MB and send each chunk by calling file upload api. Each chunk should be send as file with input file extension. That is, if a large csv file is uploaded via input type file, it will splitted into N number of csv files of <= 5MB and then the API will be called N times.
Here is the front-end:
<form method="post" action="https://my_url/upload" enctype="multipart/form-data">
<label for="name" id="firmanavn">Customer Number:</label>
<input type="text" class="num-only" id="name" name="cvr" maxlength="8" pattern="\d{8}"
oninvalid="this.setCustomValidity('Need 8 digits')"
oninput="this.setCustomValidity('')" required>
<input id="fileupload" type="file" name="file"
accept=".csv,.xls,.xlsx,text/csv,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.ms-excel"
required />
<input type="submit" id="upload" value="Upload File">
</form>
Here is the backend made by FastAPI:
@app.post("/upload", status_code=status.HTTP_201_CREATED, response_class=HTMLResponse)
async def upload_file(cvr: str = Form(...), file: UploadFile = File(...)):
if file.content_type == 'text/csv' or file.content_type == 'application/vnd.ms-excel' or file.content_type == 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet':
try:
name = cvr + '_'+ datetime.date.today().strftime('%d_%m_%Y') + '_' + file.filename
f = file.file
res = email_vask_file_drive.put(name, f)
return """ <html> <body> File uploaded successfully </body> </html>"""
Furthermore, this is what I came to close with respect to front-end chunking which chunks but cannot be processed. Also I don't understand how to call my backend API properly as this code below was taken from the following code snippet (https://blog.logrocket.com/how-to-build-file-upload-service-vanilla-javascript/):
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>File Uploader</title>
</head>
<body>
<h2>File Upload Service</h2>
<input type="text" class="num-only" id="name" name="cvr" maxlength="8" pattern="\d{8}"
oninvalid="this.setCustomValidity('Need 8 digits')" oninput="this.setCustomValidity('')"
required>
<input id="fileupload" type="file" name="file"
accept=".csv,.xls,.xlsx,text/csv,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.ms-excel"
required />
<button id="upload">Upload</button>
<small id="status"></small>
<script>
const file = document.getElementById('file');
const upload = document.getElementById('upload');
const status = document.getElementById('status');
upload.addEventListener('click', () => {
// set status to uploading
status.innerHTML = 'uploading...';
const fileReader = new FileReader();
fileReader.readAsArrayBuffer(file.files[0]);
fileReader.onload = async (event) => {
const content = event.target.result;
const CHUNK_SIZE = 5 * 1000 * 1024;
const totalChunks = event.target.result.byteLength / CHUNK_SIZE;
// generate a file name
const fileName = Math.random().toString(36).slice(-6) + file.files[0].name;
for (let chunk = 0; chunk < totalChunks + 1; chunk++) {
let CHUNK = content.slice(chunk * CHUNK_SIZE, (chunk + 1) * CHUNK_SIZE)
await fetch('https://pays_email_vask_f_o_utils.deta.dev/upload', {
'method': 'POST',
'body': CHUNK
})
}
status.innerHTML = 'Uploaded!!!';
}
});
</script>
I know beside the main concern I portrait in the question. There are many other problems which I would be happy to get feedback but chunking and uploading chunked file with correct format is main problem here.
Please help!!!