TechQA.

Upload file bigger than php.ini's "upload_max_filesize" and "post_max_size" with XMLHttpRequest

133 views Asked by At

The following code is a very simple self-hosted file transfer service.
It works for files smaller than 100 MB.

How to make this work for 1 GB files or more? (which might exceed php.ini's upload_max_filesize and post_max_size)

More generally, how can we do a POST XMLHttpRequest file upload (with a file appended to a FormData object), with a file bigger than the server's RAM?

Is there a way to to do the XHR upload by chunks, and if so, how?

<?php
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    $localfname = $_POST['fname'];
    $data = file_get_contents($_FILES['data']['tmp_name']);
    $fname = '';
    for ($i = 0; $i < 4; $i++) {  
        $fname .= '0123456789abcdefghijklmnopqrstuvwxyz'[rand(0, 35)];  
    }     
    $trimmedfname = preg_replace('/\s+/', '', basename($localfname));
    $file = fopen('files/' . $fname . '_' . $trimmedfname, 'w');
    fwrite($file, $data);
    fclose($file);
    $link = 'files/' . $fname . '_' . $trimmedfname;
    echo 'link: <div id="link"><a href="' . $link . '">' . $link . '</a></div>';
    die();
}    
?>
<body>
<div id="container" style="width: 200px; height: 200px; background-color: #eee; padding: 1em;">drag and drop your file here!</div>
<script>
var $ = document.querySelector.bind(document);
var readfiles = files => {
    var formData = new FormData();
    formData.append('fname', files[0].name);
    formData.append('data', files[0]);
    $("#container").innerHTML = 'beginning ulpoad... progress: ';
    var xhr = new XMLHttpRequest();
    xhr.open('POST', '');
    xhr.onload = () => {
        $("#container").innerHTML =  xhr.responseText;
    };
    xhr.upload.onprogress = (event) => {
        if (event.lengthComputable) {
            $("#container").innerHTML = 'ulpoading... progress: ' + (event.loaded / event.total * 100 | 0) + '%<br>';
        }
    };
    xhr.send(formData);
}
document.body.ondragover = () => { $("#container").innerHTML = 'drop your file here...'; return false; };
document.body.ondrop = (e) => { e.preventDefault(); readfiles(e.dataTransfer.files); };
</script>
</body
php forms file-upload xmlhttprequest form-data
Original Q&A
1

There are 1 answers

2
VonC On BEST ANSWER

how to chunk such a XHR request?

True, chunking a file upload involves breaking down the file into smaller parts (chunks), sending each chunk separately, and then reassembling these parts on the server.

Client Browser                     Server
    |                                |
    |--- Upload Chunk 1 -----------> |
    |                                |--- Append Chunk
    |--- Upload Chunk 2 -----------> |
    |                                |--- Append Chunk
    |                                |
    |--- Upload Last Chunk --------> |
    |                                |--- Assemble Chunks
    |<-- Return Link to Uploaded File|

That would allow you to bypass the upload_max_filesize and post_max_size limits in php.ini because each chunk can be kept small enough to fit within these limits. But you need to modify both the client-side and server-side code to handle chunked uploads.

On the client-side, modify your JavaScript to slice the file into chunks and upload each piece sequentially (simplified example of a more complex process):

function uploadChunk(file, start, chunkSize, totalChunks) {
    var end = start + chunkSize;
    var chunk = file.slice(start, end);
    var formData = new FormData();
    formData.append('fname', file.name);
    formData.append('data', chunk);
    formData.append('chunkIndex', Math.floor(start / chunkSize)); // Current chunk
    formData.append('totalChunks', totalChunks);

    return new Promise((resolve, reject) => {
        var xhr = new XMLHttpRequest();
        xhr.open('POST', '', true);
        xhr.onload = function() {
            if (xhr.status === 200) {
                resolve();
            } else {
                reject(new Error('Upload failed with status: ' + xhr.status));
            }
        };
        xhr.send(formData);
    });
}

function chunkedUpload(file) {
    const CHUNK_SIZE = 1 * 1024 * 1024; // 1MB chunk size
    const totalChunks = Math.ceil(file.size / CHUNK_SIZE);

    let promise = Promise.resolve();

    for (let start = 0; start < file.size; start += CHUNK_SIZE) {
        promise = promise.then(() => uploadChunk(file, start, CHUNK_SIZE, totalChunks));
    }

    promise.then(() => {
        console.log('All chunks uploaded successfully.');
    }).catch((error) => {
        console.error('Error uploading chunks: ', error);
    });
}

The JavaScript function chunkedUpload(file) should be triggered with the file you want to upload. You can modify your drag and drop or file input listeners to use this function.

On the server side, modify your PHP script to handle chunked uploads. The script needs to append each chunk to a temporary file. Once all chunks have been received, it can then process the file as needed.
As a basic example (For a production environment, you should include error handling, security checks, and possibly support for concurrent uploads):

if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    $chunkIndex = $_POST['chunkIndex'];
    $totalChunks = $_POST['totalChunks'];
    $localfname = $_POST['fname'];
    $chunkData = file_get_contents($_FILES['data']['tmp_name']);
    
    // Create a unique temporary file name based on the original file name.
    $tempDir = sys_get_temp_dir();
    $tempFilePath = $tempDir . DIRECTORY_SEPARATOR . 'chunk_' . md5($localfname);

    // Append the current chunk to the temporary file.
    file_put_contents($tempFilePath, $chunkData, FILE_APPEND);

    // If this is the last chunk, process the file.
    if ($chunkIndex + 1 == $totalChunks) {
        // Move/rename the temporary file to its final destination, etc.
        // For example:
        $finalFilePath = 'files/' . uniqid() . '_' . basename($localfname);
        rename($tempFilePath, $finalFilePath);
        
        echo 'File uploaded successfully. Access it <a href="' . $finalFilePath . '">here</a>.';
    } else {
        echo 'Chunk ' . ($chunkIndex + 1) . ' of ' . $totalChunks . ' uploaded successfully.';
    }
    exit;
}

How would you call the JS code?
Something like document.body.ondrop = (e) => { e.preventDefault(); chunkedUpload(e.dataTransfer.files[0]); };?
Or would you call it another way?

Yes, your approach to calling the chunkedUpload function should work.
When a user drops a file onto the designated area, the event listener prevents the default action (which could be opening the file or navigating to it), retrieves the file from the dataTransfer object, and then passes this file to the chunkedUpload function for processing.

You can add some additional steps, to make sure a smooth user experience (a bit as in this example):

document.body.ondragover = (e) => {
    e.preventDefault();
    // Optionally, add some visual feedback for the user to indicate the drop target
    document.getElementById('container').style.backgroundColor = '#f0f0f0';
    document.getElementById('container').innerHTML = 'Drop your file here...';
};

document.body.ondragleave = (e) => {
    // Revert visual feedback when the file leaves the drop target area
    document.getElementById('container').style.backgroundColor = '#eee';
    document.getElementById('container').innerHTML = 'Drag and drop your file here!';
};

document.body.ondrop = (e) => {
    e.preventDefault();
    // Reset visual feedback once the file is dropped
    document.getElementById('container').style.backgroundColor = '#eee';
    // Clear the previous message
    document.getElementById('container').innerHTML = 'Beginning upload...';

    // Start the chunked upload process for the first file
    if (e.dataTransfer.files.length > 0) {
        chunkedUpload(e.dataTransfer.files[0]);
    } else {
        // If no files were dropped, inform the user
        document.getElementById('container').innerHTML = 'No file detected. Please try again.';
    }
};

That would provide feedback during the drag-over and drag-leave events, making it clearer to users where they should drop their files. Once the file is dropped, it proceeds with the upload and gives an immediate visual response that something is happening.

Related Questions in PHP

Related Questions in FORMS

Related Questions in FILE-UPLOAD

Related Questions in XMLHTTPREQUEST

Related Questions in FORM-DATA

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json

Trending Questions