Challenges with On-the-Fly ZIP File Downloads: Seeking Solutions for Files Beyond 4GB using python

Question

ZIP File Size Limitation: The issue involves dynamically generating and downloading ZIP files in the browser using the aiozipstream package in Python. However, a limitation is encountered where files larger than 4GB cannot be successfully downloaded.

Alternative Attempt: As an alternative, the zipfile package was explored, but it lacks support for on-the-fly downloads. Instead, it initiates the download only after the entire file is prepared.

Compatibility Issue with macOS Default Extractor: The method faces compatibility issues with the default extractor in lower versions of macOS (Archive Utility.app). However, using "The Unarchiver.app" resolves the problem.

pip install aiozipstream

from django.http import StreamingHttpResponse
from zipstream.zipstream import ZipStream
import httpx
from urllib.parse import quote
from rest_framework.response import Response
from rest_framework import status


urls=[
    ("s3/..../.com", "test.wav"),
    ("s3/...../.com", "test.wav"),
]
z_file_name = "test.zip"

def file_content_generator(url, client):
    with client.stream('GET', url, timeout=None) as r:
        yield from r.iter_bytes()

def download(self, request):
    client = httpx.Client()
    files=[]
    for url, file_name in urls:
        files.append({'stream': file_content_generator(url, client), 'name': file_name})

    zf = ZipStream(files, chunksize=6400000)
    if zf:
        response = StreamingHttpResponse(zf.stream(), content_type='application/zip')
        file_expr = "filename*=utf-8''{}".format(quote(z_file_name))
        response['Content-Disposition'] = 'attachment; {}'.format(file_expr)
        return response
    else:
        return Response(status=status.HTTP_403_FORBIDDEN)```


aiozipstream Attempt: The attempt to dynamically generate and download ZIP files using aiozipstream has a limitation for files exceeding 4GB. The expectation is to find a solution or alternative method for on-the-fly downloads of larger files.

zipfile Attempt: The exploration of the zipfile package was aimed at finding an alternative solution for on-the-fly downloads. However, it was found that this package does not meet the requirement of initiating downloads dynamically.

Compatibility Check: The attempt to use the default extractor in lower versions of macOS (Archive Utility.app) was unsuccessful. The expectation was to identify the reason for this lack of compatibility, which was resolved by switching to "The Unarchiver.app." Seeking an understanding of the underlying reasons for the discrepancy.

Overall, the challenges revolve around file size limitations for on-the-fly downloads and compatibility issues with specific macOS extractors. The goal is to find effective solutions and insights into the identified problems.