How to download, then archive and send zip to the user without storing data in RAM and memory?

Question

I need to download photos from S3 via HttpClient, then archive it to zip without compression, and send the archive to the user. However I can't store the zip in RAM, and I cannot create temporary files, because I need to create very large archives (15-20 Gb of data).

I need a solution in C# ASP.

I tried this:

var response = HttpContext.Response;
        response.ContentType = "application/octet-stream";
        response.Headers.Add($"Content-Disposition", $"attachment; filename=\"ololo.zip\"");

        await using var outputStream = response.Body;

        using var zipStream = new ZipOutputStream(outputStream);
        zipStream.SetLevel(9);

        for (int i = 0; i < fileUrls.Count; i++)
        {
            var fileName = $"file{i + 1}.jpg";
            var zipEntry = new ZipEntry(fileName);
            zipStream.PutNextEntry(zipEntry);

            using var httpClient = new HttpClient();
            using var remoteStream = await httpClient.GetStreamAsync(fileUrls[i]);

            await remoteStream.CopyToAsync(zipStream);
            zipStream.CloseEntry();
        }

        zipStream.IsStreamOwner = false;
        await zipStream.FinishAsync(default);

        return new EmptyResult();

But I had errors with async operations. And I tried this:

[HttpPost]
public async Task Zip([FromBody] JsonToZipInput input)
{        
  Response.ContentType = "application/octet-stream";
  Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");

  await Response.StartAsync();
    
  using var zipArchive = new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
  foreach (var (key, value) in input.FilePathsToUrls)
  {
    var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
    await using var zipStream = zipEntry.Open();
    await using var stream = await _httpClient.GetStreamAsync(value);
    await stream.CopyToAsync(zipStream);
  }
}

But it loads all archive into the RAM.

Answer 1

You will have to roll your own here. You will need to follow the PKWare appnote with the zip specification. You can look at zipflow, in C, for guidance. Or you can try to convert zipflow to C#. zipflow streams a zip file out without having to store the whole thing in memory or in the file system.