Package files together on Amazon S3

Question

I have a base file which is to be packaged inside another file. At every request, we append/merge the license file with the original base file and give client a URL to the newly created packaged file (Base file + license). Currently, we download file from s3 into ec2 instance, merge and then upload the same back to s3 bucket. Is there a way to manage the same?

P.S. Base file is of large sizes and the license file is smaller in size.

Problem Statement: We need to create a package from a collection of products to be given as a single file to user. Currently, we download all products from s3 onto our ec2 instance, add license file, zip them into a new package file and upload the "packaged" file back to s3 and share the link with customer. What I want or atleast intend to do is instead of downloading individual files and then uploading the large file back, what if I can just upload the license file and give reference of all individual files so that the packaged files is created and saved at s3. it saves to-and-fro of data from S3

Answer 1

Amazon S3 does not have the capability to "package" files. The files would need to be created somewhere and uploaded to Amazon S3.

One way of doing this would be to use an AWS Lambda function, but the size of the files are too large for the storage space available in Lambda.

Therefore, you would need to perform these activities on an Amazon EC2 instance or in a container (eg Amazon ECS).

Of course, an easier way would be to provide TWO files to the customer - one with the application, the other with the license file.

Answer 2

the packaged files is created and saved at s3

It won't happen somewhere itself, you want some logic executed and S3 is an object storage, S3 has no such capability.

Consider what is actually the problem you want to solve? That you need to run an EC2 compute resource all the time?

In that case I see a few more options:

1. Using AWS Lambda - which is not really intended to run for a long time or store any data. (I believe there is a limit to 15 minutes). However - you may repackage and add a file to the archive in a streaming mode (not storing anything locally). Though - this approach assumes you will finish your task in the limited time

2. AWS Batch - if the repackaging or processing takes more time or needs more resources (storage), you may create a Batch job definition. AWS Batch will create the compute resources, process the defined job and clear out when done.