passing additional values to s3 event notification for lambda consumption

1.4k views Asked by At

I have to write code in react-native that allows a user to upload videos to amazon s3 to be transcoded for consumption by various devices. For the processing after the upload occurs; I am reviewing two approaches:

1) I can use Lambda with ffmpeg to handle the transcoding immediately after the uploading occurs (my fear here would be the amount of time required to transcode the videos and the effect on pricing if it takes a considerable amount of time).

2) I can have s3 pass an sns message to a rest api after the created event occurs and the rest api generate a rabbitmq message that will be processed by worker that will perform the transcoding using ffmpeg.

Option 1) seems to be the preferable option based on a completion time perspective. How concerned should I be with using 1) considering how long video transcoding might take as opposed to option 2)?

Also, regardless, I need a way to pass additional parameters to lambda or along the sns messaging that would allow me to somehow associate the user who uploaded the video with their account. Is there a way to pass additional text-based values to s3 to pass along to lambda or along sns when the upload completes, as a caveat I plan to upload the video directly to s3 using the rest layer(found this answer here: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html#RESTObjectPUT-responses-examples)?

1

There are 1 answers

2
Mark B On BEST ANSWER

AWS provides a video transcoding service for exactly this type of thing. If you don't want to do that for some reason then you need to make sure you can complete your transcoding tasks in AWS Lambda in under 5 minutes. Not sure where the second option of using RabbitMQ and workers is coming from. Why RabbitMQ instead of SQS? Would workers be processes on EC2 servers instead of Lambda functions?

Regarding your other question, you need to pass those extra parameters as metadata fields on the S3 object. In the document you linked, look at how x-amz-meta- works. Then when you later retrieve the object from S3 to transcode it you can retrieve the metadata fields at the same time.