File transfer from AWS Fsx to S3

Question

Is it possible to use a Lambda function to transfer a file from FSX to S3 bucket whenever there is a new file in FSX? Will appreciate any inputs on this, thank you!

Answer 1

one option is- you can build a lambda function that would need to be triggered by a schedule (e.g., using Amazon CloudWatch Events). As per the AWS blog linked below, it seems there is a audit file access and use that to trigger the lambda- https://docs.aws.amazon.com/fsx/latest/WindowsGuide/file-access-auditing.html

you need to configure your Lambda function to access the Amazon FSx file system- using the update-function-configuration command with the ARN of the file system's access point and specifying a local mount path for the file system within the Lambda environment.

this can be found in lambda under - configuration > File systems

the code in your lambda (if usin Python) could look similar to this(I haven't tested this):

import boto3
import os

s3 = boto3.client('s3')

def lambda_handler(event, context):
    # Assuming the file to be transferred is named 'new_file.txt'
    # and the destination S3 bucket is specified as an environment variable
    file_name = 'new_file.txt'
    destination_bucket = os.environ['DESTINATION_BUCKET']
    local_file_path = f'/mnt/efs0/{file_name}'  # Adjust the local mount path as configured
    
    try:
        # Upload the file to S3
        s3.upload_file(local_file_path, destination_bucket, file_name)
        print(f'File {file_name} uploaded to {destination_bucket}')
    except Exception as e:
        print(f'Error uploading file: {e}')

here is a blog that talks about this pattern - https://aws.amazon.com/blogs/storage/enabling-smb-access-for-serverless-workloads/ - it seems there is a way to trigger based on the file in FSX -

You can use file access auditing support on FSx for Windows File Server to generate event logs in Amazon CloudWatch whenever a file change event occurs.