botocore.exceptions.NoRegionError: You must specify a region for EmrServerlessCreateApplicationOperator

Question

I am trying to create a emr-serverless application through the EmrServerlessCreateApplicationOperator but I keep facing the error botocore.exceptions.NoRegionError: You must specify a region.

I am passing the region like below:

 create_app = EmrServerlessCreateApplicationOperator(
        task_id="create_spark_app",
        job_type="SPARK",
        release_label="emr-6.6.0",
        config={"aws_access_key_id":args["aws_access_key_id"], 
                "aws_secret_access_key": args["aws_secret_access_key"], 
                "aws_session_token": args["aws_session_token"],
                "region_name":'us-east-1'}
    )

Any help is appreciated. Thank you.

Answer 1

I was able to figure it out. I passed them as environment variables.

os.environ["AWS_ACCESS_KEY_ID"]= YOUR VALUE
os.environ["AWS_SECRET_ACCESS_KEY"]= YOUR VALUE
os.environ["AWS_SESSION_TOKEN"]= YOUR VALUE
os.environ["AWS_DEFAULT_REGION"]= YOUR VALUE

This worked.

Answer 2

You're specifying the region_name attribute as part of config argument to EmrServerlessCreateApplicationOperator. Have you considered specifying the region with AWS connection instead? When the connection ID isn't specified with operator, it should default to aws_default. Refer to this for more context. https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#default-connection-ids