How to secure sendGrid API key secure in Google cloud composer

Question

I have requirement to send notification email, once a Google Cloud Composer airflow DAG is completed. I have come to a solution and used sendGrid as suggested in Google Documents.

But the problem is API key is visible who has access of cloud composer as it is assigned to an Environment variable. When I used the same key in my local systems python program, it is sending mail and that is where I don't understand this conecept. Because if someone has this API key or someone has shared the same it is not project specific anyone can send mail using this key.

Could some one help me, how to secure sendGrid API key.

Answer 1

Analyzing and gathering the evidence of the community contributors in the relevant Stack thread1 and thread2 I would propose you to re-consider the general approach and use JSON file to store the credentials or the other sensitive system authentication data by specifying Keyfile Path in the corresponded Airflow connection.

• Save the API key in JSON file and store it in Airflow environment's Cloud Storage bucket (i.e. /home/airflow/gcs/data/keyfile.json);
• Restrict access to the JSON file, via Cloud Storage ACL as per your security considerations;
• Create a new connection in Airflow, and in the Conn Type field, select the Google Cloud Platform option and then in Keyfile Path, enter the local file path to the JSON keyfile's location;

Since Airflow connection has been created you may refer to the encrypted API key within GoogleCloudBaseHook operator via predefined extra__google_cloud_platform__key_path query parameter in the particular DAG code:

from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook

gcp_hook = GoogleCloudBaseHook(gcp_conn_id="<your-conn-id>")
key_path = gcp_hook._get_field("extra__google_cloud_platform__key_path")