Bash Operator to access GCS bucket file in GCP Astronomer

357 views Asked by At

I need to run some bash operation on a file that has been present in GCS Bucket.

bash_operator = BashOperator(
    task_id='mani_bash',
    bash_command="""if [ `awk -F: '/^[^HDR][^TRL]/ { print }' gs://<bucketname>/<location>/filename.txt | awk -F "|" '{print NF-1}' | uniq | wc -l` -eq 1 ];
 then
 if [ `awk -F: '/^[^HDR][^TRL]/ { print }' gs://<bucketname>/<location>/filename.txt | awk -F "|" '{print NF-1}' | uniq` -eq 9 ]; then 
echo 'rite'; 
fi;
 else
 echo 'not rite'; 
fi""",
)

I am getting this error.

awk: cannot open gs://bucketname/location/filename.txt (No such file or directory)

Can someone let me know, is it possible to access GCS bucket using bash operator. if yes, please let me know, how to access.

1

There are 1 answers

0
Mani Shankar.S On

gsutil cat helped as solution

bash_operator = BashOperator(
    task_id='mani_bash',
    bash_command="""if [ `gsutil cat gs://<bucketname>/<location>/filename.txt | awk -F: '/^[^HDR][^TRL]/ { print }' | awk -F "|" '{print NF-1}' | uniq | wc -l` -eq 1 ];
 then
 if [ `gsutil cat gs://<bucketname>/<location>/filename.txt | awk -F: '/^[^HDR][^TRL]/ { print }' | awk -F "|" '{print NF-1}' | uniq` -eq 9 ]; then 
echo 'rite'; 
fi;
 else
 echo 'not rite'; 
fi""",
)