I have a dataflow jobs written in python in a main.py file, when I want to deploy it as a template I use the following command:

python main.py \
    --runner=DataflowRunner' \
    --template_location=gs://dataflow-bucket/templates/my_template \
    --project=foo \
    --staging_location=gs://dataflow-bucket/staging \
    --temp_location=gs://dataflow-bucket/temp

I also have a metadata file named my_template_metadata that I currently have to separately upload using gsutil

gsutil cp my_template_metadata gs://dataflow-bucket/templates/my_template_metadata

Is there a way to deploy the metadata file in the same time the template is deployed ? it would be much more convenient (or is there a rational reason why it's not possible ?)

0 Answers