JSON Errors when trying to create a transfer_config in Google BigQuery CLI

Question

I am trying to create a transfer job on the windows commandline with

bq mk --transfer_config --data_source=amazon_s3 
--target_dataset=Usage --display_name='s3_transfer_installs_global_in_v0_test' 
--params='{"data_path_template":"mybucket", "destination_table_name_template":"in_table", "file_format":"CSV", "max_bad_records":"0", "skip_leading_rows":"1", "allow_jagged_rows":"false", "allow_quoted_newlines":"true", "access_key_id":"dfadfadf",  "secret_access_key":"sdfsfsdfsdf"}' 

but I keep getting variations of the error

Too many positional args, still have ['"allow_quoted_newlines":"true","access_key_id":',...

Output from --apilog was also not enlightening.

My JSON validates, but there might still be escape characters needed maybe?

Any help very much appreciated, have been shuffling around quotation marks and backslashes for two hours now...

Answer 1

I got the same error as you when running your query.

I tried to replace double quotes with single quotes in --params option and it seems to be working. Try the following:

bq mk --transfer_config --data_source=amazon_s3 --target_dataset=Usage --display_name='s3_transfer_installs_global_in_v0_test' --params="{'data_path_template':'mybucket', 'destination_table_name_template':'in_table', 'file_format':'CSV', 'max_bad_records':'0', 'skip_leading_rows':'1', 'allow_jagged_rows':'false', 'allow_quoted_newlines':'true', 'access_key_id':'dfadfadf',  'secret_access_key':'sdfsfsdfsdf'}"

I also tried to run the original command in Windows PowerShell and it worked without any changes.

I think the problem is in Windows cmd...