ERROR - secure_channel() got an unexpected keyword argument 'default_scopes' with .to_dataframe() on bigquery object

Question

Environment Details

• Python 3.7.12
• google-api-core 1.23.0
• google-auth 1.35.0
• bigquery 2.3.1
• let me know if i can provide any other library versions

We are querying some data from bigquery using python in airflow, and converting results into a dataframe. See this chunk of code:

from google.cloud import bigquery
bq = bigquery.Client()
query_result = bq.query(f"select count(*) as num_rows from our_project.ourdataset.our_table")
our_df = query_result.to_dataframe()

query_result is a <google.cloud.bigquery.job.query.QueryJob object at

When we run the last line our_df = query_result.to_dataframe(), we get the error TypeError: secure_channel() got an unexpected keyword argument 'default_scopes'. The whole error message is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job/query.py", line 1313, in to_dataframe
    date_as_object=date_as_object,
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/table.py", line 1695, in to_dataframe
    create_bqstorage_client=create_bqstorage_client,
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/table.py", line 1510, in to_arrow
    bqstorage_client = self.client._create_bqstorage_client()
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 446, in _create_bqstorage_client
    return bigquery_storage.BigQueryReadClient(credentials=self._credentials)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/client.py", line 386, in __init__
    or Transport == type(self).get_transport_class("grpc_asyncio")
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/transports/grpc.py", line 170, in __init__
    ("grpc.max_receive_message_length", -1),
  File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/transports/grpc.py", line 221, in create_channel
    **kwargs,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 286, in create_channel
    return grpc_gcp.secure_channel(target, composite_credentials, **kwargs)
TypeError: secure_channel() got an unexpected keyword argument 'default_scopes'

This line of code was previously working for us. The only change I recall us making since the code was last working was:

• Installed dbt==0.19.0

to.data_frame() seems like the most basic python function there is, and it is quite frustrating that it is not working here for us. What can we do to resolve this?

Answer 1

I've found the issue, but I have no idea how to resolve. dbt-bigquery 0.19.0 depends on google-api-core<1.24 and >=1.16.0 When we installed dbt, it must have changed the google-api-core version from something greater, down to 1.23.0. However, google-api-core being as low as 1.23 is now causing this other issue with to_dataframe(). I know this because when I manually upgraded google-api-core to 1.26, .to_dataframe() was working again.

EDIT: upgrading dbt to 0.20.0 allows for google-api-core 1.3+, which resolves our issue!

Answer 2

For me upgrading google-auth, google-auth-httplib2, and google-api-core did the trick.