Vertex AI SDK and BigQuery : invalid authentication credentials

Question

I try play the code given in exemple in this video : Introduction to Vertex AI SDK - https://www.youtube.com/watch?v=VaaUnIFCNX4

This the code used :

import pandas as pd
import seaborn as sns
import tensorflow as tf
import google

from google.cloud import bigquery
from google.cloud import aiplatform
from tensorflow import keras
from sklearn.preprocessing import LabelBinarizer

credentials, project_id = google.auth.default(
    scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
bqclient = bigquery.Client(credentials=credentials, project=project_id)

query_string = """
SELECT *
FROM `machine-learning-404417.Data_Beam.DryBeam`
"""

dataframe = bqclient.query(query_string).result().to_dataframe()

and I get this error :

Unauthorized: 401 POST https://bigquery.googleapis.com/bigquery/v2/projects/machine-learning-404417/jobs?prettyPrint=false: 
Request had invalid authentication credentials. 
Expected OAuth 2 access token, login cookie or other valid authentication credential. 
See https://developers.google.com/identity/sign-in/web/devconsole-project.

My service account looks good :

Regards

Hello Luciano, thanks for your help I'm running this code on Colab Enterprise :

When I try : gcloud config list i have : as result

I add also the BigQuery User role to my account : as you can see.

but error message is the same :

Unauthorized: 401 POST https://bigquery.googleapis.com/bigquery/v2/projects/machine-learning-404417/jobs?prettyPrint=false: 
Request had invalid authentication credentials. 
Expected OAuth 2 access token, login cookie or other valid authentication credential. 
See https://developers.google.com/identity/sign-in/web/devconsole-project.

Regards

Answer 1

where are you running this code? have you double check if the gcloud cli config is pointing to the service account as account? take a look on the output of gcloud config list. If not, and if you want to use this service acount, you need to set it in the SDK by gcloud config set account <service account id> -- if you have your personnal account configured on the gcloud cli, you can also add BigQuery User role to it then you code may work fine as well.

Also just as a sanity check, is the service account you showed created (or with access) on the machine-learning-404417, right?

Answer 2

The problem is that End User Credentials in your Colab Enterprise Runtime are not active. When you create the first Runtime for your user, and connect to your Runtime in Colab Enterprise, a pop-up that will prompt you for oAuth consent appears. Make sure you have gone through the flow.

A common problem is to skip the checkbox or pop-up blockers.

We have troubleshooting guide which explains additional details. Take a look here.

Also since Credentials are already part of your Runtime, you don't need google.auth.default. (It doesn't generate any errors, just not needed)

Example:

from google.cloud import bigquery

client = bigquery.Client(project=project_id)

sample_count = 2000
row_count = client.query('''
  SELECT
    COUNT(*) as total
  FROM `bigquery-public-data.samples.gsod`''').to_dataframe().total[0]

df = client.query('''
  SELECT
    *
  FROM
    `bigquery-public-data.samples.gsod`
  WHERE RAND() < %d/%d
''' % (sample_count, row_count)).to_dataframe()

print('Full dataset has %d rows' % row_count)

Answer 3

Posting @user3542259's comment as an answer for better visibility for the community:

I've found some solutions to my issue. First I wanted to use colab Enterprise provided by vertex and it's explain here : Introduction to Colab Enterprise on Vertex AI (youtube.com/watch?v=XQnVjc-B9i4) . Also I wanted to do the same with Jupyter notebook in vertex workbench and this is the solution : Bringing BigQuery data into Vertex AI Workbench (youtube.com/watch?v=0Ys52DsiNk8) . regards

Looking at the contents, these are similar to some of the Google documentation posting them for added reference for the solution:

Creating a notebook Section which includes creating a project to uploading a notebook

• Looking at the screenshot it appears that you are missing the required roles that are mentioned in the Link above: roles/aiplatform.colabEnterpriseUser.

To ensure that your user account has the necessary permissions to create a runtime in Colab Enterprise, ask your administrator to grant your user account the Colab Enterprise Admin (roles/aiplatform.colabEnterpriseAdmin) IAM role on the project. For more information about granting roles