Apache Beam + Big Query Table Read

Question

I have dataset in big query in project: Project: project-x Table: table01 Dataset: dataset01

I would like to connect to it from Apache Beam and read value of one column-column01 for example...

This is what I have:

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="Z:\DEV\CREDENTIALS\cred.json"

QUERY="""
    SELECT column01 from project-x:table01.dataset01
    """
options = {'project': 'project-x',
'runner': 'DirectRunner',
'region': 'EU'
}
pipeline_options = beam.pipeline.PipelineOptions(flags=[], **options)
pipeline=beam.Pipeline(options=pipeline_options)   
BQ_source = beam.io.BigQuerySource(query = QUERY)
BQ_data = pipeline | beam.io.Read(BQ_source)

So after executing I am getting nothing.... I think its some basic problem but I just started and really would like to see some results. Thanks for any help.

Answer 1

There is 1 error and I have 1 advice Error: The from format is project:dataset.Table is legacy SQL.

Advice: prefer standard SQL for being able to use all new bigquery features! From Format is `project.dataset.table` Back quote are required. And set the option legacy=off in beam.