We are reading avro records in a streaming manner. We would like to use BigQueryWrite.AppendRows to upsert those events to BigQuery Storage API (with a JVM process).
Could you please help me on the following:
- Does BigQuery Storage Write API (appendRows) support appending Avro Records? I only see the Json and Protobuf is documented.
- If Yes, could you please point me to an example? Thanks a lot!
[1] https://cloud.google.com/bigquery/docs/write-api
Alternative approach:
- Staging the Avro records to a GCS bucket after reading N min with a GCS Avro Writer.
- Loading from the GCS Bucket to Big Query [2].
The above seems to work. However, since we need to do an orchestration between writing to GCS and Ingesting, it seems a better solution is to use Bigquery Write API.
[2] https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro