How datastream cannot read UPDATE binary log in Google cloud Datastream

Question

Hi I have some question to ask. I use Datastream to bigquery like guide below https://cloud.google.com/datastream/docs/implementing-datastream-dataflow-analytics.

But when I start stream, I only saw data with change_type is INSERT. There is no UPDATE-INSERT event appear. Even I update 1 record, they won't appear. I use MySQL as source database and Bigquery as destination resource

I wonder if there any case that Datastream cannot read binary log with UPDATE type? Thanks for your help.

Answer 1

You are not specifying whether your MySQL instance is Amazon RDS for MySQL, Amazon Aurora MySQL, Cloud SQL for MySQL or in another case on-prem MySQL.

In first case if you are using AWS RDS for MySQL, binlogs could be deleted if you do not specify a period of time to keep them.

In that case to keep binlogs for three days, you could use:

mysql.rds_set_configuration('binlog retention hours', 72);

In all cases make sure the binary log format of the MySQL database is set to ROW with:

--binlog-format=ROW