I have two table where table1 has 500 MB and table2 has 9TB data. Where table1 data updated every day and sometimes there will be no new requests for the previous date. The main goal is to join the two tables using Dataform when new request are present in the table1 and create a view/table in the BigQuery
I have used a IF condition based on the new request that if new request are present it should join with table2 which is of 9TB else it should stop the execution.
It works when i run the if condition(sql) using Bigquery but when i use the Dataform for the same it does not work. Did anyone faced the same issue. Kindly help me with any solution. That would really help me in solving my problem in Optimizing the execution.