I am looking to perform the equivalent of SQL UNION within one pipeline, in Data Fusion. I do not see a plugin named UNION.
Can I achieve this functionality using any other existing plugin? Can I leverage a plugin designed for another purpose, to achieve this function?
What are my alternatives, if this function is not directly made available within Data Fusion?
Data is all structured, extracted from CSV and Relational tables. For schema, its assumed that they have the same schema, which is the requirement for SQL UNION.
I am assuming, stage the data in a BQ table, with APPEND mode, and then perform additional transformations in a separate pipeline?
By default if you are reading from multiple sources with same schema, they are treated as union. If the schema is different, you can use wrangler plugin to make the schema same before writing to sinks. Hope this helps