Google Dataform incremental merge

45 views Asked by At

Does anyone know how Dataform handles multiple rows with the same uniqueKey in incremental tables?

For example if i have a SQLX file like this:

config {
    type: "incremental",

    database: "adh-silver",
    schema: "financials",
    name: "24so_gl_transaction_actual",

    uniqueKey: ["pk_transaction_id"],
}

SELECT
  pk_transaction_id,
  transaction_number,
  transaction_date,
  financial_period,
  amount,
  last_modified,
  created_time
FROM
  ${ref("adh-silver", "financials", "24so_gl_transaction_historic")}
  ${
      when(
          incremental(),
          `WHERE last_modified > (SELECT MAX(last_modified) FROM ${self()})`
      )
  }

If there are multiple rows, in an incremental update, with the same "pk_transaction_id", how does Dataform know which to merge last?

In other tools it is possible to define a column to order the merges, for example with the "last_modified" column. Is there such an option in Dataform?

Thanks in advance for any help!

0

There are 0 answers