Error "Not found: Dataset *****:dataform was not found in location US." while repository located in EU

Question

I am currently working with a SQL workflow using Dataform. Within this workflow, I have declared certain tables and established dependencies among them. Everything is functioning as expected in this regard. However, I encountered an issue when attempting to create a table that depends on other tables which have not been declared explicitly.

The problem arises from the fact that the query doesn't compile due to the error message:

Not found: Dataset *****:dataform was not found in location US

Even though my repository is configured in the EU.

For illustration, here's an example:

simple_table

config {
  type: "table",
  description: ""
  }
SELECT 1 AS number
UNION ALL
SELECT 2 AS number

dependent_table

config {
  type: "table",
  description: ""
}
SELECT * FROM ${ref("simple_table")}

I tried to change the repository location, it didn't work.

Answer 1

Check the dataform.json in your project root. In my case there was

  "defaultLocation": "EU"

that was causing the same issue. In your case, "US" is probably set there.

Answer 2

You can't combine data from multiple regions, so one of the tables is in the US, and one is in the EU, or they're all in the US and your project is set to EU. As mentioned above, changing the location in dataform.json will fix this.

If you're dealing with data across different regions, the way I've resolved this in the past is to set up a daily data transfer that moves the data I need to the primary region, then use that as your source.

Answer 3

If you are running with the Dataform CLI then it's likely that your credentials are locked to that region. When you initialise the credentials with dataform init-creds bigquery, you are asked to choose between

[1] US (default)
[2] EU
[3] other

Chances are that you selected the default option.