How is includeProjectIds used in Google data catalog search?

Question

When searching using Google's data catalog, a scope parameter is required and one of its fields is includeProjectIds. According to the documentation, this parameter represents:

The list of project IDs to search within.

However, the results returned are not limited to that projectid. Am I understanding the parameter incorrectly? I can limit the results using projectid=myproject in the query, but I am trying to understand the includeProjectIds field.

Example request body:

{
  "scope": {
    "includeProjectIds": [
      "MY-PROJECT"
    ]
  },
  "query": "type=dataset"
}

Example response:

{
  "results": [
    # I expect this result:
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/MY-PROJECT/locations/us/entryGroups/@bigquery/entries/....",
      "linkedResource": "//bigquery.googleapis.com/projects/MY-PROJECT/datasets/dataset_name",
      "modifyTime": "2000-01-01T00:00:00Z",
      "integratedSystem": "BIGQUERY",
      "description": "My description"
    },
    # But I don't expect this:
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/NOT-MY-PROJECT/locations/us/entryGroups/@bigquery/entries/....",
      "linkedResource": "//bigquery.googleapis.com/projects/NOT-MY-PROJECT/datasets/dataset_name",
      "modifyTime": "2000-01-01T00:00:00Z",
      "integratedSystem": "BIGQUERY",
      "description": "My description"
    },
    ...
  ]
}

Answer 1

You may try using curl method instead because it uses the service account key of your service account. For service account and keys creation you can refer to this documentation. I tried this and the output was correct having only my project datasets displayed in the results.

Command:

-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://datacatalog.googleapis.com/v1/catalog:search"

Inside request.json:

{
  "scope": {
    "includeProjectIds": [
      "my-project"
    ]
  },
  "query": "type=dataset"
}

Output:

a@cloudshell:~ (a)$ curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://datacatalog.googleapis.com/v1/catalog:search"
{
  "results": [
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/my-project/locations/us/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzLzIwMjIwMTEx",
      "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/2",
      "modifyTime": "2022-01-11T01:15:11Z",
      "integratedSystem": "BIGQUERY"
    },
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/my-project/locations/us/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzL0dDUFF1aWNrU3RhcnQ",
      "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/G",
      "modifyTime": "2022-01-07T07:54:58Z",
      "integratedSystem": "BIGQUERY"
    },
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/my-project/locations/us-east1/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzLzIwMjExMjI4",
      "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/20",
      "modifyTime": "2021-12-28T02:00:47Z",
      "integratedSystem": "BIGQUERY"
    },
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/my-project/locations/us-central1/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzL2JxbWxfdHV0b3JpYWw",
      "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/b",
      "modifyTime": "2021-12-14T14:58:09Z",
      "integratedSystem": "BIGQUERY"
    },
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/my-project/locations/us-central1/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzL2JhYnluYW1lcw",
      "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/ba",
      "modifyTime": "2021-12-14T14:01:46Z",
      "integratedSystem": "BIGQUERY"
    }
  ]
}