What index do I need in Google Cloud Datastore to filter based on ID and another property?

Question

I use Google Cloud Datastore to store entities with a generated ID by Datastore as primary key. In addition, I store other properties for each entity, in this example owner.

An entity looks like this in the Google Cloud Datastore Console:

|Name/ID             |owner           |...|
|id=5657437197565952 |5634472569470976|...|

I then want to filter entities based on the key (id) and the property owner.

To do this, I understand that I need a composite index to be able to filter on both fields (id) and owner at the same time. So I created a composite index using the following index.yaml file:

indexes:
- kind: job
  ancestor: no
  properties:
  - name: __key__
  - name: owner

# AUTOGENERATED

The index shows up in the cloud console and if I filter on both fields using the cloud console UI it filters the correct entity in the table. The filter I do there is:

• Key is exactly as large as: key(job,5657437197565952)
• owner equals the string: 5634472569470976

But when I try to retrieve this entity with the Node.js library by Google using the following code, I do not get any result:

const {Datastore} = require('@google-cloud/datastore');

async function quickStart() {
  const projectId = 'myproject';

  // Creates a client
  const datastore = new Datastore({
    projectId: projectId,
  });

  const keySelector = datastore.key(['myentity', '5657437197565952']);
  const query = datastore.createQuery('myentity').filter('key', keySelector).filter('owner', '=', '5634472569470976');

  const val = await query.run((err, entities, info) => {
    console.log(entities);
  });

  console.log('Got ' + val);
}
quickStart().catch(console.error);

The entities are empty and val is just the following if stringified:

[[],{"moreResults":"NO_MORE_RESULTS","endCursor":"CgA="}]

I also tried to create another index with the name id instead of __key__ but had no luck.

My question is now: what composite index do I need to perform this filter on both fields so that the NodeJS code also retrieves the entity correctly?

Answer 1

Since you have the key ID it is not necessary to bother with queries, filters and indexes anymore: there can be maximum one entity matching such query, which you can get by direct key lookup. I think would be something along these lines (sorry, I'm not node.js fluent):

const key = datastore.key(['myentity', 5657437197565952]);
const entity = await datastore.get(key);

A positive side effect is that key lookups are strongly consistent (queries are eventually consistent), see comparison table at Eventual Consistency in Cloud Datastore.

But if you insist on filtering on the key then you'd have to reference the property using the special __key__ name (as you did in your index file). From Filters:

Key filters

To filter on the value of an entity's key, use the special property __key__:

In your case it'd be something like this:

const keySelector = datastore.key(['myentity', 5657437197565952]);
const query = datastore.createQuery('myentity')
                       .filter('__key__', '=', keySelector)
                       .filter('owner', '=', '5634472569470976');

Note that the key id in the keySelector object creation needs to be without quotes!