App Engine - NDB query with projection requires subproperty?

Question

I have the following objects:

class Address(ndb.Model):
  type = ndb.StringProperty() # E.g., 'home', 'work'
  street = ndb.StringProperty()
  city = ndb.StringProperty()

class Friend(ndb.Model):
  first_name = ndb.StringProperty() # E.g., 'home', 'work'
  last_name = ndb.StringProperty()

class Contact(ndb.Model):
  name = ndb.StringProperty()
  addresses = ndb.StructuredProperty(Address, repeated=True)
  friends = ndb.StructuredProperty(Friend, repeated=True)

And now to optimize the performance of my queries, I want to build a query that will return all the contacts including only the properties name & addresses.

So I built a projection query like this:

qry = Contact.query(projection=['name', 'addresses'])

Which triggers this error:

InvalidPropertyError: Structured property addresses requires a subproperty

Any idea how to make a projection query that includes structured properties?

Answer 1

That query is not possible. Unfortunately the exception isn't very clear, but the issue is the result of how sub-properties are actually stored. ndb explodes your StructuredProperty so that each value is included separately. So, for example, the entity:

Contact(name="Bob", addresses=[
    Address(type="Work", city="San Francisco"),
    Address(type="Home", city="New York")])

This will get expanded to an entity with properties:

name = "Bob"
addresses.type = ["Work", "Home"]
addresses.city = ["San Francisco", "New York"]

When datastore indexes this, you'll end up with the following in your index (in the EntitiesByProperty index, to be specific):

"addresses.city" "New York"      Key(Contact, <id>)
"addresses.city" "San Francisco" Key(Contact, <id>)
"addresses.type" "Home"          Key(Contact, <id>)
"addresses.type" "Work"          Key(Contact, <id>)
"name"           "Bob"           Key(Contact, <id>)

Projection queries work by doing an index scan without looking up the actual entity. In this case, each index row does not have enough information to populate the entire address.

tldr; You can't project a StructuredProperty.

Answer 2

Think about structured property as a list of properties and request each of them:

Contact.query(projection=['name', 'addresses.type', 'addresses.street', 'addresses.city'])

But since it is repeated you will get duplicated contacts with each variation of the address -- you have to combine them manually. You can read more in the docs about projection queries.