Overcoming the size entry size limit for appengine search?

Question

I am trying to use a searchable model for blog entries and it worked fine on the development platform but when I try to add an entry in the cloud I get an error:

Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/init.py", line 517, in call
    handler.post(*groups) File
"/base/data/home/apps/smart-fast/1.348228399174418277/admin.py", line 76, in post
    article.put()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/init.py", line 895, in put
    return datastore.Put(self._entity, config=config)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 404, in Put
    return _GetConnection().async_put(config, entities, extra_hook).get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 601, in get_result
    self.check_success()
File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 572, in check_success
    rpc.check_success()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 502, in check_success
    self.__rpc.CheckSuccess()
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 126, in CheckSuccess 
    raise self.exception
ApplicationError:

ApplicationError: 1 Too many indexed > properties for entity: app:
"smart-fast",path < Element { type: "Article", id: 2002 }> This index put
it over the limit: entity_type: "Article",ancestor: false,
Property { name: "searchable_text_index", direction: ASCENDING},
Property { name: "searchable_text_index", direction: ASCENDING},
Property { name: "date", direction: ASCENDING}

These entries aren't that large(<500 words) is the limit that low? The only way I can think to get around this is to store the entry as a non-searchable model and also break the entry text up into smaller searchable models, that each reference the main entry. Any help is greatly appreciated

Answer 1

What you seem to be running into are called Exploding Indexes. You can read more about them here: http://code.google.com/appengine/docs/python/datastore/queries.html#Big_Entities_and_Exploding_Indexes

Basically you can't use AppEngine to create a full-text engine, unless you have a very small dataset. It's either you'll run into exploding indexes or you'll run into other issues (i.e. merge join timeouts). I recommend that you look into a service called IndexTank which is a very good full-text search service. It has a full REST api and python client so it's easy to get going on AppEngine.