How to delete a Datastore database with Google App Engine search data in it without Python 2.7

Question

Trying to delete the (defaut) Datastore database I get this message:

400: Database contains Google App Engine search data. 
To delete this database you must first remove the search data.

This data is there, unused, from years and years ago when this project used the Python 2.7 appengine just for while. It's a risible weight so never bothered deleting.

Going to https://console.cloud.google.com/appengine/search?project=<PROJECT_ID> I see there's no option to remove the search data.

In App Engine Search API docs there is some python code to delete an index where I probably need Python 2.7 which I don't even have installed anymore. Everywhere deprecation warnings.

Using a new project is NOT an option as this is a production environment which is tied to so many other systems and it would be a pain in the heck to change all the references to a new project — or at least it's not an option for such a meaningless constraint.

And also I exclude using another database than (default), I would lose the free tier quota and also the application code should be adapted to use another db.

This also prevents me being able to do a disaster recovery using backups created with the recently added gcloud alpha firestore backups feature.

Edit:

Trying to follow the documentation I've been pointed to and also this Installing the App Engine services SDK I tried to modify my existing Flask app so I could access search from a flask shell prompt.

I added this to my requirements.txt

appengine-python-standard>=1.0.0

Then, after creating a fresh new virtualenv including the above, I added the wsgi_app lines to my flask app:

from flask import Flask
from google.appengine.api import wrap_wsgi_app

app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)

And in flask shell I tried:

In [1]: from google.appengine.api import search

In [2]: index = search.Index('general-index')

In [3]: index.get_range(ids_only=True)
---------------------------------------------------------------------------
RPCFailedError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 index.get_range(ids_only=True)
[...]
RPCFailedError: Attempted RPC call without active security ticket

It seems I have to bring in also the local development server

gcloud components install app-engine-python

which on my linux box can be done with:

You cannot perform this action because the Google Cloud CLI component manager 
is disabled for this installation. You can run the following command 
to achieve the same result for this installation: 

sudo apt-get install google-cloud-sdk-app-engine-python

which is bringing in Python2.7 as well

$ sudo apt-get install google-cloud-sdk-app-engine-python
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  google-cloud-sdk libpython2.7-minimal libpython2.7-stdlib python2.7 python2.7-minimal
Suggested packages:
  google-cloud-sdk-app-engine-java google-cloud-sdk-pubsub-emulator google-cloud-sdk-bigtable-emulator google-cloud-sdk-datastore-emulator kubectl python2.7-doc binfmt-support
The following packages will be REMOVED:
  google-cloud-cli
The following NEW packages will be installed:
  google-cloud-sdk google-cloud-sdk-app-engine-python libpython2.7-minimal libpython2.7-stdlib python2.7 python2.7-minimal
0 upgraded, 6 newly installed, 1 to remove and 30 not upgraded.
Need to get 117 MB of archives.
After this operation, 60,7 MB of additional disk space will be used.

All this to delete 13,4kb worth of search index...

Also I suspect that running the delete_index routine in dev_appserver.py would eventually delete only local (inexisting) search indexes leaving the one in cloud I want to remove untouched.

Can someone from Google shed a light on this please???

Answer 1

Search API is available for Python 3 via the bundled API/Services.

See documentation on how to enable bundled API for Python 3 in an App.

After enabling it, you can then use the python code snippet you referenced (update any Python 3 incompatible bits) to delete your index

Answer 2

TL;DR Even after deleting the index I still get '400: Database contains Google App Engine search data. To delete this database you must first remove the search data.' I hope NOW Google team can give some answers

Here what I tried.

I deployed a separate service in production to run the delete index code. No local testing as it would find no indexes to delete.

app.yaml:

runtime: python310
app_engine_apis: true
service: cleanup

requirements.txt:

flask
appengine-python-standard>=1.0.0

main.py:

from flask import Flask
from google.appengine.api import wrap_wsgi_app
from google.appengine.api import search

app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)

def delete_index(index):
    #https://cloud.google.com/appengine/docs/legacy/standard/python/search#deleting_an_index
    # index.get_range by returns up to 100 documents at a time, so we must
    # loop until we've deleted all items.
    while True:
        # Use ids_only to get the list of document IDs in the index without
        # the overhead of getting the entire document.
        document_ids = [
            document.doc_id
            for document
            in index.get_range(ids_only=True)]

        # If no IDs were returned, we've deleted everything.
        if not document_ids:
            break

        # Delete the documents for the given IDs
        index.delete(document_ids)

    # delete the index schema
    index.delete_schema()

@app.route('/')
def home():
    index = search.Index('general-index')
    delete_index(index)
    return 'OK', 200

Then deployed with:

gcloud app deploy --project=my-project app.yaml

My service ended up on https://cleanup-dot-my-project.ey.r.appspot.com (with ey.r depending from the region I setup my project)

Calling the URL above it responded with a nice OK and my Search index was gone <3

Delete the cleanup service with:

gcloud app services delete cleanup --project=my-project

Now in the console https://console.cloud.google.com/appengine/search?project=my-project I read

Your default namespace doesn't have any indexes.

Nevertheless, trying to delete the (default) database, I still get the same error message both in the UI

400: Database contains Google App Engine search data. To delete this database you must first remove the search data.

or using gcloud

gcloud firestore databases delete --database='(default)' --project=my-project
The database 'projects/my-project/databases/(default)' will be deleted.

Do you want to continue (Y/n)? Y

ERROR: (gcloud.firestore.databases.delete) FAILED_PRECONDITION: 
Database contains Google App Engine search data. To delete this
database you must first remove the search data.