How to get App Engine's local dev server to use local emulated Google Cloud Storage in Python 3?

Question

I've been migrating from Python 2.7 webapp2 to Python 3 Flask.

I'm currently stuck on trying to get the Python 3 runtime to mimic the Python 2 runtime behaviour when it comes to accessing Google Cloud Storage.

The old Python 2.7 runtime automatically used the local Cloud Storage emulator, but I can only get the Python 3 runtime to save to the cloud based Google Cloud Storage.

This means anything running on local dev that uses the blob key of a Cloud Storage object (e.g. the Images API), is failing.

I tried auto-authenticating with Cloud Storage like this:

from google.cloud import storage
from google.auth.app_engine import Credentials
    
credentials = Credentials()
    
client = storage.Client(credentials=credentials)

then any call (e.g. stat()) to Cloud Storage would timeout.

So to get it working I download a JSON credential file and used this instead:

from google.cloud import storage
from google.auth.app_engine import Credentials

from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file('/file/path.json')

client = storage.Client(credentials=credentials)

I suspect this is the culprit (using JSON file credentials instead of auto-auth), but not sure how to get auto-auth working without timing out.

Any ideas?

Answer 1

1. from google.cloud import storage means you're using Cloud Storage (calling it via the Python Cloud Storage Client Library). Therefore, it will connect to 'production' i.e. 'Cloud' and not your local machine unless you've started an emulator (like Cloud Datastore Emulator for Datastore). However, Cloud Storage doesn't have such an emulator (I think there are 3rd parties but none from Google).

   Even running with dev_appserver.py won't solve your problem because per Google documentation (Python 3, Python 2)

   The App Engine local development server doesn't emulate Cloud Storage, so all Cloud Storage requests must be sent over the Internet to an actual Cloud Storage bucket.

2. In Python 2, you were probably doing something like import cloudstorage and that seemed to have been specifically designed for GAE and it had an emulator (see this). Cloud Storage is meant to work for different Apps and not just GAE

Update - Adding working code (tested; it works)

Notes

1. Code supports uploading multiple images but it returns after processing the first image. You can modify the code to add processed images to a list and then return the list

2. If you run this on dev env, the image will be uploaded and you'll get a serving image url but it will give you a 404 if you try to open it in a browser because the baseurl is your localhost but the image is actually on cloud

3. I also had to include google-cloud-datastore in requirements.txt because I noticed that Images.getBaseURI errored out without it. Don't know if this will be required in production

import logging 
from flask import Flask, request
import google.appengine.api
from google.appengine.ext import blobstore
from google.appengine.api import images

DEFAULT_BUCKET = <YOUR_DEFAULT_BUCKET>

from google.cloud import storage
storage_client = storage.Client() 
bucket = storage_client.get_bucket(DEFAULT_BUCKET)

app = Flask(__name__)
app.wsgi_app = google.appengine.api.wrap_wsgi_app(app.wsgi_app)


@app.route("/upload_image/", methods =["POST"])
def upload_image():
    files = request.files.getlist("file")
    for f in files:
        ext = f.filename.split(".")[1]
        
        # We need the image_width because images.get_serving_url now displays a default size of 512
        image_stream = f.stream.read()
        image_width = images.Image(image_stream).width

        blob = bucket.blob(f.filename)

        # Upload the image
        blob.upload_from_string(image_stream, content_type=f"image/{ext}")

        # Create a blob_key which you can store in your datastore for use to pull up the image later
        blob_key = blobstore.create_gs_key(f"/gs/{DEFAULT_BUCKET}/{f.filename}") 

        # Get the image serving url
        url = images.get_serving_url(blob_key) 

        # Append the image size
        url = f"{url}=s{image_width}"
        
        return url
        

@app.route("/")
def index():
    upload_url = "/upload_image/" 
    output = f"""
                <html><body>
                <form action="{upload_url}" method="POST" enctype="multipart/form-data">
                  Upload File: <input type="file" name="file" multiple><br>
                  <input type="submit" name="submit" value="Submit">
                </form>
                </body></html>
                """
    return output