how make correct predictions of jpeg image in cloud-ml

Question

I want to predict a jpeg image in cloud-ml.

My training model is the inception model, and I would like to send the input to the first layer of the graph: 'DecodeJpeg/contents:0' (where I have to send a jpeg image). I have set this layer as possible input by adding in retrain.py:

inputs = {'image_bytes': 'DecodeJpeg/contents:0'}
tf.add_to_collection('inputs', json.dumps(inputs))

Then I save the results of the training in two files (export and export.meta) with:

saver.save(sess, os.path.join(output_directory,'export'))

and I create a model in cloud-ml using these files.

As suggested in some posts (here, here, and here from Google cloud official blog) I'm trying to make the prediction with

gcloud beta ml predict --json-instances=request.json --model=MODEL

where the instance is the jpeg image decoded in base64 format with:

python -c 'import base64, sys, json; img = base64.b64encode(open(sys.argv[1], "rb").read()); print json.dumps({"key":"0", "image_bytes": {"b64": img}})' image.jpg &> request.json

However the request return me:

error: 'Prediction failed: '

What is the problem of my procedure? Do you have any suggestion? I particular from this post I assume that cloud-ml automatically convert the base64 image in jpeg format when it reads a request with image_bytes. Is it correct? Otherwise how can I do?

Answer 1

CloudML requires you to feed the graph with a batch of images.

I'm pretty sure this is the issue with re-using retrain.py. See that code's sess.run line; it is feeding a single image at a time. Compare with the batched jpeg placeholder in the flowers sample.

Answer 2

Note that three slightly different TF graphs need to be constructed: Training, Evaluation, and Prediction. See this recent blog post for details. The training and Prediction graphs directly consume embedding from preprocessing so they do not contain an Inception graph. For prediction, we need to take image bytes as input and use Inception to extract embeddings.

For online prediction, you need to export the prediction graph.You should also specify the outputs and a key for inputs.

To build the prediction graph (the code):

def build_prediction_graph(self):
   """Builds prediction graph and registers appropriate endpoints."""
   tensors = self.build_graph(None, 1, GraphMod.PREDICT)
   keys_placeholder = tf.placeholder(tf.string, shape=[None])
   inputs = {
      'key': keys_placeholder.name,
      'image_bytes': tensors.input_jpeg.name
   }

   tf.add_to_collection('inputs', json.dumps(inputs))

   # To extract the id, we need to add the identity function.
   keys = tf.identity(keys_placeholder)
   outputs = {
       'key': keys.name,
       'prediction': tensors.predictions[0].name,
       'scores': tensors.predictions[1].name
   }
   tf.add_to_collection('outputs', json.dumps(outputs))

To export the preciction graph:

def export(self, last_checkpoint, output_dir):
  # Build and save prediction meta graph and trained variable values.
  with tf.Session(graph=tf.Graph()) as sess:        
    self.build_prediction_graph()
    init_op = tf.global_variables_initializer()
    sess.run(init_op)
    self.restore_from_checkpoint(sess, self.inception_checkpoint_file,
                                 last_checkpoint)
    saver = tf.train.Saver()
    saver.export_meta_graph(filename=os.path.join(output_dir, 'export.meta'))
    saver.save(sess, os.path.join(output_dir, 'export'), write_meta_graph=False)

last_checkpoint must point to the latest checkpoint file from training:

self.model.export(tf.train.latest_checkpoint(self.train_path), self.model_path)

Answer 3

In your post, you indicated that your inputs collection has only "image_bytes" tensor alias. However, in the code where you are framing the request, you are including 2 inputs: One is "key" and the other is "image_bytes". So, my suggestion would be to remove "key" from the request or add "key" to the inputs collection.

Second issue is that the shape of DecodeJpeg/contents:0', is (). For Cloud ML, you need to have a shape like (None, ) so that you can feed that in.

There are some suggestions in other answers to your question here, on how you might be able to follow the public posts to modify your graph, but at hand I can tell these two issues.

Let us know if you encounter any further issues.