predict() in tensorflowjs is not working as expected after conversion from python

Question

EDIT: Provided additional information based on the comment from @mrk

1. I use KerasCV RetinaNet to detect certain objects and then save it using: model.save("saved_models/retinanet_model.keras")

The model works fine in python.

2. I would like to use this model in nodejs; hence, converted it using: tensorflowjs_converter --input_format=keras --output_format=tfjs_graph_model saved_models/retinanet_model.keras tfjs_models/retinanet_model

• The conversion operation provides the following warning message: WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. 'model.compile_metrics' will be empty until you train or evaluate the model.

3. I then load the model in nodejs using: retinaNetModel = await tf.loadGraphModel("file://./tfjs_models/retinanet_model/model.json");

However, the retinaNetModel.predict() does not match the original model output and cannot predict anything.

There are several observations I want to share, which make me think the model is not converted properly somehow; despite the fact that I only get a warning as opposed to an error.

• In python, if the model cannot find anything, I get -1 for bounding_box, classes, etc. However, in the tensorflowjs, it looks like I have random negative values. I get two Tensors as a response where the first tensor looks like containing bounding_box values and the second Tensor containing the class probabilities. For example, the second tensor has something like:

[
  [
    [ -4.490991592407227, -4.7061991691589355, -4.518319606781006 ],
    [ -4.743373394012451, -4.800686359405518, -4.49501371383667 ],
    ...
    [ -4.731906414031982, -4.996838569641113, -4.9045610427856445 ],
  ]
]

• Another unexpected behavior is related to the size of the output tensor values. In tensorflowjs, I get 76725 items/arrays for classes and bounding_boxes. Why such a high number? In python, the model generates 100 length arrays for bounding_box, classes, etc.

I use the same machine for all (training the RetinaNet, conversion and loading to tensorflowjs in nodejs). pip list | grep tensorflow generates the following

tensorflow                     2.15.0
tensorflow-datasets            4.9.3
tensorflow-decision-forests    1.8.1
tensorflow-estimator           2.15.0
tensorflow-hub                 0.15.0
tensorflow-io-gcs-filesystem   0.34.0
tensorflow-macos               2.15.0
tensorflow-metadata            1.14.0
tensorflowjs                   4.14.0

I use the same image data. In fact I wrote the image array from python and read it in nodejs to make sure it is perfectly identical. I reshaped the data array via let imageTensor = tf.tensor(pixeldata).reshape([640, 640, 3]); as my image is 640x640.

Any advise/pointers that you might have would be greatly appreciated.

Thanks,

Doug

Answer 1

With the given information it is a little hard to give specific advice. However I would proceed as follows.

1. Check that your data pipeline works correct in the node.js environment.
2. Did the model conversion work out without issues - check your logs for errors and if none, also warnings.
3. Are you using a compatible tensorflow.js version for the converted model?
4. How does your model output look like, is there some postprocessing step missing?

If this does not solve your problem, the answers will definitely help us to help you.