Autokeras exported model performs not as expected

Question

I just installed autokeras on python3.6. After some bug fixing it works well and I can train models with my dataset. After training i get a model with an accuracy of 0.991 and loss of 0.06. With the predict-function it classifies my test dataset almost without any fault. But when I export and save it as a keras model the model performs really bad. It classifies everything wrongly and the predict-function returns just random decimals and not the number of the expected class. I think I am just missing some commands since the image classifier works well. Any help or tips would be very nice.

Edit: I basically followed the example on https://autokeras.com/start/. I preprocessed my data to be in shape (n, 150, 150, 3) and range between 0 and 1. The code for each image is the following:

img='/home/example_image.png'
x=image.img_to_array(image.load_img(img, target_size=(150, 150)))
x=x.reshape(1, 150, 150, 3)
x = x.astype('float32') / 255

I also tried with 50x50 shape for the images because autokeras seems to work better with smaller shapes. My labels are a list of length n. Then I use the code from the website:

clf = ImageClassifier(verbose=True)
clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
y = clf.evaluate(x_test, y_test)
print(y)

It returns a very good model with high accuracy and small loss. I can't show you the screenshots of the output because I am currently at home and my laptop is too weak to reproduce this problem. When I use the following command:

clf.predict(example_img)

I also get a correct result with the number of object in the image. My problem starts when I try to export the model with the commands given on their website:

from autokeras import ImageClassifier
clf = ImageClassifier(verbose=True, augment=False)
clf.load_searcher().load_best_model().produce_keras_model().save('my_model.h5')

After that I load my model.

import keras
from keras.models import load_model
model = load_model('my_model.h5')

It gives me some error:

"UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually. warnings.warn('No training configuration found in save file: '"

I searched for it but people said to ignore it. But when I try to use predict with this loaded model I always get an array of some random decimal values which are often negative. The classification is always wrong since the highest value in this array is for a wrong class.

I will edit this text with more screenshots and details when I am at the office again with my ML computer.

Edit 2: When I train the model this is the best model :

+--------------------------------------------------------------------------+
|    Father Model ID     |                 Added Operation                 |
+--------------------------------------------------------------------------+
|                        |           ('to_add_skip_model', 1, 5)           |
|                        |            ('to_wider_model', 1, 64)            |
|                        |            ('to_wider_model', 5, 64)            |
|           5            |          ('to_conv_deeper_model', 9, 3)         |
|                        |         ('to_conv_deeper_model', 23, 3)         |
|                        |          ('to_concat_skip_model', 5, 9)         |
+--------------------------------------------------------------------------+

Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           6            |  0.014135746611282229  |          1.0           |
+--------------------------------------------------------------------------+

After that I used final_fit and tried the predict_function:

>>> clf.predict(test_images)
array([ 0.,  0.,  0., ..., 12., 12., 12.])

This is the expected output. Evaluation gives also good results:

>>> y = clf.evaluate(test_images, test_labels)
>>> print(y)
0.9969230769230769

Then I export the model as a keras model and load it again:

>>> from keras import models
>>> clf.load_searcher().load_best_model().produce_keras_model().save('keras_best_model.h5')
>>> model = models.load_model('keras_best_model.h5')

Everything does fine and I get no error. But when I use the predict_function now it returns wrong results:

>>> model.predict(test_images[0].reshape(1, 50, 50, 3))
array([[ 2.5287893, -2.2281592, -2.8172228,  1.1171696, -5.8477755,
        -3.1250796,  1.4904132,  1.1068834, -4.982565 , -1.6350467,
        -4.3806715, -2.7464929, -6.1051216]], dtype=float32)

So clf.predict gives me one number and model.predict give me an array. Shouldn't it be the same? Almost all predictions are wrong. When I evaluate the model this is the output:

>>> model.evaluate(test_images, test_labels1)
2600/2600 [==============================] - 1s 569us/step
[5.251570468682509, 0.10115384615384615]

It is totally bad in comparison to the clf evaluation and I don't know why.

Answer 1

In the new version, (0.3.6), it seems

clf.load_searcher().load_best_model().produce_keras_model().save('keras_best_model.h5') 

is removed. For the moment you can use "autokeras.utils.pickle_to_file" for saving and "autokeras.utils.pickle_from_file" to load the model. This is not a usual keras model, however we can get same results.