pytorch inception model outputs the wrong label for every input image

Question

For the pytorch models I found this tutorial explaining how to classify an image. I tried to apply the same prcedure for an inception model. However the model fails for every image I load in

Code:

# some people need these three lines to make it work
#from torchvision.models.inception import model_urls
#name = 'inception_v3_google'
#model_urls[name] = model_urls[name].replace('https://', 'http://')

from torch.autograd import Variable
import torchvision
import requests
from torchvision import models, transforms
from PIL import Image
import io
from PIL import Image

LABELS_URL = 'https://s3.amazonaws.com/outcome-blog/imagenet/labels.json'

# cat
IMG_URL1 = 'http://farm2.static.flickr.com/1029/762542019_4f197a0de5.jpg'

# dog
IMG_URL2 = 'http://farm3.static.flickr.com/2314/2518519714_98b01968ee.jpg'

# lion
IMG_URL3 = 'http://farm1.static.flickr.com/62/218998565_62930f10fc.jpg'


labels = {int(key):value for (key, value)
          in requests.get(LABELS_URL).json().items()}


model = torchvision.models.inception_v3(pretrained=True)
model.training = False
model.transform_input = False



def predict_url_img(url):
    response = requests.get(url)
    img_pil = Image.open(io.BytesIO(response.content))

    normalize = transforms.Normalize(
       mean=[0.485, 0.456, 0.406],
       std=[0.229, 0.224, 0.225]
    )
    preprocess = transforms.Compose([
       transforms.Scale(256),
       transforms.CenterCrop(299),
       transforms.ToTensor(),
       normalize
    ])

    img_tensor = preprocess(img_pil)
    img_tensor.unsqueeze_(0)
    img_variable = Variable(img_tensor)
    fc_out = model(img_variable)
    print("prediction:", labels[fc_out.data.numpy().argmax()])

predict_url_img(IMG_URL1)
predict_url_img(IMG_URL2)
predict_url_img(IMG_URL3)

As output I get this:

('prediction:', u"plunger, plumber's helper")

('prediction:', u'plastic bag')

('prediction:', u"plunger, plumber's helper")

Answer 1

I found out that one needs to call model.eval() before applying the model. Because of the batch normalisations and also dropout layers, the model bahaves differently for training and testing.

Answer 2

I executed your code adding the model.eval() line and got:

('prediction:', u'Egyptian cat')
('prediction:', u'groenendael')
('prediction:', u'lion, king of beasts, Panthera leo')

But if I change the preprocessing (just removing your normalize operation and setting model.transform_input = True to use the default preprocessing for Inception v3 in PyTorch) results are slightly different:

('prediction:', u'tiger cat')
('prediction:', u'Border collie')
('prediction:', u'lion, king of beasts, Panthera leo')

I'm not a cat and dog expert, but a quick Google search suggests that these results are more accurate (the cat in your input picture seems closer to a tiger cat than an Egyptian cat, and Border collie dogs look more similar to the input than groenendael as well).

My point is that I think one should use inception_v3 default preprocessing (which is the default behavior unless you change it to model.transform_input = False) instead of classical normalization, but I couldn't find a definitive answer regarding this.

This thread discusses that matter.

Hope this helps someone.