Logistic Regression Cifar10- image classification using tensorflow 1.x

Question

I'm trying to implement a simple logistic regression for image classification using the Cifar10 dataset. I'm only allowed to use TensorFlow 1.x for the training. (I am allowed to use Keras and other libraries for manipulating the data)

My problem is that the model I built does not learn ... All epochs give values of 0.1 in the accuracy of both the test and the train.

I think there is some problem with manipulating the data itself before I send to the model, I would be happy to help understand why the model is not learning.

code:

%tensorflow_version 1.x

import tensorflow as tf
import numpy as np
import keras
import cv2 as cv2
import matplotlib.pyplot as plt
from keras.utils import to_categorical
from keras.datasets import mnist, cifar10


def get_cifar10():
    """Retrieve the CIFAR dataset and process the data."""
    # Set defaults.
    nb_classes = 10
    batch_size = 64
    input_shape = (3072,)

    # Get the data.
    (x_train, y_train), (x_test, y_test) = cifar10.load_data()
    x_train = x_train.reshape(50000, 3072)
    x_test = x_test.reshape(10000, 3072)
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    # x_train /= 255
    # x_test /= 255

    # convert class vectors to binary class matrices
    y_train = to_categorical(y_train, nb_classes)
    y_test = to_categorical(y_test, nb_classes)

    return (nb_classes, batch_size, input_shape, x_train, x_test, y_train, y_test) 

nb_classes, batch_size, input_shape, x_train, x_test, y_train, y_test = get_cifar10()


features = 3072
categories = nb_classes

x = tf.placeholder(tf.float32, [None, features])
y_ = tf.placeholder(tf.float32, [None, categories])
W = tf.Variable(tf.zeros([features,categories]))
b = tf.Variable(tf.zeros([categories]))

y = tf.nn.softmax(tf.matmul(x, W) + b)

loss = -tf.reduce_mean(y_*tf.log(y))

update = tf.train.GradientDescentOptimizer(0.0001).minimize(loss)
correct_prediction = tf.equal(tf.argmax(y, 1),tf.argmax(y_, 1)) 
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))


sess = tf.Session()
sess.run(tf.global_variables_initializer())
for epoch in range(0,1000):
    sess.run(update, feed_dict = {x:x_train, y_:y_train}) #BGD 
    train_acc = sess.run(accuracy, feed_dict={x:x_train, y_:y_train})
    test_acc = sess.run(accuracy, feed_dict={x:x_test, y_:y_test})
    if(epoch % 10 == 0):
      print ("epoch: %3d train_acc: %f test_acc: %f" % (epoch,train_acc, test_acc))

Running the model gives the following:

epoch:   0 train_acc: 0.099880 test_acc: 0.099900
epoch:  10 train_acc: 0.100000 test_acc: 0.100000
epoch:  20 train_acc: 0.100000 test_acc: 0.100000
epoch:  30 train_acc: 0.100000 test_acc: 0.100000
epoch:  40 train_acc: 0.100000 test_acc: 0.100000
epoch:  50 train_acc: 0.100000 test_acc: 0.100000
epoch:  60 train_acc: 0.100000 test_acc: 0.100000
epoch:  70 train_acc: 0.100000 test_acc: 0.100000
epoch:  80 train_acc: 0.100000 test_acc: 0.100000
epoch:  90 train_acc: 0.100000 test_acc: 0.100000
epoch: 100 train_acc: 0.100000 test_acc: 0.100000
epoch: 110 train_acc: 0.100000 test_acc: 0.100000
epoch: 120 train_acc: 0.100000 test_acc: 0.100000
epoch: 130 train_acc: 0.100000 test_acc: 0.100000

Thanks in advance!

Answer 1

So you got three problems

1. Uncomment these two lines:

   # x_train /= 255
   # x_test /= 255
   

You should normalize your input.

2. The loss is not the mean of the log losses, but only the sum (you are working with mutually exclusive classes)

   loss = -tf.reduce_sum(y_*tf.log(y))
   

3. Change your optimizer, or the learning rate. I have used Adam and the loss is now ok

   update = tf.train.AdamOptimizer(0.0001).minimize(loss)
   

The run on colab

output:

epoch:   0 train_acc: 0.099940 test_acc: 0.099900
epoch:  10 train_acc: 0.258440 test_acc: 0.258300
epoch:  20 train_acc: 0.287600 test_acc: 0.291300
epoch:  30 train_acc: 0.306160 test_acc: 0.308000
epoch:  40 train_acc: 0.320680 test_acc: 0.321400
epoch:  50 train_acc: 0.332040 test_acc: 0.331700
epoch:  60 train_acc: 0.340040 test_acc: 0.337500
epoch:  70 train_acc: 0.345100 test_acc: 0.345100
epoch:  80 train_acc: 0.350460 test_acc: 0.348900
epoch:  90 train_acc: 0.354780 test_acc: 0.353200
epoch: 100 train_acc: 0.358020 test_acc: 0.356400
epoch: 110 train_acc: 0.361180 test_acc: 0.359400
epoch: 120 train_acc: 0.364420 test_acc: 0.361600
epoch: 130 train_acc: 0.367260 test_acc: 0.362900
epoch: 140 train_acc: 0.369220 test_acc: 0.365700
epoch: 150 train_acc: 0.371540 test_acc: 0.367900
epoch: 160 train_acc: 0.373560 test_acc: 0.368700
epoch: 170 train_acc: 0.375220 test_acc: 0.371300
epoch: 180 train_acc: 0.377040 test_acc: 0.372900
epoch: 190 train_acc: 0.378840 test_acc: 0.375000
epoch: 200 train_acc: 0.380340 test_acc: 0.377500
epoch: 210 train_acc: 0.381780 test_acc: 0.379800
epoch: 220 train_acc: 0.383640 test_acc: 0.380400
epoch: 230 train_acc: 0.385340 test_acc: 0.380600
epoch: 240 train_acc: 0.386500 test_acc: 0.381300
epoch: 250 train_acc: 0.387640 test_acc: 0.381900
...

Obviously it is not the best to use a LogistRegressor on images. To obtain better and faster results it's better to use CNN