Can the validation error of a dataset be higher than the test error during the whole process of training a neural network?

Question

I'm training a convolutional neural network using pylearn2 library and during all the ephocs, my validation error is consistently higher than the testing error. Is it possible? If so, in what kind of situations?

Answer 1

^{moving the comment to an answer; modifying my previous answer seemed wrong}

The full dataset may not be properly shuffled so the examples in the test set may be easier to classify.

Doing the experiment again with examples redistributed among the train / valid / test subsets would show if this is the case.

Answer 2

Training set is a set of images that are fed to the network, errors are computed on the other end, then the parameters of the network are adjusted based on those errors. Validation set is a set of images that are fed to the network, errors are computed but parameters of the network are NOT adjusted.

Basically, you use validation to see how well the network performs on images it was not trained against.

In this view you should expect in most cases to have a higher error on valid_y_misclass than on train_y_miscalss.

See here for a discussion of the image sets.

---

Edit: example using pylearn2 notation

Size of train set: 700 examples; size of valid set: 300 examples

After some training (say 5 epochs) the network nails 650 out of 700 examples in the training set and 200 out of 300 in valid set.

As a result, after 5 epochs:

train_y_misclass = (700 - 650) / 700 = 0.07142857142
valid_y_misclass = (300 - 200) / 300 = 0.33333333333

valid_y_misclass > train_y_misclass and this is to be expected.