Nvidia DIGITS: Loss coverage (val and train) go straight to zero

360 views Asked by At

EDIT: FOUND THE PROBLEM. I forgot to add a custom class when creating the dataset. Amateur mistake, but leaving this up for others who make a similar error.

After successfully following the KITTI tutorial provided by NVIDIA, I began trying to run DIGITS on my own dataset. The problem is that almost immediately after starting, the loss coverage for both the train and validation goes straight to almost zero, on the scale of 10e-4 loss.

This is a picture of the loss as a function of the epoch (account is too new to embed image).

Extra details:

  • ~900 training images and ~300 validation images (Note: I have been told by a friend that he got it to work with this data set, so I don't believe the number of images is the problem)
  • The images are very simple and contain a model of a car on a mostly plain background.
  • I had to create the validation data by taking images/labels from the training data
  • The model I used is the DetectNet model mentioned in the DIGITS object detection tutorial.
  • I used the GoogLeNet pretrained model also mentioned in the tutorial.

What I tried:

  • Changing the hyperparameters appears to do nothing
  • As mentioned earlier, running the model for longer than 1 epoch does not change anything (loss coverage stays close to zero)
0

There are 0 answers