I am fine-tuning a pre-trained Caffe model. For some reason, the test iteration always returns loss = 0 and top-1 as -nan. The training iteration values for loss and top-1 look reasonable to me.
I expected that top-1 is figure of accuracy for particular classification and would always range between [0-1] while loss could be float.
Here is the output over 300 iterations, I display results every 100 iterations:
I1002 08:08:24.124348 415 solver.cpp:337] Iteration 0, Testing net (#0)
I1002 08:08:24.221879 415 solver.cpp:404] Test net output #0: loss_aeroplane_1 = 0
I1002 08:08:24.221915 415 solver.cpp:404] Test net output #1: loss_aeroplane_2 = 0
I1002 08:08:24.221930 415 solver.cpp:404] Test net output #2: loss_aeroplane_3 = 0
I1002 08:08:24.221944 415 solver.cpp:404] Test net output #3: loss_aeroplane_4 = 0
I1002 08:08:24.221958 415 solver.cpp:404] Test net output #4: loss_aeroplane_5 = 0
I1002 08:08:24.222059 415 solver.cpp:404] Test net output #11: top-1_aeroplane_1 = -nan
I1002 08:08:24.222121 415 solver.cpp:404] Test net output #12: top-1_aeroplane_2 = -nan
I1002 08:08:24.222144 415 solver.cpp:404] Test net output #13: top-1_aeroplane_3 = -nan
I1002 08:08:24.222167 415 solver.cpp:404] Test net output #14: top-1_aeroplane_4 = -nan
I1002 08:08:24.222190 415 solver.cpp:404] Test net output #15: top-1_aeroplane_5 = -nan
I1002 08:08:27.674801 415 solver.cpp:228] Iteration 0, loss = 0
I1002 08:08:27.674849 415 solver.cpp:244] Train net output #0: loss_aeroplane_1 = 0.015304
I1002 08:08:27.674875 415 solver.cpp:244] Train net output #1: loss_aeroplane_2 = 5.35489
I1002 08:08:27.674890 415 solver.cpp:244] Train net output #2: loss_aeroplane_3 = 0
I1002 08:08:27.674903 415 solver.cpp:244] Train net output #3: loss_aeroplane_4 = 22.1282
I1002 08:08:27.674918 415 solver.cpp:244] Train net output #4: loss_aeroplane_5 = 4.31547
I1002 08:08:27.675045 415 solver.cpp:244] Train net output #11: top-1_aeroplane_1 = 1
I1002 08:08:27.675061 415 solver.cpp:244] Train net output #12: top-1_aeroplane_2 = 0
I1002 08:08:27.675077 415 solver.cpp:244] Train net output #13: top-1_aeroplane_3 = -nan
I1002 08:08:27.675091 415 solver.cpp:244] Train net output #14: top-1_aeroplane_4 = 0
I1002 08:08:27.675112 415 solver.cpp:244] Train net output #15: top-1_aeroplane_5 = 0.5
I1002 08:08:27.675243 415 sgd_solver.cpp:106] Iteration 0, lr = 1e-06
I1002 08:14:02.664109 415 solver.cpp:337] Iteration 100, Testing net (#0)
I1002 08:14:02.720834 415 solver.cpp:404] Test net output #0: loss_aeroplane_1 = 0
I1002 08:14:02.720870 415 solver.cpp:404] Test net output #1: loss_aeroplane_2 = 0
I1002 08:14:02.720908 415 solver.cpp:404] Test net output #2: loss_aeroplane_3 = 0
I1002 08:14:02.720922 415 solver.cpp:404] Test net output #3: loss_aeroplane_4 = 0
I1002 08:14:02.720942 415 solver.cpp:404] Test net output #4: loss_aeroplane_5 = 0
I1002 08:14:02.721050 415 solver.cpp:404] Test net output #11: top-1_aeroplane_1 = -nan
I1002 08:14:02.721076 415 solver.cpp:404] Test net output #12: top-1_aeroplane_2 = -nan
I1002 08:14:02.721098 415 solver.cpp:404] Test net output #13: top-1_aeroplane_3 = -nan
I1002 08:14:02.721122 415 solver.cpp:404] Test net output #14: top-1_aeroplane_4 = -nan
I1002 08:14:02.721133 415 solver.cpp:404] Test net output #15: top-1_aeroplane_5 = -nan
I1002 08:14:06.076464 415 solver.cpp:228] Iteration 100, loss = 0.977386
I1002 08:14:06.076515 415 solver.cpp:244] Train net output #0: loss_aeroplane_1 = 0
I1002 08:14:06.076532 415 solver.cpp:244] Train net output #1: loss_aeroplane_2 = 5.73755
I1002 08:14:06.076545 415 solver.cpp:244] Train net output #2: loss_aeroplane_3 = 0
I1002 08:14:06.076555 415 solver.cpp:244] Train net output #3: loss_aeroplane_4 = 5.3247e-06
I1002 08:14:06.076563 415 solver.cpp:244] Train net output #4: loss_aeroplane_5 = 0
I1002 08:14:06.076637 415 solver.cpp:244] Train net output #11: top-1_aeroplane_1 = -nan
I1002 08:14:06.076644 415 solver.cpp:244] Train net output #12: top-1_aeroplane_2 = 0
I1002 08:14:06.076653 415 solver.cpp:244] Train net output #13: top-1_aeroplane_3 = -nan
I1002 08:14:06.076669 415 solver.cpp:244] Train net output #14: top-1_aeroplane_4 = 1
I1002 08:14:06.076690 415 solver.cpp:244] Train net output #15: top-1_aeroplane_5 = -nan
I1002 08:14:06.076812 415 sgd_solver.cpp:106] Iteration 100, lr = 1e-06
I1002 08:19:40.601531 415 solver.cpp:337] Iteration 200, Testing net (#0)
I1002 08:19:40.658157 415 solver.cpp:404] Test net output #0: loss_aeroplane_1 = 0
I1002 08:19:40.658193 415 solver.cpp:404] Test net output #1: loss_aeroplane_2 = 0
I1002 08:19:40.658202 415 solver.cpp:404] Test net output #2: loss_aeroplane_3 = 0
I1002 08:19:40.658221 415 solver.cpp:404] Test net output #3: loss_aeroplane_4 = 0
I1002 08:19:40.658231 415 solver.cpp:404] Test net output #4: loss_aeroplane_5 = 0
I1002 08:19:40.658363 415 solver.cpp:404] Test net output #11: top-1_aeroplane_1 = -nan
I1002 08:19:40.658376 415 solver.cpp:404] Test net output #12: top-1_aeroplane_2 = -nan
I1002 08:19:40.658387 415 solver.cpp:404] Test net output #13: top-1_aeroplane_3 = -nan
I1002 08:19:40.658407 415 solver.cpp:404] Test net output #14: top-1_aeroplane_4 = -nan
I1002 08:19:40.658418 415 solver.cpp:404] Test net output #15: top-1_aeroplane_5 = 1
I1002 08:19:44.013634 415 solver.cpp:228] Iteration 200, loss = 1.24157
I1002 08:19:44.013686 415 solver.cpp:244] Train net output #0: loss_aeroplane_1 = 1.98091
I1002 08:19:44.013706 415 solver.cpp:244] Train net output #1: loss_aeroplane_2 = 6.87694
I1002 08:19:44.013716 415 solver.cpp:244] Train net output #2: loss_aeroplane_3 = 1.29873
I1002 08:19:44.013727 415 solver.cpp:244] Train net output #3: loss_aeroplane_4 = 17.1241
I1002 08:19:44.013738 415 solver.cpp:244] Train net output #4: loss_aeroplane_5 = 0
I1002 08:19:44.013875 415 solver.cpp:244] Train net output #11: top-1_aeroplane_1 = 0.75
I1002 08:19:44.013890 415 solver.cpp:244] Train net output #12: top-1_aeroplane_2 = 0.5
I1002 08:19:44.013903 415 solver.cpp:244] Train net output #13: top-1_aeroplane_3 = 0
I1002 08:19:44.013917 415 solver.cpp:244] Train net output #14: top-1_aeroplane_4 = 0
I1002 08:19:44.013931 415 solver.cpp:244] Train net output #15: top-1_aeroplane_5 = 1
I1002 08:19:44.014006 415 sgd_solver.cpp:106] Iteration 200, lr = 1e-06
I1002 08:25:19.499899 415 solver.cpp:337] Iteration 300, Testing net (#0)
I1002 08:25:19.556656 415 solver.cpp:404] Test net output #0: loss_aeroplane_1 = 0
I1002 08:25:19.556694 415 solver.cpp:404] Test net output #1: loss_aeroplane_2 = 0
I1002 08:25:19.556702 415 solver.cpp:404] Test net output #2: loss_aeroplane_3 = 0
I1002 08:25:19.556730 415 solver.cpp:404] Test net output #3: loss_aeroplane_4 = 0
I1002 08:25:19.556743 415 solver.cpp:404] Test net output #4: loss_aeroplane_5 = 0
I1002 08:25:19.556854 415 solver.cpp:404] Test net output #11: top-1_aeroplane_1 = -nan
I1002 08:25:19.556874 415 solver.cpp:404] Test net output #12: top-1_aeroplane_2 = -nan
I1002 08:25:19.556893 415 solver.cpp:404] Test net output #13: top-1_aeroplane_3 = -nan
I1002 08:25:19.556913 415 solver.cpp:404] Test net output #14: top-1_aeroplane_4 = -nan
I1002 08:25:19.556931 415 solver.cpp:404] Test net output #15: top-1_aeroplane_5 = -nan
I1002 08:25:22.910521 415 solver.cpp:228] Iteration 300, loss = 1.06406
I1002 08:25:22.910571 415 solver.cpp:244] Train net output #0: loss_aeroplane_1 = 0.776529
I1002 08:25:22.910589 415 solver.cpp:244] Train net output #1: loss_aeroplane_2 = 8.75287
I1002 08:25:22.910600 415 solver.cpp:244] Train net output #2: loss_aeroplane_3 = 4.64917e-06
I1002 08:25:22.910616 415 solver.cpp:244] Train net output #3: loss_aeroplane_4 = 12.4562
I1002 08:25:22.910634 415 solver.cpp:244] Train net output #4: loss_aeroplane_5 = 7.18792
I1002 08:25:22.910776 415 solver.cpp:244] Train net output #11: top-1_aeroplane_1 = 0
I1002 08:25:22.910794 415 solver.cpp:244] Train net output #12: top-1_aeroplane_2 = 0
I1002 08:25:22.910815 415 solver.cpp:244] Train net output #13: top-1_aeroplane_3 = 1
I1002 08:25:22.910849 415 solver.cpp:244] Train net output #14: top-1_aeroplane_4 = 0
I1002 08:25:22.910869 415 solver.cpp:244] Train net output #15: top-1_aeroplane_5 = 0
I1002 08:25:22.910969 415 sgd_solver.cpp:106] Iteration 300, lr = 1e-06
Also, can someone please let me know how is Caffe calculating loss here? Is this 'total loss' (sum of losses for all classes) or what?
I1002 08:25:22.910521 415 solver.cpp:228] Iteration 300, loss = 1.06406