Yolov3 training new data on old custom trained weight

Question

I trained a model to detect water bottle and other of it's characteristics. i noticed that it's not detecting cap_missing properly so, i thought of training the trained weight (my old weight) on newer data.

1. I have downloaded the new dataset for it. Labeled it as well.
2. I also created the train.txt
3. The classes are same as my older model so, i haven't changed the cfg file or the .names file. .

As i run the command:

!./darknet detector train data/custom.data cfg/yolov3-custom.cfg backup/yolov3-custom_last.weights -dont_show

(on google colab)

the weight file is from the /backup folder where, the old training saved it's weights. . I get the result:

Pictures of Log: log-1 log-2

And here the cell stops without any training on new data.

Here is the drive link: https://drive.google.com/drive/folders/1KgSXcFjp8q2AuJ4EPuwQIDkgyIOYJDJA?usp=sharing

Answer 1

Both the problem and the solution to it are in your screenshots. You've told Darknet you want a certain number of max batches, and you've reached that exact limit. So Darknet is telling you there is nothing remaining to be done. The key text is this:

If you want to train from the beginning, then use flag in the end of training command: -clear

(This also tells me you are using an old version of Darknet, since last year I fixed up the text in newer versions of Darknet.)

If you want to re-use the weights to train some more, you must either raise your max batches to a higher value, or you need to use the -clear command to reset the image counter to zero. At that point, Darknet can continue to train.

This is also explained in the Darknet/YOLO FAQ here: https://www.ccoderun.ca/programming/yolo_faq/#training_command