retraining the pre-trained model with new data and decreasing the model success with each training

Question

I have BraTS2020 dataset and I want to train my UNet model with it. Although I use Colab Pro, I cannot work with all data on Colab due to RAM limitation.

I divided the data into 4 parts and continued to retrain my saved model with each part. But with each new piece of data, the model's success decreased. Is there another way I can use all the data to increase the success of the model?

Answer 1

You are doing transfer learning.

While there are many parameters to tune that might explain this phenomenon, I would first try and reduce the learning-rate. Another thing to check: the image preprocessing/transform should be the same used for training the original model.