Hugging Face BERT custom model cannot produce config.json when trainer(model) is saved

Question

`Hello experts,

I am trying to write BERT model with my own custom model (adding layer end of BERT).It goes well and I would like to save model to avoid future training. I am using trainer API and so, there are two ways to save model, right?

1. training_arugments and trainer (save the model under a folder)

This type of saving cannot save config.json although other files can be saved. If I am saving model with only BERT model without adding any layers, it can save all files including json.config.

2. trainer.save_model

This also cannot save json.config. So, the question is how can we get all completed saved information for custom model ?? becuase without config.json file we cannot load and resuse the model again. Your suggestions will be highly appreicated how to save the custom model. `

Answer 1

When using the HuggingFace Trainer class, the trainer's object typically handle saving the model and its configuration. However, when dealing with custom layers added to a pre-trained model like BERT (such as in your case), you might encounter issues with the automatic saving of the config.json file.

To address this, after training, you can manually create a configuration dictionary that includes details about the BERT and your custom added layers, and then use Python's json to save the config file. Alternatively, you can create a custom model class that includes your additional layers (which I assume you have already implemented). Then, override save_pretrained method to handle both BERT's configuration and your custom layers.