How to compare two deep learning models performance?

Question

I'm new to deep learning so if the question doesn't make sense plz correct me.

In traditional machine learning I know how to compare models and choose one of the as the best with the metrics I choose.

However, in deep learning, each model is build with different layers, so how can I control variables to determine which model is the best fairly? Or usually people don't compare in this way?

For example I have a sequential data, I can use both CNN and LSTM model, so should I compare model with only one layer of CNN and one layer of LSTM? After that I can add more layers or tuning my model?

Or someone can just tell me the process of how to compare and choose the best deep learning model with chosen metrics?

Answer 1

For sequential data as you mention, Number of layers has nothing to do with the comparison of two models. At certain number of layers your accuracy will start to decrease because of overfitting. Comparing 1 layer of CNN with 1 layer of LSTM is not a correct approach.

You need to check the following factors for comparison like accuracy,precision,recall,f1-score depending on your application objective.

For example , if you are working on the language translation data LSTM would be better choice, because it over comes the problem of vanishing gradient.