I'm working on a project on recommender systems written by python using the Bayesian Personalized Ranking optimization. I am pretty confident my model learns the data I provided well enough, but now it's time I found out the exact model hyperparameters and try to avoid overfitting. Since the movielens dataset only provided me with 5-fold train-test datasets without validation sets, I want to split the original dataset by myself to validate my model.
Since the movielens dataset holds 943 user data with each user guaranteed to have ranked at least 20 movies, I'm thinking of splitting the data so that both TRAIN and TEST datasets contain the same number of users(e.g. 943), and distributing 80% of the implicit feedback data to TRAIN, and the other to TEST. After training validation will take place using the mean value of Recall at k precision through all 943 users.
Is this the right way to split the dataset? I am curious because the original movielens test dataset doesn't seem to contain test data for all 943 users. If a certain user doesn't have any test data to predict, how do I evaluate using recall@k -- when doing so would cause a zero division? Should I just skip that user and calculate the mean value with the rest of the users?
Thanks for the lengthy read, I hope you're not as confused as I am.
How I would split it is the whole data set on 80% (train)- 10% (validation) - 10% (test). It should work out :)