I am building a slope one based recommender. The system has 40 million users and 1 million items. Each user has rated around 10 items. The deviation matrix is calculated as part of batch processing and is loaded in memory ( putting into a distributed cache)
The issue i am facing is performance at runtime !!! At run-time for recommendenation for a user the logic is as follows: for each item not rated by user (this num is quite high = 1 million - 10) for each item rated by user (10) calculate the avg deviation ...
Now the fundamental issue is the outer loop is too high ( 1 million - 10). Doing some many calculations at runtime will never perform !
Have you guys solved this issue ? Is there a trick to reduce this computaion at runtime ?
Thanks, Siddharth