11.5 Evaluating Recommender Systems

Evaluation is often treated as a prediction task, but the way we split the data matters as much as the metric.

The unit of splitting is a rating, not a row of a table. If we randomly shuffle users the way we did in classification, we can leak a user’s later ratings into the training set and then “predict” their earlier ones. A standard protocol is:

  • hold out some observed entries, at random, or better, the most recent ratings of each user,
  • fit the model on the remaining observed entries only,
  • score only the held-out cells.

RMSE (and MAE) make sense when the system is really asked to predict a number of stars. RMSE was the official score of the Netflix Prize, and it is still the default number people report. It has drawbacks. In many cases we care more about higher ratings (such as 4 or 5) that signal strong interest, and about what appears at the top of a list. Ranking metrics such as Precision@\(K\), Recall@\(K\), or NDCG@\(K\) are usually closer to how recommendations are actually shown.

One more warning: when we compute Precision@\(K\) we have to decide what counts as a “relevant” item among the ones the user never saw. Held-out high ratings are the usual proxy. That proxy is imperfect, which is one reason live experiments still matter once a system is in production.