Rethinking collaborative filtering: A practical perspective on state-of-the-art research based on real world insights

A decade has passed since the seminal Netflix Prize competition and Collaborative Filtering (CF) models are still at the forefront of Recommender System research. Significant progress has been achieved over this time, yet key aspects of the basic problem formulation have not been seriously challenged. Most state-of-the-art models still assume a supervised model in which the ultimate goal is to predict future user-item interactions based on the generalization of historical data. We wish to initiate a discussion on some key assumptions behind much of the mainstream research: What is the difference between predicting future user actions and optimizing Key Performance Indicators (KPIs)? Does a trade-off between accuracy and diversity really exist? Is supervised CF based on historical data still relevant in the age of modern reinforcement learning models such as contextual bandits? What evaluation metrics can be used prior to online experimentations and what are their limitations? Our main thesis is that at the core of all these issues lies a gap between most Collaborative Filtering models and the true objective of industry systems.