Risk Estimation with Active Labeling

We consider a setting where, for a given model, with a given labeling budget, we need to accurately evaluate its risk repeatedly, when the set of items considered for the risk evaluation, as well as the loss function, change over time. This is a natural setting in a non-stationary environment, such as that of large retail chains, whose catalogue changes year-round. Since evaluating risk often requires human judgement, the cost can increase dramatically over time. We propose a new estimator that minimize the labeling cost by reusing all available labels when possible and by actively selecting items to be labeled in an optimal way. We show that an optimal sampling profile can be derived, efficiently and at scale, as the solution of an optimization problem. We show how this approach with only a small added computational and storage cost, can efficiently reduce the labeling work required to measure the risk of a model in a non-stationary environment in a production system. We extend these results to the F_α measure and weighted risks. The presented approach is related to the Horvitz-Thompson estimator, importance sampling and active learning and provides a scalable, robust solution for risk evaluation in non-stationary environment that cannot be achieved with either Horvitz-Thompson estimator nor importance sampling.