CFDB: Machine Learning Model Analysis via Databases of CounterFactuals

Data Scientists often design and train complex Machine Learning models that evolve over time due to re-training on new data, a revised architecture, or both. To assist Data Scientists in this process, many methods for analyzing models have been recently developed. A prominent approach for model analysis is based on the notion of Counterfactuals. A counterfactual (CF) intuitively explains the label assigned by the model to a particular instance by identifying perturbations to the instance that lead to a different predicted label. A large body of recent literature has demonstrated the usefulness of CFs for deriving insights on the model at large. The analyzed CFs come in various flavors and are applied to instances chosen based on various criteria, in the context of different analysis goals. In this work we propose to demonstrate CFDB (Counterfactuals Database), a unified framework for querying Counterfactuals. CFDB allows to consolidate common approaches in CF-based analysis and to provide multiple levels of abstractions in a relational framework. We will demonstrate CFDB in the context of the Lending Club Loan Data, showing its usefulness by formulating and executing multiple analyses over evolving classifiers for Loan Approval.