Who's a good decision maker? Data-driven expert worker ranking under unobservable quality

Tomer Geva, Maytal Saar-Tsechansky

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Evaluation of expert workers by their decision quality has substantial practical value, yet using other expert workers for decision quality evaluation tasks is costly and often infeasible. In this work, we frame the Ranking of Expert workers according to their unobserved decision Quality (REQ) - without resorting to evaluation by other experts - as a new Data Science problem. This problem is challenging, as the correct decisions are commonly unobservable and substantial parts of the information available to the decision maker is not available for retrospective decision evaluation. We propose a new machine learning approach to address this problem. We evaluate our method on one dataset representing real expert decisions and two public datasets, and find that our approach is successful in generating highly accurate rankings. Moreover, we observe that our approach's superiority over the baseline is particularly prominent as evaluation settings become increasingly challenging.

Original languageEnglish
Title of host publication2016 International Conference on Information Systems, ICIS 2016
PublisherAssociation for Information Systems
ISBN (Electronic)9780996683135
StatePublished - 2016
Event2016 International Conference on Information Systems, ICIS 2016 - Dublin, Ireland
Duration: 11 Dec 201614 Dec 2016

Publication series

Name2016 International Conference on Information Systems, ICIS 2016

Conference

Conference2016 International Conference on Information Systems, ICIS 2016
Country/TerritoryIreland
CityDublin
Period11/12/1614/12/16

Keywords

  • Decision evaluation
  • Predictive modeling
  • Supervised learning
  • Worker ranking

All Science Journal Classification (ASJC) codes

  • Information Systems

Fingerprint

Dive into the research topics of 'Who's a good decision maker? Data-driven expert worker ranking under unobservable quality'. Together they form a unique fingerprint.

Cite this