Intelligent Calibration for Bias Reduction in Sentiment Corpora Annotation Process

Idan Toker, David Sarne, Jonathan Schler

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


This paper focuses in the inherent anchoring bias present in sequential reviews-sentiment corpora annotation processes. It proposes employing a limited subset of meticulously chosen reviews at the outset of the process, as a means of calibration, effectively mitigating the phenomenon. Through extensive experimentation we validate the phenomenon of sentiment bias in the annotation process and show that its magnitude can be influenced by pre-calibration. Furthermore, we show that the choice of the calibration set matters, hence the need for effective guidelines for choosing the reviews to be included in it. A comparison of annotators performance with the proposed calibration to annotation processes that do not use calibration or use a randomly-picked calibration set, reveals that indeed the calibration set picked is highly effective-it manages to substantially reduce the average absolute error compared to the other cases. Furthermore, the proposed selection guidelines are found to be highly robust in picking an effective calibration set also for domains different than the one based on which these rules were extracted.

Original languageEnglish
Title of host publicationTechnical Tracks 14
EditorsMichael Wooldridge, Jennifer Dy, Sriraam Natarajan
Number of pages8
ISBN (Electronic)1577358872, 9781577358879
StatePublished - 25 Mar 2024
Event38th AAAI Conference on Artificial Intelligence, AAAI 2024 - Vancouver, Canada
Duration: 20 Feb 202427 Feb 2024

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence


Conference38th AAAI Conference on Artificial Intelligence, AAAI 2024

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence


Dive into the research topics of 'Intelligent Calibration for Bias Reduction in Sentiment Corpora Annotation Process'. Together they form a unique fingerprint.

Cite this