Do hard topics exist? A statistical analysis

J. Shane Culpepper, Guglielmo Faggioli, Nicola Ferro, Oren Kurland

Research output: Contribution to journalConference articlepeer-review


Several recent studies have explored the interaction effects between topics, systems, corpora, and components when measuring retrieval effectiveness. However, all of these previous studies assume that a topic or information need is represented by a single query. In reality, users routinely reformulate queries to satisfy an information need. Recently there has been renewed interest in the notion of “query variations” which are essentially multiple user formulations for an information need. Like many retrieval models, some queries are highly effective while others are not. In this work1, we explore the fundamental problem of studying the interaction components of an IR experimental collection. Our findings show that query formulations have a comparable effect size to the topic factor itself, which is known to be the factor with the greatest effect size in prior ANOVA studies. This suggests that topic difficulty is an artifact of the collection considered and highlights the importance of further research in understanding link between the complexity of a topic and the query rewriting in IR related tasks.

Original languageEnglish
JournalCEUR Workshop Proceedings
StatePublished - 2021
Event11th Italian Information Retrieval Workshop, IIR 2021 - Bari, Italy
Duration: 13 Sep 202115 Sep 2021

All Science Journal Classification (ASJC) codes

  • Computer Science(all)


Dive into the research topics of 'Do hard topics exist? A statistical analysis'. Together they form a unique fingerprint.

Cite this