Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension

Research output: Contribution to journalArticlepeer-review


Understanding program code is a complicated endeavor. As a result, studying code comprehension is also hard. The prevailing approach for such studies is to use controlled experiments, where the difference between treatments sheds light on factors which affect comprehension. But it is hard to conduct controlled experiments with human developers, and we also need to find a way to operationalize what “comprehension” actually means. In addition, myriad different factors can influence the outcome, and seemingly small nuances may be detrimental to the study’s validity. In order to promote the development and use of sound experimental methodology, we discuss both considerations which need to be applied and potential problems that might occur, with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance. A common thread is that decisions that were taken in an effort to avoid one threat to validity may pose a larger threat than the one they removed.

Original languageAmerican English
Article number123
Number of pages42
JournalEmpirical Software Engineering
Issue number6
StatePublished - Nov 2022


  • Code comprehension
  • Controlled experiment
  • Experimental methodology
  • Threats to validity

All Science Journal Classification (ASJC) codes

  • Software


Dive into the research topics of 'Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension'. Together they form a unique fingerprint.

Cite this