Design choices in randomization tests that affect power

Abba M. Krieger, David Azriel, Michael Sklar, Adam Kapelner

Research output: Contribution to journalArticlepeer-review

Abstract

We consider the problem of evaluating designs for a small-sample two-arm randomized experiment using the power of the one-sided randomization test as a criterion. Our evaluation assumes a linear response in one observed covariate, an unobserved component and an additive treatment effect where the model's randomness is due to different treatment allocations. It is well-known that the power depends on the allocations’ imbalance in the observed covariate. We show that power is additionally affected by two other design choices: the number of allocations and the degree of the allocations’ dependence. We prove that the more allocations, the higher the power and the lower the variability in power. Our theoretical findings and simulation studies show that the designs with the highest power provide thousands of highly independent allocations each providing nominal imbalance in the observed covariates. These high-powered designs exhibit less randomness than complete randomization and more randomness than recently proposed designs that employ numerical optimization. This advantage of high power is easily accessible to practicing experimenters via the popular rerandomization design and a greedy pair switching design, where both outperform complete randomization and numerical optimization. The tradeoff we find also provides a means to specify rerandomization’s imbalance threshold parameter.

Original languageEnglish
JournalCommunications in Statistics - Theory and Methods
DOIs
StateAccepted/In press - 2022

Keywords

  • Experimental design
  • optimization
  • rerandomization
  • restricted randomization
  • statistical power

All Science Journal Classification (ASJC) codes

  • Statistics and Probability

Fingerprint

Dive into the research topics of 'Design choices in randomization tests that affect power'. Together they form a unique fingerprint.

Cite this