Confidence Intervals for the Parameters of the Most Frequent Events

Consider a survey asking people for their favorite food. We would like to report the K most popular foods in the sample, along with their associated confidence intervals (CIs). The common practice is to construct K marginal (binomial) CIs at a desired confidence level and expect a matching coverage rate. This approach is genuinely wrong, as a result of selective inference. Specifically, the inferred parameters are not fixed but data-dependent, and may vary among different samples. This means that standard binomial CIs, which implicitly assume fixed parameters, fail to provide the desired coverage rate. In this work we introduce a novel selective inference scheme for the most frequent events in the sample. Our proposed scheme is closed-form, intuitive, simple to apply and guarantees the prescribed confidence level. Further, we introduce a matching lower bound which demonstrates the tightness of our results. We illustrate our proposed scheme in synthetic and real-world experiments, showing a significant improvement over currently known alternatives.