Simultaneous directional inference

We consider the problem of inference on the signs of n>1 parameters. We aim to provide 1-α post hoc confidence bounds on the number of positive and negative (or non-positive) parameters, with a simultaneous guarantee, for all subsets of parameters. We suggest to start by using the data to select the direction of the hypothesis test for each parameter; then, adjust the p-values of the one-sided hypotheses for the selection, and use the adjusted p-values for simultaneous inference on the selected n one-sided hypotheses. The adjustment is straightforward assuming the p-values of one-sided hypotheses have densities with monotone likelihood ratio, and are mutually independent. We show the bounds we provide are tighter (often by a great margin) than existing alternatives, and that they can be obtained by at most a polynomial time. We demonstrate their usefulness in the evaluation of treatment effects across studies or subgroups. Specifically, we provide a tight lower bound on the number of studies which are beneficial, as well as on the number of studies which are harmful (or non-beneficial), and in addition conclude on the effect direction of individual studies, while guaranteeing that the probability of at least one wrong inference is at most 0.05.