Abstract
An analyst is given a training set consisting of regression datasets Dj of different sizes, which are distributed according to some Gj, j = 1,…,J, where the distributions Gj are assumed to form a random sample generated by some common source. In particular, the Dj ’s have a common set of covariates and they are all labeled. The training set is used by the analyst for selection of subsets of covariates denoted by P*(n), whose role is described next. The multi-task problem we consider is as follows: given a number of random labeled datasets (which may be in the training set or not) DJk of size nk, k =1,…,K, estimate separately for each dataset the regression coefficients on the subset of covariates P*(nk) and then predict future dependent variables given their covariates. Naturally, a large sample size nk of DJk allows a larger subset of covariates, and the dependence of the size of the selected covariate subsets on nk is needed in order to achieve good prediction and avoid overfitting. Subset selection is notoriously difficult and computationally demanding, and requires large samples; using all the regression datasets in the training set together amounts to borrowing strength toward better selection under suitable assumptions. Furthermore, using common subsets for all regressions having a given sample size standardizes and simplifies the data collection and avoids having to select and use a different subset for each prediction task. Our approach is efficient when the relevant covariates for prediction are common to the different regressions, while the models’ coefficients may vary between different regressions. Last but not least, we propose a simple and meaningful measure, GENO, that allows comparisons of the predictive value of different subsets of covariates by comparing the sample size they require in order to achieve the same prediction error.
Original language | English |
---|---|
Pages (from-to) | 4966-5013 |
Number of pages | 48 |
Journal | Electronic Journal of Statistics |
Volume | 15 |
Issue number | 2 |
DOIs | |
State | Published - 2021 |
Keywords
- Equivalent number of observations (ENO)
- GENO
- Mallows C
- Model selection
- Random covariates
- Transfer learning
All Science Journal Classification (ASJC) codes
- Statistics and Probability
- Statistics, Probability and Uncertainty