Analyzing Big EHR Data—Optimal Cox Regression Subsampling Procedure with Rare Events

Nir Keret, Malka Gorfine

Research output: Contribution to journalArticlepeer-review


Massive sized survival datasets become increasingly prevalent with the development of the healthcare industry, and pose computational challenges unprecedented in traditional survival analysis use cases. In this work we analyze the UK-biobank colorectal cancer data with genetic and environmental risk factors, including a time-dependent coefficient, which transforms the dataset into “pseudo-observation” form, thus, critically inflating its size. A popular way for coping with massive datasets is downsampling them, such that the computational resources can be afforded by the researcher. Cox regression has remained one of the most popular statistical models for the analysis of survival data to-date. This work addresses the settings of right censored and possibly left-truncated data with rare events, such that the observed failure times constitute only a small portion of the overall sample. We propose Cox regression subsampling-based estimators that approximate their full-data partial-likelihood-based counterparts, by assigning optimal sampling probabilities to censored observations, and including all observed failures in the analysis. The suggested methodology is applied on the UK-biobank for building a colorectal cancer risk-prediction model, while reducing the computation time and memory requirements. Asymptotic properties of the proposed estimators are established under suitable regularity conditions, and simulation studies are carried out to evaluate their finite sample performance. Supplementary materials for this article are available online.

Original languageEnglish
Pages (from-to)2262-2275
Number of pages14
JournalJournal of the American Statistical Association
Issue number544
StatePublished - 2023


  • A-optimal
  • Big data
  • L-optimal
  • Left-truncation
  • Loewner ordering
  • Optimal sampling
  • Rare events
  • Reservoir sampling
  • Survival analysis

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Statistics, Probability and Uncertainty


Dive into the research topics of 'Analyzing Big EHR Data—Optimal Cox Regression Subsampling Procedure with Rare Events'. Together they form a unique fingerprint.

Cite this