Scalable Estimation for High Velocity Survival Data Able to Accommodate Addition of Covariates
Year: 2022
Type: article
Abstract: With the rapidly increasing availability of large-scale streaming data, there has been a growing interest in developing methods that allow the processing of the data in batches without requiring storage of the full dataset.In this paper, we propose a hybrid likelihood approach for scalable estimation of the Cox model using individual-level data in the current data batch and summary statistics calculated from historical data.We show that the proposed scalable estimator is asymptotically as efficient as the maximum likelihood estimator calculated using the entire dataset with low data storage requirements and low loading and computation time.A challenge in analyzing survival data batches that is not accommodated in extant methods is that new covariates may become available midway through data collection.To accommodate addition of covariates, we develop a hybrid empirical likelihood approach to incorporate the historical covariate effects evaluated in a reduced Cox model.The extended scalable estimator is asymptotically more efficient 1 Statistica Sinica: Newly accepted Paper (accepted author-version subject to English editing) than the maximum likelihood estimator obtained using only the data batches that include the additional covariates.The proposed approaches are evaluated by numerical simulations and illustrated with an analysis of Surveillance, Epidemiology, and End Results (SEER) breast cancer data.
Source: Statistica Sinica
Authors: Ying Sheng, Yifei Sun, Charles E. McCulloch, Chiung-Yu Huang
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4308599494