Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks
Year: 2018
Type: article
Abstract: Missing data is a ubiquitous problem. It is especially challenging in medical settings because many streams of measurements are collected at different-and often irregular-times. Accurate estimation of the missing measurements is critical for many reasons, including diagnosis, prognosis, and treatment. Existing methods address this estimation problem by interpolating within data streams or imputing across data streams (both of which ignore important information) or ignoring the temporal aspect of the data and imposing strong assumptions about the nature of the data-generating process and/or the pattern of missing data (both of which are especially problematic for medical data). We propose a new approach, based on a novel deep learning architecture that we call a Multi-directional Recurrent Neural Network that interpolates within data streams and imputes across data streams. We demonstrate the power of our approach by applying it to five real-world medical datasets. We show that it provides dramatically improved estimation of missing measurements in comparison to 11 state-of-the-art benchmarks (including Spline and Cubic Interpolations, MICE, MissForest, matrix completion, and several RNN methods); typical improvements in Root Mean Squared Error are between 35%-50%. Additional experiments based on the same five datasets demonstrate that the improvements provided by our method are extremely robust.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: EAGER-DynamicData: Real-time Discovery and Timely Event Detection from Dynamic and Multi-Modal Data Streams, Online Learning in Big-Data Stream Mining, PFI:BIC Human-Centered Smart-Integration of Mobile Imaging and Sensing Tools with Machine Learning for Ubiquitous Quantification of Waterborne and Airborne Nanoparticles
OpenAlex ID: W2770565645