Year: 2015
Type: conference-paper
Abstract: At the core of Machine Learning (ML) analytics is often an expert-suggested model, whose parameters are refined by iteratively processing a training dataset until convergence. The completion time (i.e. convergence time) and quality of the learned model not only depends on the rate at which the refinements are generated but also the quality of each refinement. While data-parallel ML applications often employ a loose consistency model when updating shared model parameters to maximize parallelism, the accumulated error may seriously impact the quality of refinements and thus delay completion time, a problem that usually gets worse with scale. Although more immediate propagation of updates reduces the accumulated error, this strategy is limited by physical network bandwidth. Additionally, the performance of the widely used stochastic gradient descent (SGD) algorithm is sensitive to step size. Simply increasing communication often fails to bring improvement without tuning step size accordingly and tedious hand tuning is usually needed to achieve optimal performance.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
Awards: Collaborative Research: PRObE - The NSF Parallel Reconfigurable Observational Environment for Data Intensive Super-Computing and High End Computing, G5800872287, Collaborative Research: PRObE - The NSF Parallel Reconfigurable Observational Environment for Data Intensive Super-Computing and High End Computing, BIGDATA: F: DKA: Collaborative Research: Theory and Algorithms for Parallel Probabilistic Inference with Big Data, via Big Model, in Realistic Distributed Computing Environments, Genome-Transcription-Phenome-Wide Association: a new paradigm for association stu, G8790040577
OpenAlex ID: W1969805974