ZipML: An End-to-end Bitwise Framework for Dense Generalized Linear Models.
Year: 2016
Type: preprint
Abstract: We present ZipML, the first framework for training dense generalized linearmodels using end-to-end low-precision representation--in ZipML, all movements
of data, including those for input samples, model, and gradients, are
represented using as little as two bits per component. Within our framework, we
have successfully compressed, separately, the input data by 16x, gradient by
16x, and model by 16x while still getting the same training result. Even for
the most challenging datasets, we find that robust convergence can be ensured
using only an end-to-end 8-bit representation or a 6-bit representation if only
samples are quantized. Our work builds on previous research on using low-precision representations
for gradient and model in the context of stochastic gradient descent. Our main
technical contribution is a new set of techniques which allow the training
samples to be processed with low precision, without affecting the convergence
of the algorithm. In turn, this leads to a system where all data items move in
a quantized, low precision format. In particular, we first establish that
randomized rounding, while sufficient when quantizing the model and the
gradients, is biased when quantizing samples, and thus leads to a different
training result. We propose two new data representations which converge to the
same solution as in the original data representation both in theory and
empirically and require as little as 2-bits per component. As a result, if the
original data is stored as 32-bit floats, we decrease the bandwidth footprint
for each training iteration by up to 16x. Our results hold for models such as
linear regression and least squares SVM. ZipML raises interesting theoretical questions related to the robustness of
SGD to approximate data, model, and gradient representations. We conclude this
working paper by a description of ongoing work extending these preliminary
results.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2554844166