FracGrad: A Discretized Riemann–Liouville Fractional Integral Approach to Gradient Accumulation for Deep Learning
Year: 2025
Type: article
Abstract: Gradient accumulation enables training large-scale deep learning models under GPU memory constraints by aggregating gradients across multiple microbatches before parameter updates. Standard gradient accumulation treats all microbatches uniformly through simple averaging, implicitly assuming that all stochastic gradient estimates are equally reliable. This assumption becomes problematic in non-convex optimization where gradient variance across microbatches is high, causing some gradient estimates to be noisy and less representative of the true descent direction. In this paper, FracGrad is proposed, a simple weighting scheme for gradient accumulation that biases toward recent microbatches via a power-law schedule derived from a discretized Riemann–Liouville integral. Unlike uniform summation, FracGrad reweights each microbatch gradient by wi(α)=(N−i+1)α−(N−i)α∑j=1N[(N−j+1)α−(N−j)α], controlled by α∈(0,1]. When α=1, standard accumulation is recovered. In experiments on mini-ImageNet with ResNet-18 using up to N=32 accumulation steps, the best FracGrad variant with α=0.1 improves test accuracy from 16.99% to 31.35% at N=16. Paired t-tests yield p≈2×10−6.
Source: Fractal and Fractional
Author: Minhyeok Lee
Institution: Chung-Ang University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,258
Award: G3074897916
OpenAlex ID: W4416226570