Year: 2025
Type: preprint
Abstract: The practice of speculative decoding, whereby inference is probabilistically supported by a smaller, cheaper, ``drafter'' model, has become a standard technique for systematically reducing the decoding time of large language models. This paper conducts an analysis of speculative decoding through the lens of its potential disparate speed-up rates across tasks. Crucially, the paper shows that speed-up gained from speculative decoding is not uniformly distributed across tasks, consistently diminishing for under-fit, and often underrepresented tasks. To better understand this phenomenon, we derive an analysis to quantify this observed ``unfairness'' and draw attention to the factors that motivate such disparate speed-ups to emerge. Further, guided by these insights, the paper proposes a mitigation strategy designed to reduce speed-up disparities and validates the approach across several model pairs, revealing on average a 12% improvement in our fairness metric.
Source: arXiv (Cornell University)
Authors: Sandler, Jameson, Üstün, Ahmet, Romanelli, Marco, Hooker, Sara, Fioretto, Ferdinando
Institutions: University of Virginia, Hofstra University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: SaTC: CORE: Small: Privacy and Fairness in Critical Decision Making, Collaborative Research: RI: Small: End-to-end Learning of Fair and Explainable Schedules for Court Systems, RI:Small:Constrained Generative Models for Scientific and Engineering Applications, G6671297155, CAREER: End-to-end Constrained Optimization Learning
OpenAlex ID: W4414820862
Cites:
Cited by:
Related to: