Year: 2020
Type: conference-paper
Abstract: We propose an efficient batching strategy for variable-length decoding on GPU architectures.During decoding, when candidates terminate or are pruned according to heuristics, our streaming approach periodically "refills" the batch before proceeding with a selected subset of candidates.We apply our method to variable-width beam search on a state-of-theart machine translation model.Our method decreases runtime by up to 71% compared to a fixed-width beam search baseline and 17% compared to a variable-width baseline, while matching baselines' BLEU.Finally, experiments show that our method can speed up decoding in other domains, such as semantic and syntactic parsing.
Authors: Kevin Yang, Violet Yao, John DeNero, Dan Klein
Institution: University of California, Berkeley
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W3089846531