Year: 2023
Type: conference-paper
Abstract: This work aims at decreasing the end-to-end generation latency of large language models (LLMs).One of the major causes of the high generation latency is the sequential decoding approach adopted by almost all state-of-the-art LLMs.In this work, motivated by the thinking and writing process of humans, we propose Skeleton-of-Thought (SoT), which first guides LLMs to generate the skeleton of the answer, and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point in parallel.Not only does SoT provide considerable speed-ups across 12 LLMs, but it can also potentially improve the answer quality on several question categories.SoT is an initial attempt at datacentric optimization for inference efficiency, and further underscores the potential of pushing LLMs to think more like a human for answer quality. * Equal contribution.† The main updates in arXiv V2 are as follows: (1) Add the quality and efficiency evaluation of SoT on GPT-4.(2) Use GPT-4 as the judge for answer quality evaluation.The old results with ChatGPT-3.5 as the judge are moved to App.I.3.(3) Add the SoT with Router (SoT-R) method ( § 4) which adaptively triggers SoT on suitable questions.( 4) Move detailed answer analysis to the appendices.
Source: Lirias
Authors: Ning, Xuefei, Lin, Zinan, Zhou, Zixuan, Wang, Zifu, Yang, Huazhong, Wang, Yu
Institutions: Tsinghua University, Microsoft (United States), KU Leuven
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Microsoft Research
OpenAlex ID: W7034070351