Year: 2024
Type: conference-paper
Abstract: In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks.We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks.Then, we used nine popular LLMs with five promptengineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6.Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks.In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains.All code, data, and supplemental materials are available on GitHub at: https://github.com/ matthewrenze/jhu-llm-temperature.
Author: Matthew Renze
Institution: Johns Hopkins University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4404781852