Year: 2025
Type: conference-paper
Abstract: Warning: This paper contains examples of stereotypes and biases.Measuring social bias in large language models (LLMs) is crucial, but existing bias evaluation methods struggle to assess bias in longform generation.We propose a Bias Benchmark for Generation (BBG), an adaptation of the Bias Benchmark for QA (BBQ), designed to evaluate social bias in long-form generation by having LLMs generate continuations of story prompts.Building our benchmark in English and Korean, we measure the probability of neutral and biased generations across ten LLMs.We also compare our long-form story generation evaluation results with multiplechoice BBQ evaluation, showing that the two approaches produce inconsistent results.
Authors: Jiho Jin, Woosung Kang, Junho Myung, Alice Oh
Institution: Korea Advanced Institute of Science and Technology
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: gold
Awards: G1931183695, G4700831490
OpenAlex ID: W4412888298