Year: 2026
Type: conference-paper
Abstract: Prompt underspecification is a common challenge when interacting with LLMs.In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer unspecified requirements by default (41.1%), such behavior is fragile: Underspecified prompts are 2x as likely to regress across model or prompt changes, sometimes with accuracy drops exceeding 20%. 1 This instability makes it difficult to reliably build LLM applications.Moreover, simply specifying all requirements does not consistently help, as models have limited instruction-following ability and requirements can conflict.Standard prompt optimizers likewise provide little benefit.To address these issues, we propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% on average over baselines.We further advocate for a systematic process of proactive requirements discovery, evaluation, and monitoring to better manage prompt underspecification in practice.* Now at Google DeepMind.
Authors: Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kaestner, Tongshuang Wu
Institution: Carnegie Mellon University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4417287666