Work

Who's Keeping Score? Interactive Steering of LLM-Powered Scoring with Attune

Year: 2026

Type: conference-paper

Abstract: Large language models (LLMs) are increasingly used to score text records at scale (e.g., rating candidate resumes on a 1–5 scale). However, existing LLM-powered approaches do not account for the fact that effective scoring requires both holistic understanding of records and locally consistent judgments across similar ones. We present Attune, a mixed-initiative system for steerable LLM-powered scoring. Given a task description and scoring range, Attune performs pairwise comparisons across records to develop a global understanding first, and then resolves these comparisons into consistent score assignments—deriving scoring criteria and rules bottom-up in the process. These serve as shared representations of scoring logic that users can inspect and edit. Based on insights from a formative study (n = 12), Attune’s interface introduces novel steering interactions that allow users to deterministically refine scoring logic. Users can provide examples, directly edit criteria, rules, or target distributions, and give natural language feedback—with all refinements compiling into constraints that guide re-scoring. We validate our approach through a technical evaluation across three workloads and a user study with domain experts (n = 8) in healthcare, law, education, and AI evaluation.

Source: ACM Symposium on User Interface Software and Technology (UIST)

DOI: 10.1145/3830398.3830567