Year: 2025
Type: preprint
Abstract: Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. _routing_). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform _unsupervised_ routing. We propose SMOOTHIE, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, SMOOTHIE constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown “true” outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that SMOOTHIE’s LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that SMOOTHIE outperforms baselines for routing by up to 10 points accuracy.
Source: Qeios
DOI: 10.32388/f319w4
Institution: Stanford University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Awards: AF: Medium: Collaborative Research: Beyond Sparsity: Refined Measures of Complexity for Linear Algebra, SHF: Medium: Collaborative Research: From Volume to Velocity: Big Data Analytics in Near-Realtime, Collaborative Research: Hardware-Aware Matrix Computations for Deep Learning Applications, RTML: Large: Continuous Adaptation for Decision Streams, Mobility Data Integration to Insight
OpenAlex ID: W4406202452