Work

POaaS: Minimal-Edit Prompt Optimization as a Service to Lift Accuracy and Cut Hallucinations on On-Device sLLMs

Year: 2026

Type: conference-paper

Abstract: Small language models (sLLMs) are increasingly deployed on-device, where imperfect user prompts-typos, unclear intent, or missing context-can trigger factual errors and hallucinations.Existing automatic prompt optimization (APO) methods were designed for large cloud LLMs and rely on search that often produces long, structured instructions; when executed under an on-device constraint where the same small model must act as optimizer and solver, these pipelines can waste context and even hurt accuracy.We propose POaaS, a minimal-edit prompt optimization layer that routes each query to lightweight specialists (Cleaner, Paraphraser, Fact-Adder) and merges their outputs under strict drift and length constraints, with a conservative skip policy for well-formed prompts.Under a strict fixedmodel setting with Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct,POaaS improves both task accuracy and factuality while representative APO baselines degrade them, and POaaS recovers up to +7.4% under token deletion and mixup.Overall, per-query conservative optimization is a practical alternative to searchheavy APO for on-device sLLMs.

Source: Fact Extraction and VERification Workshop (FEVER)

DOI: 10.18653/v1/2026.fever-1.2