Work

Measuring a Scheduling Agent Honestly: Failure Modes of Model-Driven Agent Loops and a Blind-Set Evaluation of a Read-Only-Model Design

Year: 2026
Type: preprint
Abstract: Agent frameworks commonly place a large language model in the control loop: the model chooses tools, sequences them, and decides when a task is complete. We report on operating such a design for meeting scheduling, a domain where most errors are visible to people outside the organization and cannot be undone. From our engineering record we derive a taxonomy of eight recurring failure modes, and observe that every mitigation we shipped took a decision away from the model and added latency and cost. We then describe a replacement design in which the model's only role is to read a request into a fixed, validated form, and all planning, time arithmetic and execution are ordinary code behind a single human approval gate. Finally we describe an evaluation protocol built to resist self-deception (situations rather than sentences, independently written phrasings, a held-back half and a blind final set, three runs per phrasing, and release-blocking serious errors) and report current results without adjustment: 93.8% on everyday requests and 94.0% on requests involving outside guests on the blind final set, against our own launch target of 99.5%, with zero serious errors in either suite.
Author: Lal, Raj
Language: English
Open Access status: green
OpenAlex ID: W7235378224
Cites:
Cited by:
Related to: 10