Tool-Aware Planning for Contact-Center Analytics: Evaluating LLMs through Lineage-Guided Query Decomposition
Year: 2026
Type: conference-paper
Abstract: We present a domain-grounded benchmark and evaluation framework for tool-aware plan generation in contact-center analytics, where answering a business-insights query requires decomposing it into executable steps over structured tools (Text2SQL over Snowflake), unstructured tools (RAG over transcripts), and LLM-based synthesis, with explicit depends_on relations for safe parallel execution.Our contributions are threefold: (i) a reference-based plan evaluation framework with two complementary views-a metric-wise evaluator spanning seven dimensions (e.g., tool-prompt alignment, query adherence) and a one-shot evaluator that compares a candidate plan against a reference plan; (ii) a lineagedriven data curation methodology that uses an iterative evaluator→optimizer loop to refine initial plans into high-quality plan lineages while reducing manual effort; and (iii) a large-scale study of 14 LLMs across model families and sizes on their ability to generate step-by-step, executable, tool-assigned plans, evaluated with and without lineage in the prompt.Empirically, LLMs continue to struggle on compound queries and on plans longer than four steps; the highest aggregate metric-wise score is 84.8 (Claude-3-7-Sonnet), while the strongest oneshot A+ rate (Extremely Good or Very Good) is only 49.75% (o3-mini).Lineage yields mixed overall gains but improves several strong models and often helps step executability.Overall, our results expose persistent weaknesses in tool understanding-especially tool-prompt alignment and tool-usage completeness, and, show that shorter, simpler plans remain markedly easier.The benchmark, evaluation framework, and findings provide a practical path for assessing and improving agentic planning with tools in enterprise question-answering settings 1 . 1 Public dataset (https://github.com/ Observeai-Research/tool-aware-planning-dataset/) with human-annotated reference plans, plan lineages, and per-planner outputs for all 14 planners Anthropic.2024.Claude 3.5 Haiku.https: //www.anthropic.com/news
Authors: Varun Nathan, Shreyas Guha, Ayush Kumar
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166844353