Work

Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models

Year: 2025

Type: conference-paper

Abstract: Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks.Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step.However, the performance of IR models in tool retrieval tasks remains underexplored and unclear.Most tooluse benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios.In this paper, we propose TOOLRET, a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets.We benchmark six types of models on TOOL-RET.Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on TOOLRET.This low retrieval quality degrades the task pass rate of tool-use LLMs.As a further step, we contribute a large-scale training dataset with over 200k instances, which substantially optimizes the tool retrieval ability of IR models. 1

DOI: 10.18653/v1/2025.findings-acl.1258