Work

Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents

Year: 2026

Type: conference-paper

Abstract: Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks.However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability.In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories.STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task.By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress.Together, these components yield augmented step-level transitions for both high-level and low-level policies.Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.318

Authors: Shuai Zhen, Yanhua Yu, Ruopei Guo, Nan Cheng, Yang Deng

Institutions: Beijing University of Posts and Telecommunications, China Mobile Group Design Institute Co., Ltd. (China), Singapore Management University

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Natural Science Foundation of China, Beijing University of Posts and Telecommunications, China Scholarship Council

OpenAlex ID: W7166876537