Work

Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents

Year: 2026

Type: conference-paper

Abstract: Vision-and-Language Navigation (VLN) poses significant challenges for agents to interpret natural language instructions and navigate complex 3D environments.While recent progress has been driven by large-scale pre-training and data augmentation, current methods still struggle to generalize to unseen scenarios, particularly when complex spatial and temporal reasoning is required.In this work, we propose SkillNav, a modular framework that introduces structured, skill-based reasoning into Transformer-based VLN agents.Our method decomposes navigation into a set of interpretable atomic skills (e.g., Vertical Movement, Area and Region Identification, Stop and Pause), each handled by a specialized agent.To support targeted skill training without manual data annotation, we construct a synthetic dataset pipeline that generates diverse, linguistically natural, skill-specific instructiontrajectory pairs.We then introduce a novel training-free Vision-Language Model (VLM)based router, which dynamically selects the most suitable agent at each time step by aligning sub-goals with visual observations and previous actions.SkillNav obtains competitive results on commonly used benchmarks and establishes state-of-the-art generalization on GSA-R2R, a benchmark with novel instruction styles and unseen environments.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.595

Authors: Tianyi Ma, Yue Zhang, Zehao Wang, Parisa Kordjamshidi

Institutions: Michigan State University, KU Leuven

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Office of Naval Research

Award: G8876996369