Get started
Work

3D visual grounding based on active perception

Year: 2026
Type: article
Abstract: 3D visual grounding (3DVG) is a key capability for embodied agents, requiring a robot to localize in 3D the object referred to by a free-form language expression. Existing LLM/VLM-based approaches often make one-shot decisions from incomplete observations or invoke expensive multimodal reasoning over the full scene, leading to ambiguous grounding, hallucinatory predictions, and limited reliability. We address this challenge by formulating 3DVG as state-driven closed-loop grounding over a reusable scene memory. Given a query decomposed into a context-ordered grounding sequence, our framework maintains the current candidate set and accumulated structured evidence, predicts informative viewpoints for active perception, and performs candidate-conditioned verification only within the projected local region of the predicted view. Newly acquired observations are sparsely written back to scene memory, enabling local memory recovery and candidate refresh without global recomputation. We further introduce a feedback-driven re-grounding mechanism with resolved, ambiguous, missing, and conflict states: the system advances only when a unique candidate is verified; otherwise, it continues active perception or backtracks to recover upstream anchors. By using the VLM only for candidate verification while reading final 3D geometry from scene memory, and by confining high-cost multimodal reasoning to active local candidates, the proposed framework improves robustness under partial observability and mitigates hallucination risk. Experiments on ScanRefer, Sr3D, Nr3D, and OpenTarget demonstrate the effectiveness of our method and show strong performance against competitive baselines.
DOI: 10.1007/s40747-026-02514-x
Author: Liang Geng
Language: English
Open Access status: gold
Estimated APC paid: $2,490
OpenAlex ID: W7212127239
FWCI:
Cites:
Cited by:
Related to: