Year: 2026
Type: preprint
Abstract: Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.
Source: arXiv (Cornell University)
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: FET: Small: Brain-Inspired Hyperdimensional Computing for IoT Applications, AI Institute for Learning-Enabled Optimization at Scale (TILOS), Category II: A Prototype National Research Platform, IUCRC Phase I University of California, San Diego: Center for Power Management Integration (PMIC), Collaborative Research: MLWiNS: Hyperdimensional Computing for Scalable IoT Intelligence Beyond the Edge, Collaborative Research: IIS: RI: Medium: Lifelong learning with hyper dimensional computing, CCRI: ENS: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI)
OpenAlex ID: W7140160147
Cites:
Cited by:
Related to: