Get started
Work

GLUE3D: General language understanding evaluation for 3D point clouds

Year: 2025
Type: article
Abstract: • We propose GLUE3D, a novel multimodal benchmark specifically designed to evaluate Multimodal Large Language Models through binary, multiple-choice, and captioning tasks in both the three-dimensional and vision domains. • Our results demonstrate a substantial performance gap between 2DsLLMs and 3D-LLMs, with 3D models often failing to outperform even textual-only baselines. • As part of our benchmark, we release 128 carefully curated textured polygon meshes. Polygon meshes support generation of several more 3D representations, enabling future extensions to alternative modalities and a ensuring long-term adaptability of our benchmark. Multimodal Large Language Models have achieved impressive results on text and image benchmarks, yet their capacity to ground language in 3D geometry is still largely unexplored. Existing 3D evaluations are either confined to specialised domains, such as indoor scans, or hampered by poor texture fidelity, and none allow a fair, modality-aligned comparison with the 2D counterparts. Without a rigorous benchmark, it remains unclear whether current 3D-aware models genuinely grasp shape, colour, pose, and quantity, or merely echo memorised textual priors. We address this gap with GLUE3D ( General Language Understanding Evaluation for 3D Point Clouds ), a benchmark built around 128 richly textured meshes spanning creatures, objects, architecture and transport. Each asset is provided both as a 50 k-point RGB point cloud and as a matched 512 × 512 rendering, enabling point-for-point evaluation across modalities. Over these assets we manually curate 1024 binary probes, 256 multiple-choice questions, 256 open-ended questions, and 128 caption prompts that jointly evaluate sub-entity recognition, physical state, colour attribution and counting, giving a fine-grained understanding of 3D geometry and its semantics. A comprehensive study of twelve recent systems reveals a pronounced modality gap: the image-conditioned Qwen-2.5-VL achieves 79% accuracy on binary probes and 74% on multiple-choice questions, while the best point-cloud model reaches only 55% and 33% respectively. Caption-quality assessments follow the same pattern, underscoring the gap that remains for genuine 3D understanding. We make GLUE3D publicly available, along with its evaluation scripts and baseline scores, to advance progress in geometry-aware multimodal language understanding.
DOI: 10.1016/j.inffus.2025.104007
Language: English
Open Access status: hybrid
Estimated APC paid: $4,740
OpenAlex ID: W4416808497
FWCI:
Cites: 3
Cited by:
Related to: