Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction
Year: 2026
Type: conference-paper
Abstract: Recently developed pre-trained text-and-layout models (PTLMs) have shown remarkable success in multiple information extraction tasks on visually-rich documents (VrDs).However, despite achieving extremely high performance on benchmarks, their real-world performance falls short of expectations.Owing to this issue, we investigate the prevailing evaluation pipeline to reveal that: (1) The inadequate annotations within benchmark datasets introduce spurious correlations between task inputs and labels, which would lead to overly-optimistic estimation of model performance.(2) The evaluation solely relies on the performance on benchmarks and is insufficient to comprehensively explore the capabilities of methods in real-world scenarios.These problems impede the prevailing evaluation pipeline from reflecting the realworld performance of methods, misleading the design choices of method optimization.In this work, we introduce EC-FUNSD, an entity-centric dataset crafted for benchmarking information extraction from visually-rich documents (VrD-IE).This dataset disentangles the falsely-coupled segment and entity annotations that arises from the block-level annotation of FUNSD.Using the proposed dataset, we evaluate the real-world VrD-IE capabilities of PTLMs from multiple aspects, including their absolute performance, as well as generalization, robustness and fairness.The results indicate that prevalent PTLMs do not perform as well as anticipated in real-world VrD-IE scenarios.We hope that our study can inspire reflection on the directions of PTLM development.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140073839