Year: 2023
Type: preprint
Abstract: The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
Source: arXiv (Cornell University)
Authors: Lee, Chen-Yu, Li, Chun-Liang, Zhang, Hao, Dozat, Timothy, Perot, Vincent, Su, Guolong, Zhang, Xiang, Sohn, Kihyuk, Glushnev, Nikolai, Wang, Renshen, Ainslie, Joshua, Long, Shangbang, Qin, Siyang, Fujii, Yasuhisa, Hua, Nan, Pfister, Tomas
Institution: Google (United States)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4372272971