Better Where It Answered, Absent Where It Did Not: A Pre-registered Comparison of a Tesseract and Rules Pipeline with a Gemini Vision Route on the 50-Document FUNSD Test Split
Year: 2026
Type: preprint
Abstract: Every document-processing team faces the same fork: build on classical OCR plus rules it controls, or send pages to a vision model and take the extraction it gives back. The claims made for both routes are usually benchmark-free. This study measured the fork under a pre-registration locked before any run existed. On the 50-document test split of FUNSD (Jaume, Ekenel and Thiran, 2019), 816 human-annotated field pairs, two arms were scored by one shared scorer under one frozen normalization and matching rule. Arm A is classical: Tesseract 5.3.4 OCR followed by three deterministic extraction rules, no learned model. Arm B is a vision route: Gemini 3 Flash Preview, one call per document with a frozen prompt at temperature 0, reached through an existing connected route, with one retry on error and failures counted, never dropped. All three hypotheses were decidable. H1 CONFIRMED: the vision arm's field F1 is 0.4029 against the classical arm's 0.0645, a difference of 0.3385 against a pre-registered line of 0.15. H2 CONFIRMED: the classical arm's OCR word error rate is 0.6942 (6,060 edits over 8,729 reference tokens), against a pre-registered line of 0.10. H3 INCONCLUSIVE: the classical arm is faster at the median (3.58 s against 7.46 s per document, a ratio of 2.08), but not by the registered factor of 5, and not by zero either. The number that conditions everything else: the vision route answered only 21 of the 50 documents. The other 29 failed under the frozen failure rule and scored zero predicted pairs: 27 were HTTP 429 rate-limit rejections from the access route and 2 were call timeouts. On the 21 documents the route did answer, its descriptive F1 is 0.7687, against the classical arm's 0.0628 on the same documents. On accuracy the vision route wins by a wide margin wherever it answers at all. On reliability the classical pipeline is the predictable one: it processed all 50 documents on local hardware, while the vision route, submitted in a burst through the route as it exists, was refused three times in five. Both numbers are the finding, and neither is quoted here without the other.
Author: Ushno, Jesun Ahmad
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236140768
