Typeset Replacement of Handwritten Text and Mathematics on Lecture Slides Using Vision-Language Models
Year: 2026
Type: article
Abstract: University lecturers often annotate printed slides during class with handwritten notes and mathematical derivations. Generic OCR skips this ink or returns unrelated strings, and no public dataset fits the task: German handwriting corpora hold prose or historical script, handwritten-mathematics corpora are language-neutral stroke data, and none captures a single lecturer’s annotations where German text and free-form mathematics share a slide. We ask whether a general vision-language model (VLM) can outperform purpose-built OCR and handwritten-mathematics recognisers in this data-scarce setting, and drive a pipeline that typesets the recognised annotations back onto the slide. A YOLOv8x detector locates handwritten regions and labels them as text or math. Qwen3-VL-8B transcribes each class through a tailored prompt, and a LaTeX+dvipng stage renders the output into the original ink footprint. We hand-annotated 165 slides from one professor, evaluated every stage on held-out crops, and ran the pipeline over a 579-slide corpus. On handwritten mathematics the VLM reaches roughly an 18% character error rate, less than half the next-best system (PaddleOCR-VL) and far below SmolDocling, TrOCR, Pix2Tex and TAMER, which all exceed 80% raw CER; on German text it reaches under 6% with no fine-tuning. The pipeline renders 95% of detected math regions, and a 100-region audit judged 83% of rendered replacements fully correct. Two further findings: a warm-started detector beats a newer architecture trained from scratch, and meta-learning on out-of-domain handwriting fails to transfer, where a small adapter is cheaper. Code, configurations, evaluation scripts and annotated evaluation sets are released at github.com/ragav1n/ink2digital
Source: IEEE Access
Authors: N. Ragavenderan, Judith Jakob, S. Manonmani
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,075
OpenAlex ID: W7206200869