Year: 2026
Type: conference-paper
Abstract: We annotate PubMed Central paragraphs for document type, domain, and educational quality using a two-stage pipeline: Llama-3.1-70Blabels 400K paragraphs, then a fine-tuned XLM-RoBERTa propagates annotations to the full corpus.This paragraph-level approach captures content diversity within scientific articles that document-level labels miss.The resulting Biomed-Enriched corpus contains 2M clinical case paragraphs, providing a publicly available alternative to restricted clinical datasets.For decoders, continual pretraining experiments enable targeted improvements, with clinical upsampling boosting performance by 4 points on MMLU ProfMed and educational filtering improving MedQA and MedMCQA by ∼1 point.Combinations of these techniques led to faster convergence, reaching the same performance with a third of training tokens.For encoders, our best recipe matches BioClinical-ModernBERT on 11 tasks (77.3% vs. 77.1% F1) while using 2.5× fewer tokens and only public data.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166873841