Work

BioELECTRA:Pretrained Biomedical text Encoder using Discriminators

Year: 2021
Type: conference-paper
Abstract: Recent advancements in pretraining strategies in NLP have shown a significant improvement in the performance of models on various text mining tasks.In this paper, we introduce Bio-ELECTRA, a biomedical domain-specific language encoder model that adapts ELECTRA (Clark et al., 2020) for the Biomedical domain.BioELECTRA outperforms the previous models and achieves state of the art (SOTA) on all the 13 datasets in BLURB benchmark and on all the 4 Clinical datasets from BLUE Benchmark across 7 NLP tasks.BioELEC-TRA pretrained on PubMed and PMC full text articles performs very well on Clinical datasets as well.BioELECTRA achieves new SOTA 86.34%(1.39%accuracy improvement) on MedNLI and 64% (2.98% accuracy improvement) on PubMedQA dataset.
Language: English
Open Access status: hybrid
OpenAlex ID: W3166508187
FWCI: 9.16
Cites: 46
Cited by: 183
Related to: 10