Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition
Year: 2026
Type: conference-paper
Abstract: NER requires token-level classification using both left and right context, which makes encoder-only models like BERT naturally wellsuited for the task.Decoder-only LLMs, by contrast, use causal masking during training, so their token representations lack right-side context, limiting their effectiveness on structured prediction tasks like NER despite their strong general capabilities.To address this, the authors propose fine-tuning decoder-only LLMs with causal attention replaced by full attention, combined with label-supervised discriminative training.While similar ideas exist in prior work, those studies were limited in scope.This work evaluates seven LLMs across four model families (Gemma, Qwen2.5, Llama3.1,Llama3.2) and compares full finetuning against LoRA.Results show that the proposed approach with an appropriate LoRA configuration outperforms encoder baselines (BERT, RoBERTa, DeBERTa), and achieves strong NER performance without auxiliary data or architectural modifications, though it does not reach SOTA on BC5CDR and CoNLL2003.
Authors: Ken Yano, Hiroya Takamura
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7166879499