Work

Fine-Tuning Locally Deployable Scientific Language Models: Reliability and Resource Tradeoffs

Year: 2026
Type: preprint
Abstract: Commercial cloud-based language models increasingly support scientific research, but transmitting unpublished data raises concerns about privacy and institutional control. Small, locally deployable open-weight models offer an alternative, yet whether domain fine-tuning improves scientific reliability enough to justify its resource costs remains uncertain. Using electrochemistry as a test domain, we converted 4,983 open-access papers into 86,329 questionanswer pairs and fine-tuned eleven models (3-32 billion parameters) using low-rank adaptation. We compared adapted and original models on 8,481 held-out questions using reference-similarity metrics, three LLM judges and a blind five-expert pilot, and evaluated continued pretraining and retrieval-augmented generation (RAG) alongside financial and environmental costs. Fine-tuning increased ROUGE-1 reference-answer similarity for ten models (11.1-86.5%) but lowered judged answer quality in 18 of 19 configurations (mean change −0.621 on a five-point scale), consistent with expert preferences. Apparent gains depended on model, checkpoint, training seed, and answer length, while larger models or more synthetic data did not consistently help. Qualitative examples also exposed overconfident errors. Continued pretraining alone offered limited benefit; combining it with supervised fine-tuning improved scores over fine-tuning alone but not over unmodified models in judged quality. In a single-model, open-book pilot, RAG achieved higher judge scores at $6 versus $302 in development cost, with estimated energy, emissions, and water footprints of 2.8 kWh, 0.9 kg CO2e, and 6 L versus 132 kWh, 44 kg CO2e, and 271 L, despite 2.0-3.1-fold more computation per query. These results support retrieval with an unmodified local model as a practical baseline for privacy-sensitive research and highlight the need to justify finetuning through independently verified accuracy, robust benchmarking, and full resource costs rather than textual similarity alone.
Source: ChemRxiv
Language: English
Open Access status: green
OpenAlex ID: W7226018991
Cites: 23
Cited by:
Related to: 10