Low-Cost LoRA Fine-Tuning and Deployment-Efficiency Profiling of a 0.5B-Parameter Language Model for Closed-Vocabulary Structured Extraction on Consumer Apple Silicon
Year: 2026
Type: preprint
Abstract: We present a fully reproducible case study in fine-tuning and deploying a sub-billion-parameter language model for closed-vocabulary structured extraction on commodity hardware. Using low-rank adaptation (LoRA), we fine-tune Qwen2.5-0.5B-Instruct to extract a strict four-field JSON object (sentiment, a bounded topic set, a price-mention flag, and an implied 1–5 rating) from unstructured mobile-application reviews, and we evaluate it with a fully deterministic parse-and-compare protocol that uses no model-based judge. Training is performed entirely on an Apple M4 Mac mini using the Metal Performance Shaders (MPS) backend in 32-bit precision; because 4-bit bitsandbytes quantization is CUDA-only, this configuration deviates from canonical QLoRA and the associated quantization-recovery control cannot be produced, a limitation we state explicitly rather than omit. On a held-out test set of 200 examples, fine-tuning improves the unweighted aggregate score from 62.3% to 76.6% (+14.2 points), with the largest gain on the closed-vocabulary topic field (26.1% to 73.5%). We then profile the quantized deployment (4-bit weights, served via Ollama) using a measurement methodology adapted from the low-bit-inference literature: peak-memory decomposition, key–value (KV) cache-quantization tolerance, the memory-bandwidth throughput ceiling, and energy per token. The central deployment finding is a qualitative collapse under aggressive KV-cache quantization: an 8-bit KV cache is near-lossless (77.95% to 77.02%), whereas a 4-bit KV cache reduces aggregate accuracy to 0.38% and valid-JSON parse rate to 10.5%. We discuss the label-circularity threat that limits the interpretation of three of the four fields, and we release all code, data splits, model artifacts, and raw measurements
Author: Deshpande, Tanmay
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7170702522
Cites:
Cited by:
Related to: