Work

TRIASE-SLO: Risk-Gated and SLO-Aware Routing Between FP16 and INT4 Language-Model Workers for Radiology Impression Serving

Year: 2026
Type: article
Abstract: Language models can draft radiology impressions, but a 4-bit copy of a model does not always reproduce the output of its full-precision version. TRIASE-SLO routes each request between FP16 and AWQ INT4 copies of one LoRA-adapted Qwen2.5-3B-Instruct model. A frozen rule-based hazard score computed from the FINDINGS text sends every request scoring at least 0.5 to FP16, and a secondary latency-aware score places the remaining requests. We evaluated the design in a locked confirmation in which every arm reserved the same two A100 MIG slices: four arms, seven loads and 12 paired deployment blocks of 450 previously inspected test reports (151,200 requests, none failed). The gate improved agreement with FP16. With full TRIASE-SLO routing, agreement with finding-bearing FP16 references was 99.11–99.49%, compared with 66.15–82.31% under hazard-blind routing, while 34–48% of requests still used INT4. A consistent serving-efficiency advantage was not demonstrated: full routing did not improve tail latency or throughput over two FP16 workers (median p95 change +0.33%; slower at C = 2 after Holm correction), and a consistent speed advantage of INT4 appeared only under hazard-blind routing. The demonstrated benefits of the INT4 copy are a smaller checkpoint and a larger KV-cache capacity. The incremental value of the score remains uncertain, because its prespecified C = 8 throughput benefit over gate-only routing (+1.86%, 95% CI −3.27% to +7.27%) was not confirmed. No INT4-using arm met the zero-drift engineering floor. All quality measures are rule-based FP16-consistency proxies from one public dataset, not evidence of clinical correctness or safety.
Language: English
Open Access status: diamond
OpenAlex ID: W7236071528
FWCI:
Cites: 39
Cited by:
Related to: 10