Speech-Based Cognitive Screening: A Systematic Evaluation of LLM Adaptation Strategies (Preprint)
Year: 2025
Type: preprint
Abstract: BACKGROUND Over half of U.S. adults with Alzheimer’s disease and related dementias (ADRD) remain undiagnosed. Speech-based screening algorithms offer a scalable approach, but the relative value of large language model (LLM) adaptation strategies is unclear. OBJECTIVE To compare LLM adaptation strategies for ADRD detection from the DementiaBank speech corpus using both text-only and multimodal models. METHODS We analyzed audio-recorded speech from 237 participants and report performance on a held-out test set (n=71). Nine text-only LLMs (3B–405B; open-weight and commercial) and three multimodal audio–text models were evaluated. Adaptations included: (i) in-context learning (ICL) with four demonstration selection policies (most-similar, least-similar, class-centroid/prototype, random); (ii) reasoning-augmented prompting (self-/teacher-generated rationales, self-consistency, Tree-of-Thought with domain experts); (iii) parameter-efficient fine-tuning (token-level vs. added classification head); and (iv) multimodal audio–text integration. The primary outcome was F1 for the cognitively impaired (CI) class; AUC-ROC was reported when available. RESULTS Class-centroid (prototype) demonstrations achieved the highest ICL performance across model sizes (F1 up to 0.81). Reasoning primarily benefited smaller models: teacher-generated rationales increased LLaMA-8B from F1 0.72 to 0.76; expert-role Tree-of-Thought improved its zero-shot score from 0.65 to 0.71. Token-level fine-tuning produced the highest scores (LLaMA 3B: F1=0.83, AUC=0.91; LLaMA 70B: F1=0.83, AUC=0.86; GPT-4o: F1=0.80, AUC=0.87). A classification head markedly improved MedAlpaca 7B (F1 0.06 to 0.82), indicating model-dependent benefits of this approach. Among multimodal models, fine-tuned Phi-4 Multimodal reached F1=0.80 (CI) and 0.75 (cognitively normal) but did not exceed the top text-only systems. CONCLUSIONS Detection accuracy is influenced by demonstration selection, reasoning design, and tuning method. Token-level fine-tuning is generally most effective, while a classification head benefits models that perform poorly under token-based supervision. Properly adapted open-weight models can match or exceed commercial LLMs, supporting their use in scalable speech-based ADRD screening. Current multimodal models may require improved audio–text alignment and/or larger training corpora.
Language: English
Subfield: Psychiatry and Mental health
Field: Medicine
Domain: Health Sciences
Open Access status: gold
OpenAlex ID: W7084093982