Work

Understanding User Intent in Code-Mixed Sexual and Reproductive Health Queries in Urban India: Hierarchical Classification Approach Using Large Language Models

Year: 2026

Type: article

Abstract: BACKGROUND: Sexual and reproductive health (SRH) remains a stigmatized and taboo topic in many parts of the world, limiting access to reliable information. These challenges are heightened in the Global South, where linguistic and cultural diversity further complicate information access. In India (our study context), many individuals frequently express SRH concerns in code-mixed language, such as Hinglish (code-mixed Hindi and English), and use colloquial terms. Large language models (LLMs) could help answer SRH questions, but most systems are trained for English and may perform poorly on code-mixed text and cultural context. Our research aims to address this gap by assessing the current state of LLMs in understanding user intent in SRH queries for a low-resource language. OBJECTIVE: We evaluate the effectiveness of proprietary, multilingual open-weight, and Indic LLMs in zero-shot settings for identifying user intent in code-mixed Hinglish SRH queries. Our goal is to assess how well LLMs assign the correct label in a two-level hierarchical classification (topic and subtopic). We take a hierarchical approach because SRH queries are complex and context-dependent; flat labels could obscure clinically important distinctions and can lead to misdirected guidance. We also characterize common error types that drive misclassification. METHODS: We analyzed 4,161 de-identified questions about SRH in Hinglish, collected by our partner nonprofit health organization (Myna Mahila Foundation) in an underserved community in urban Mumbai. Queries were annotated into 8 topics and 40 subtopics using a hierarchical framework that captured linguistic, cultural, and contextual variation. We evaluated proprietary, multilingual open-weight, and Indic-specific LLMs in zero-shot settings. Performance was measured using hierarchical F1 (hF1), Exact Match, and topic- and subtopic-level accuracy. RESULTS: Proprietary models achieved the strongest results, with GPT-5 performing best overall (hF1= 0.784). Among open-weight systems, Sarvam-M emerged as the top-performing Indic model (hF1 = 0.757), ranking just below the top-performing proprietary model and performing comparably with Claude-3.5-Sonnet (0.745) as well as large multilingual systems such as Llama-3.3-70B-Instruct (0.742) and Gemma-3-27B-IT (0.739). Other Indic models performed considerably lower (e.g., Llama-3-Gaja-Hindi-8B, 0.596; Krutrim-2-Instruct, 0.558; Airavata, 0.404). Smaller multilingual open-weights models, including Mixtral-8x7B-Instruct (0.593), Llama-3.1-8B-Instruct (0.630), Gemma-2-9B-IT (0.657), consistently outperformed them, showing that parameter size alone does not explain performance gaps. While models generally captured broad topical intent, they frequently failed at fine-grained intent recognition, especially with euphemisms, colloquial expressions, and local and culturally situated questions. CONCLUSIONS: Hierarchical classification revealed persistent gaps in how LLMs handle code-mixed queries. Proprietary models performed best, but Sarvam-M shows that open-weight Indic systems can achieve performance close to state-of-the-art models when supported by robust training data, cultural adaptation, and appropriate model scale. These findings highlight the potential of localized, culturally-aligned models to advance linguistically inclusive AI tools and expand equitable access to SRH information in underserved populations globally.

Source: Journal of Medical Internet Research

DOI: 10.2196/86545