Year: 2026
Type: conference-paper
Abstract: Membership inference attacks (MIAs) have emerged as critical tools for auditing privacy risks in large language models (LLMs), aiming to determine whether a given text was included in a model's training corpus.However, most existing MIAs require access to per-token logits or probabilities, making them inapplicable in practice to proprietary LLMs that expose only textual continuations.To address this underexplored setting, we propose Word-level Probability MIA (WPMIA), a statistically principled MIA for strict black-box privacy auditing.WP-MIA estimates word-level generation probabilities via Monte Carlo sampling with local kernel smoothing, then aggregates these estimates into a sequence-level likelihood estimator.Furthermore, WPMIA constructs the likelihood conditioned on different prefixes, thereby amplifying the distributional differences between members and non-members.We evaluate WPMIA across various open-source LLMs and find that it consistently outperforms existing black-box baselines.Importantly, we also evaluate WP-MIA on modern proprietary LLMs, including GPT-5-Chat, Gemini-2.5-Flash,and Claude-4.5-Haiku,achieving an average TPR@5%FPR of 42.0 across these models.These results offer a sound foundation for future research on strict black-box membership inference.Code is available at https://github.com/niusj03/WPMIA.
Authors: Shengjie Niu, Yeheng Ge, Jian Huang
Institution: Hong Kong Polytechnic University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
Funder: Hong Kong Polytechnic University
Award: G1258512577
OpenAlex ID: W7212287809