PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility
Year: 2026
Type: conference-paper
Abstract: Vision Language Models (VLMs) are increasingly integrated into privacy-critical domains, yet existing evaluations of personally identifiable information (PII) leakage largely treat privacy as a static extraction task and ignore how a subject's online presence-the volume of their data available online-influences privacy alignment.We introduce PII-VisBench, a novel benchmark containing 4,000 unique probes designed to evaluate VLM safety through the continuum of online presence.The benchmark stratifies 200 subjects into four visibility categories: high, medium, low, and zero-based on the extent and nature of their information available online.We evaluate 18 open-source VLMs (0.3B-32B) based on two key metrics: percentage of PII probing queries refused (Refusal Rate) and the fraction of non-refusal responses flagged for containing PII (Conditional PII Disclosure Rate).Across models, we observe a consistent pattern: refusals increase and PII disclosures decrease (9.10% high → 5.34% low) as subject visibility drops.We identify that models are more likely to disclose PII for high-visibility subjects, alongside substantial model-family heterogeneity and PII-type disparities.Finally, paraphrasing and jailbreak-style prompts expose attack-and model-dependent failures, motivating visibilityaware safety evaluation and training interventions.We have made our implementation publicly available.1 PII-VisBench is different in several ways: first, we work with images rather than text.Instead of asking "Tell me about [Name]", we show the model an image and ask for private details -the model must figure out who the person is from the image alone.This is a more realistic and more dangerous scenario, since the user never has to reveal who they are asking about.Second, we measure visibility differently -using web search hit counts and reverse image search results -capturing how findable someone is online as a visual identity, not just how many Wikipedia views they have.Third, we do not give the model any explicit privacy rules during testing.We want to see how models behave on their own, without being told what to protect.Fourth, we cover a broader range of sensitive data, including SSNs, passport numbers, bank details, phone numbers, and emails -not just general public-figure attributes.Finally, while PRIVQA finds that lesserknown people get weaker protection, we identify a complementary risk on the other end: highly visible people are also at greater risk, because models may reveal information about them from memory, even without any jailbreak or adversarial prompting.
Institution: University of California, Riverside
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166816506