Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) and Vision-Language Models (VLMs) have made significant advancements in a wide range of natural language processing and vision-language tasks. Access to large web-scale datasets has been a key factor in their success. However, concerns have been raised about the unauthorized use of copyrighted materials and potential copyright infringement. Existing methods, such as sample-level Membership Inference Attacks (MIA) and distribution-based dataset, inference distinguish member and non-member data by leveraging the common observation that models tend to memorize and show greater confidence in member data. Nevertheless, these methods face challenges when applied to LLMs and VLMs, such as the requirement for ground-truth member data or non-member data that shares the same distribution as the test data. In this paper, we propose a novel dataset-level membership inference method based on Self-Comparison. We find that a member prefix followed by a non-member suffix (paraphrased from a member suffix) can further trigger the model's memorization on training data. Instead of directly comparing member and non-member data, we introduce paraphrasing to the second half of the sequence and evaluate how the likelihood changes before and after paraphrasing. Unlike prior approaches, our method does not require access to ground-truth member data or non-member data in identical distribution, making it more practical. Extensive experiments demonstrate that our proposed method outperforms traditional MIA and dataset inference techniques across various datasets and models, including GPT-4o.
Institutions: Michigan State University, Sony Corporation (United States)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: Interactive, Individualized Professional Learning for Elementary School Teachers: Enhancing Content and Pedagogical Content Knowledge as a Basis for Improving Practice, G125128811, G1432373144, NeTS: Small: Exploring the Non-Standardized Polices, Operations, and Requirements for 5G Cellular Networks and Beyond: Advancing the Modeling, Tools, and Evaluation, III:Medium:Computation and Communication Efficient Distributed Learning, Collaborative Research: SaTC: CORE: Medium: Safeguarding Next-Generation Emergency Services (NG-9-1-1) over Cellular Networks: From Design to Practice, Intelligent, Adaptive Program with Just-in-time Feedback for Preservice Teachers, Collaborative Research: III: Medium: Graph Neural Networks for Heterophilous Data: Advancing the Theory, Models, and Applications, TRTech-PGR: Maize Hub: An Open-source Simulation Platform & Resources Integrating Extensive Field Trial Data, DNA Sequences, and Historical Weather Records, G7452299184, TRTech-PGR: Connecting sequences to functions within and between species through computational modeling and experimental studies
OpenAlex ID: W4409657193