Year: 2026
Type: preprint
Abstract: The global use of artificial intelligence has increased interest in assessing the ability to generate culturally localized content, including stories. Cultural localization in stories often occurs through either templated localization -- the use of cultural markers (e.g., names, locations) in a generic narrative -- or holistic localization -- the variation of plots, values, and themes, in addition to cultural markers. We propose a method to measure the degree to which content was generated through templated localization. Specifically, we identify the lexical tokens that distinguish stories across nationalities and measure the similarity of the narratives that remain after removing them. In stories generated by five models on 125 topics for 193 nationalities, our method is able to detect that only a small subset (9-17%) of the vocabulary accounts for the variation across nationalities and that the narratives that remain after removing them contain repeated multi-word sequences, suggesting the presence of a shared culturally-agnostic narrative template. Finally, we characterize the cultural markers for their stereotypicality and offensiveness, finding that markers from 19 countries, mostly located in the Global South, are on average offensive.
Source: arXiv (Cornell University)
Institution: Carnegie Mellon University
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Track 4: Advanced CI Coordination Ecosystem: Monitoring and Measurement Services, TITLE: CMU/NIST AI MEASUREMENT SCIENCE & ENGINEERING COOPERATIVE RESEARCH CENTERPURPOSE: ESTABLISH THE CMU/NIST AI MEASUREMENT SCIENCE & ENGINEERING COOPERATIVE RESEARCH CENTER (AIMSEC) TO FACILITATE COLLABORATIVE RESEARCH AND EXPERIMENTATION FOCUSED ON ADVANCING OUR NATIONAL CAPABILITY FOR TEST AND EVALUATION OF MODERN AI CAPABILITIES AND TOOLS. THERE ARE THREE PRINCIPAL GOALS: 1. ADVANCE MODERN AI RISK MANAGEMENT 2. VALIDATE EVALUATION APPROACHES THROUGH STAKEHOLDER PARTNERSHIPS. 3. TRANSLATE ASSESSMENT CAPABILITIES AND METHODOLOGIES TO PRACTICE. RESEARCH WILL FOCUS ON ADVANCING MEASUREMENT SCIENCE FOR MODERN AI SYSTEMS, INCLUDING MACHINE LEARNING (ML) AND GENERATIVE AI (GENAI) SYSTEMS, SUCH AS LARGE LANGUAGE MODELS (LLMS). ACTIVITIES TO BE PERFORMED: CREATE A PRACTICAL TAXONOMY OF THREATS AGAINST LARGE LANGUAGE MODELS (LLMS). THE PROPOSED WORK INCLUDES DEVELOPING TECHNIQUES TO REDUCE MEMORIZATION AND PROTECT CONFIDENTIAL INFORMATION PROVABLY, EVEN IF IT RESULTS IN LESS EXACT OUTPUTS. DEVELOP NEW MEASURES OF ROBUSTNESS FOR ML MODELS THAT BETTER REFLECT A MODEL?S ABILITY TO WITHSTAND ATTACKS IN PRACTICAL SCENARIOS. DEVELOP RICHER DEFINITIONS OF ALIGNMENT, CORRECTNESS, AND INTENT TO DETERMINE THE SUITABILITY OF GENERATIVE AI SYSTEMS FOR THEIR INTENDED USES. CREATE A STATE-OF-THE-ART SUITE OF ATTACKS AGAINST LLMS FOR TESTING NEW MODELS. THESE ACTIVITIES WILL BE ORGANIZED IN THREE STREAMS OF EFFORT ALIGNED WITH NIST AI PRIORITIES. - VALIDITY, RELIABILITY, SAFETY, PRIVACY, AND SECURITY- ACCOUNTABILITY, TRANSPARENCY, FAIRNESS, AND EXPLAINABILITY- GENERATIVE AI EVALUATIONEXPECTED OUTCOMES: DEVELOPMENT OF AI SYSTEM-LEVEL TOOLING, METRICS, EVALUATION PROCEDURES, DEVELOPMENT PROCESSES, AND BEST PRACTICES TO ENSURE AI SYSTEM BUILDERS CONSISTENTLY ENGINEER SAFE AI SYSTEMS. GOALS ARE TO ADVANCE MODERN AI RISK MANAGEMENT, VALIDATE EVALUATION APPROACHES THROUGH STAKEHOLDER PARTNERSHIPS, AND TRANSLATE ASSESSMENT CAPABILITIES AND METHODOLOGIES TO PRACTICE. RESEARCH WILL FOCUS ON ADVANCING MEASUREMENT SCIENCE FOR MODERN AI SYSTEMS, INCLUDING MACHINE LEARNING (ML) AND GENERATIVE AI (GENAI) SYSTEMS, SUCH AS LARGE LANGUAGE MODELS (LLMS). THESE EFFORTS ARE EXPECTED TO YIELD ADDITIONAL APPROACHES TO MITIGATION OF WEAKNESSES AND VULNERABILITIES, PRIVACY PROTECTION, A PRACTICAL TAXONOMY OF THREATS AGAINST LLMS, APPROACHES TO MEET AI GOVERNANCE CHALLENGES, AND NEW SOCIO-TECHNICAL DEFINITIONS AND METRICS.INTENDED BENEFICIARIES: AI STAKEHOLDERS, INCLUDING GOVERNMENT, INDUSTRY, ACADEMIA, AND THE GENERAL PUBLIC, WILL BENEFIT FROM THIS WORK OF ADVANCING CAPABILITY FOR TESTING AND EVALUATING MODERN AI CAPABILITIES AND TOOLS.SUBRECIPIENT ACTIVITIES: THERE ARE NO PLANNED SUBAWARDS., Track 1: ACCESS Resource Allocations Marketplace and Platform Services (RAMPS), ACO: An Open CI Ecosystem to Advance Scientific Discovery (OpenCI), Track 3: COre National Ecosystem for CyberinfrasTructure (CONECT), Track 2: Customized Multi-tier Assistance, Training, and Computational Help (MATCH) for End User ACCESS to CI
OpenAlex ID: W7164852007
Cites:
Cited by:
Related to: