Multilingual semantic distance: Automatic verbal creativity assessment in many languages.
Year: 2023
Type: article
Abstract: Creativity research commonly involves recruiting human raters to judge the originality of responses to divergent thinking tasks, such as the alternate uses task (AUT). These manual scoring practices have benefited the field, but they also have limitations, including labor-intensiveness and subjectivity, which can adversely impact the reliability and validity of assessments. To address these challenges, researchers are increasingly employing automatic scoring approaches, such as distributional models of semantic distance. However, semantic distance has primarily been studied in English-speaking samples, with very little research in the many other languages of the world. In a multilab study (N= 6,522 participants), we aimed to validate semantic distance on the AUT in 12 languages: Arabic, Chinese, Dutch, English, Farsi, French, German, Hebrew, Italian, Polish, Russian, and Spanish. We gathered AUT responses and human creativity ratings (N= 107,672 responses), as well as criterion measures for validation (e.g., creative achievement).We compared two deep learning-based semantic models—multilingual bidirectional encoder representations from transformers and cross-lingual language model RoBERTa—to compute semantic distance and validate this automated metric with human ratings and criterion measures. We found that the top-performing model for each language correlated positively with human creativity ratings, with correlations ranging from medium to large across languages. Regarding criterion validity, semantic distance showed small-to-moderate effect sizes (comparable to human ratings) for openness, creative behavior/achievement, and creative self-concept. We provide open access to our multilingual dataset for future algorithmic development, along with Python code to compute semantic distance in 12 languages.
DOI: 10.1037/aca0000618
Authors: John D. Patterson, Hannah M. Merseal, Dan R. Johnson, Sergio Agnoli, Matthijs Baas, Brendan S. Baker, Baptiste Barbot, Mathias Benedek, Khatereh Borhani, Qunlin Chen, Julia F. Christensen, Giovanni Emanuele Corazza, Boris Forthmann, Maciej Karwowski, Nastaran Kazemian, Ariel Kreisberg-Nitzav, Yoed N. Kenett, Allison Link, Todd Lubart, Maxence Mercier, Kirill Miroshnik, Marcela Ovando-Tellez, Ricardo Primi, Rogelio Puente-Díaz, Sameh Said-Metwaly, Claire Stevenson, Meghedi Vartanian, Emannuelle Volle, Janet G. van Hell, Roger E. Beaty
Language: English
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: hybrid
Funders: National Science Foundation, Deutsche Forschungsgemeinschaft, Agence Nationale de la Recherche, Fundação de Amparo à Pesquisa do Estado de São Paulo, National Natural Science Foundation of China, Universiteit van Amsterdam, Austrian Science Fund, Jacobs Foundation, Max-Planck-Gesellschaft, Narodowym Centrum Nauki, Friedrich Naumann Stiftung
Awards: Normative and intentional development of teacher social-emotional skills and their relationships with student achievement and social-emotional skills, Collaborative Research: Measuring and Enhancing Scientific Creative Thinking for STEM Education and Research: Classroom-Aligned Assessment and Network Neuroscience-Based Mechanisms, G1386765427, SFB 1052: Mechanismen der Adipositas, Divergent Thinking in the Engineering Design Context: Experimentation to Connect Performance to Neurocognitive Responses, G195474297, G2012522975, G2605684771, G2729871496, G3010527862, Collaborative Research: Neurocognitive Studies to Enhance STEM Education: Divergent Thinking in Female and Male Engineering Students, G3487620501, G4253563803, G5562328029, Semantic and brain networks underlying creative abilities, Using Computational Modeling to Transform Assessments of Creativity in Engineering Design, G7139023546, Neural efficiency, intelligence and motivation
OpenAlex ID: W4386745708