Generating Facial Expressions to Understand Internal Representations of Facial Expressions in Large Language Models
Year: 2026
Type: preprint
Study design: Study Protocol
Abstract: Research Questions: The primary purposes of this proposed research are as follows: 1. How do large language models’ (LLMs) internal representations of facial expressions compare to those of humans? We will use Facial Expression Recognition (FER) and facial expression generation as a benchmark task to explore how humans and LLMs compare when asked to generate human facial expressions based on text-based natural language interpretations generated by LLMs. 2. Do large language models exhibit demographic bias in facial expression generation, and how do these biases manifest? We will investigate what demographics LLMs utilise to represent different emotions. Specifically, we aim to determine whether the models are biased toward generating certain demographic characteristics for specific emotions (for instance, consistently representing sadness as female or surprise as Asian). Identifying these patterns will allow us to measure the level of bias in these models so that future mitigation strategies can be developed. Design: To study internal representations of facial expressions in LLMs, we will be using images from the AffectNet-8 dataset (Mollahosseini et al., 2017), which have demographics and facial expressions annotated. The demographics include gender (Female and Male), race (Asian, Black, Indian, and White), which we will call ethnicity, and age (00-02, 03-09, 10-19, 20-29, 30-39, 40-49, 50-59, 60-69, 70+), while the emotions include: Anger, Contempt, Disgust, Fear, Happiness, Neutral, Sadness, and Surprise. Age will be grouped into under 20, 20-59, and 60+, as there are fewer representations of ages at both extremes in AffectNet-8, particularly 70+. We will randomly select one image for each emotion by each demographic (e.g. one happiness image from a female Indian face aged 20-59). We will then ask 4 different LLMs to classify the emotion out of the possible 7 (excluding neutral, as AUs should all be inactive) and explain their decision. The output of the models will be used as the main stimuli in the survey, where we will be involving human participants in one branch, and the same LLMs in another branch. We will ask the LLMs, with cleared memory, to generate human faces based on the texts they have generated, while we will ask the human participants to generate faces using the software Gaussian Avatars (Qian et al., 2024). Finally, the Action Units (AU) will be extracted from all generated images to assess the similarity between human and LLM generations. Because there are only 14 different avatars readily available to use in Gaussian Avatars, we have decided to only include AffectNet-8 representations that can be matched with one of the avatars in terms of demographics. However, there are unfortunately no Asian avatars available beyond Indian, which the AffectNet-8 annotations group as a separate category, as well as avatars under the age of 20. We are left with 7 distinct demographic groups, each with one example of each of the 7 facial expressions (no neutral), which results in a sample of 49 images. We will be using 4 different LLMs to generate stimuli for each image, and we will therefore have a sample of 196 stimuli for participants to recreate. We have selected models based on four criteria: • They have multimodal capabilities, which are necessary for them to receive images as input and to generate images. • They rank among the top models on the Vision Arena leaderboard (Arena Intelligence, 2026), which specifically ranks multimodal LLMs. • Their APIs are available on the website OpenRouter (OpenRouter, Inc, 2026). This is a practical requirement for us to be able to generate the stimuli and the final images. • We will only select one high-ranking model from any single lab; therefore, no different releases of the same models. The models that have been selected are: • Claude Opus 4.7 Thinking (Anthropic, 2026) • Gemini 3 Pro (Google LLC, 2026) • GPT 5.5 (OpenAI, 2026) • Qwen 3.7 Plus (Alibaba, 2026) Procedure: Participants will be invited to an online study through Prolific (https://www.prolific.com), the study itself will be hosted on Qualtrics (https://uofg.qualtrics.com/), and the Gaussian Avatars software will run online on Hugging Face Spaces (https://huggingface.co/spaces). In the online study, participants will first be presented with an information sheet and a consent form. They will then be asked about their age, gender, and ethnicity. We will ask about ethnicity because we wish to stratify the sample to include an even number of the same demographics represented in the study. Then they will be presented with the Gaussian Avatars software (Qian et al., 2024), where they will be asked to recreate photo-realistic images of faces based on the stimuli generated by the LLMs, see Figure 1. First, they will be shown a random training image, where they will be asked to try all sliders to familiarise themselves with how the software works. When they are ready, they will be asked to move on to the main section of the experiment. In the main section, they will receive a random instruction from the stimuli generated by the LLMs, combined with a random photo-realistic avatar (14 possible avatars) whose demographics match that of the original AffectNet-8 image. After they have completed 10 facial expressions, they will be informed that they are finished and may press the "next page" button. Here, they will be presented with the debrief form, followed by a button or link to bring them back to Prolific. Participants: We will have 195 different stimuli and want them all to be used by at least 3 participants. To minimise task fatigue but still maintain a variety of stimuli for each participant to use, each participant will be asked to generate 10 images based on 10 LLM-generated stimuli. If each participant sees 20 stimuli, and we want all stimuli to be seen by 3 participants, it means we will need a sample of at least 60 participants. We wish to have all categories represented in the stimuli be represented in the participant sample to avoid demographic bias from influencing results. Therefore, the sample will be stratified to include approximately equal numbers of White, Black, and Indian participants, male and female, adults between 18 and 19 years old, between 20 and 59, and over 60. Finally, all participants will need to be fluent in English and be over the age of 18. Analysis plan: After all data has been collected from participants and we have generated the faces using the LLMs, we will use software such as OpenFace (Baltrusaitis et al., 2015) to extract AUs from all images. For every text stimulus, we will average the extracted AUs generated by human participants as well as the averages of AUs extracted from the LLM images. We propose using Representational Similarity Analysis (RSA). For each Source (Human, LLM, Original), construct a dissimilarity matrix (RDM) across the 49 identity–emotion cells using pairwise distances (e.g., 1 − cosine similarity) between AU profiles. We also intend to fit a PERMANOVA on raw AU vectors (Source × Emotion × Demographic Group). This will then show whether the generation was similar between them, and if so, conclude whether the LLMs’ internal representation of facial expressions based on text information is consistent with that of humans. We will also compare the AUs of all generated faces with ground truth AUs from the AffectNet-8 dataset, which will show the loss of data when the LLMs attempt to explain what they see. This, in turn, can highlight which elements are ignored and which are focused on. This can then be compared to facial expression theories (Li & Mahmoud, 2024; Lucey et al., 2010), which describe which AUs are active during different expressions, and whether the same AUs are present in the LLMs’ decision-making. Finally, we will also be exploring the demographic representation within each emotion to understand if LLMs attribute certain emotions to certain demographics, for instance, if all sad expressions were female, or all happy expressions were black, etc. This can inform us of biases and fairness issues within the models. Overview of Field: Facial Expression Recognition: Facial Expression Recognition (FER) has promising implications for technologies that could predict people’s mental health trajectories (Zhang et al., 2020). Researchers are predicting mental health disorders such as depression, bipolar disorder, and anxiety using multi-modal models (Ringeval et al., 2019; Wang et al., 2020; Zhang et al., 2020). However, FER models are prone to bias, as the training data is often lacking in diversity, and therefore the models do not recognise expressions equally well in all demographic groups (Jannat & Canavan, 2021; Labzour et al., 2023). This is particularly pronounced in groups that are often not represented enough in the training data, such as older adults (Guodong Guo et al., 2013; Jannat & Canavan, 2021; Labzour et al., 2023; Lopes et al., 2018). Beyond improving the data used for training, improving the transparency of FER models is another option for researchers to control for biases and, in turn, improve the fairness of the models. Interpretability and AI Fairness: Improving fairness and transparency in FER research involves providing users with interpretations of how models make their decisions (Jung et al., 2023; Miller, 2019; Nauta et al., 2023). The terms Interpretations and explanations are often used interchangeably in the literature (Joyce et al., 2023); however, I am using the term interpretation to refer to inherently interpretable models, as opposed to post-hoc methods of explanation such as Shapley Values (Lipovetsky & Conklin, 2001) or LIME (Ribeiro et al., 2016). Interpretations serve multiple purposes; one example is the human-in-the-loop approach (Mosqueira-Rey et al., 2023), where interpretations enable users to correct models when they utilise incorrect features in their decision-making processes. Furthe
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7202171616