Year: 2025
Type: conference-paper
Abstract: Understanding how facial affect analysis (FAA) systems perform across different demographic groups requires reliable measurement of sensitive attributes such as ancestry, often approximated by skin tone, which itself is highly influenced by lighting conditions. This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness $\left(L^{*}\right)$ and Hue $\left(H^{*}\right)$. Using AffectNet and a MobileNetbased model, we assess fairness across skin tone groups defined by each method. Results reveal a severe underrepresentation of dark skin tones ($\sim \mathbf{2} \%$), alongside fairness disparities in F1-score (up to 0.08) and TPR (up to 0.11) across groups. While ITA shows limitations due to its sensitivity to lighting, the $H^{*}-L^{*}$ method yields more consistent subgrouping and enables clearer diagnostics through metrics such as Equal Opportunity. Grad-CAM analysis further highlights differences in model attention patterns by skin tone, suggesting variation in feature encoding. To support future mitigation efforts, we also propose a modular fairness-aware pipeline that integrates perceptual skin tone estimation, model interpretability, and fairness evaluation. These findings emphasize the relevance of skin tone measurement choices in fairness assessment and suggest that ITA-based evaluations may overlook disparities affecting darker-skinned individuals.
Authors: Ana M. Cabanas, Alma Pedro, Domingo Mery
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
SDG: Reduced inequalities
Open Access status: green
Awards: FORTALECIMIENTO DE LA INVESTIGACION INTERDISCIPLINARIA EN FISICA MEDICA Y DEL FUTURO DOCTORADO EN CIENCIAS MENCION FISICA Y DEL MAGISTER EN FISICA MEDICA EN DIAGNOSTICO POR IMAGEN DE LA UNIVERSIDAD DE TARAPACA, ENHANCING EQUITY IN COMPUTER VISION AND DEEP LEARNING: INTEGRATING SKIN COLOR MEASUREMENTS FOR FAIR AND DIVERSE DATASETS, MILLENNIUM INSTITUTE FOR INTELLIGENT HEALTHCARE ENGINEERING, NATIONAL CENTER FOR ARTIFICIAL INTELLIGENCE RESEARCH (NAIR)
OpenAlex ID: W4413018253