Unpacking Random Forest-Based Detection of Differential Item Functioning (DIF): The Roles of DIF and Item Impact Effect Sizes and Factor Correlation
Year: 2026
Type: preprint
Abstract: Reliable methods for detecting differential item functioning (DIF) and distinguishing it from item impact are essential for designing fair and valid tests. The RF-XAI method is one approach for identifying DIF items based on random forests (RFs) and permutation importance (PI) scores. In this simulation study, we evaluated the capacity of RF-XAI to detect DIF under varying DIF and item impact effect sizes, and whether PI of non-DIF-non-impact items depends on their correlation with DIF or impact items, indicating a potential PI bias for correlated predictors. Further, we explored conditional permutation importance (CPI) scores’ eligibility as an alternative variable importance metric for DIF detection, hypothesizing greater robustness to predictor intercorrelation. Results indicated that RF‐XAI with PI scores reliably identified items exhibiting large DIF, but fell short when moderate DIF occurred. While DIF detection rates were not largely affected by item impact effects, false positive rates were elevated in impact conditions. Non‐DIF-non‐impact items correlated with DIF or impact items did not have substantially higher mean PI scores than those items uncorrelated with DIF or impact items. However, they exhibited an increase in false positive rates with the number of correlated true predictors, supporting that PI biases for correlated predictors may exist. Yet, in the current form, CPI emerged as an unsuitable alternative to PI for DIF detection due to CPI scores near zero with minimal variance. Overall, PI‐based RF‐XAI offers a promising approach for large‐effect DIF identification, while CPI's low variance suggests that the RF-XAI method requires a revised decision criterion when CPI is used.
Source: PsyArXiv (OSF Preprints)
Authors: Karoline Wolandt, Elisabeth Barbara Kraus
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7208772263