Making model judgments ROC(K)-solid: Tailored cutoffs for fit indices through simulation and ROC analysis in structural equation modeling
Year: 2022
Type: preprint
Abstract: To evaluate model fit in structural equation modeling, researchers compare fit indices against fixed cutoff values (e.g., CFI ≥ .950 indicates a well-fitting model). Although using fixed cutoffs is widespread, methodologists have long cautioned against overgeneralizing such cutoffs, highlighting that one should only apply fixed cutoffs in highly similar settings to the simulation scenarios from which those cutoffs originate. Values of fit indices vary depending on the model, estimation, and data characteristics of the empirical setting. Conclusions regarding whether a model does or does not fit the data may ultimately be biased. As a solution, methodologists have proposed four principal approaches to obtain so-called tailored (or “dynamic”) cutoffs that are explicitly generated for the setting of interest. We herein review these approaches. Notably, none of these approaches have yet provided guidelines on which fit index (out of all fit indices of interest) best evaluates whether the model fits the data (or not) in the specific setting. Thus, we proposed the so-called simulation-cum-ROC approach that combines a Monte Carlo simulation with receiver operating characteristic (ROC) analysis. The simulation-cum-ROC approach generates tailored cutoffs while identifying the most reliable fit indices in the given setting. We provide computational R code and a Shiny app for easy implementation of the approach. Thus, prior knowledge of Monte Carlo simulations and ROC analysis is not needed to generate tailored cutoffs with the simulation-cum-ROC approach.
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
Funder: Universität Mannheim
OpenAlex ID: W4283378975