The Answer is “No”: A Comment on Peugh and Feldon (2020)

To the editor:

In Peugh and Feldon's (2020) publication, “‘How Well Does Your Structural Equation Model Fit Your Data?’: Is Marcoulides and Yuan's Equivalence Test the Answer?” the authors (we) reviewed longstanding problems with common strategies for assessment model fit using chi-square, comparative fit index (CFI), and root mean square error of approximation (RMSEA), then introduced and demonstrated a prospective alternative solution: equivalence testing as developed by Marcoulides and Yuan (2017). We were cautiously optimistic as to whether equivalence testing (EQT) was a better technique for assessing model fit versus previous strategies but cautioned that further empirical testing was needed. Based on research published since (McNeish, 2023; Peugh et al., 2023; McNeish and Wolf, 2024), our response to the question “Is Marcoulides and Yuan's equivalence test the answer?”, is “No.” While the research field would still benefit from a more reliable test of model fit, robust evidence suggests that the equivalence testing approach described by Peugh and Feldon does not meet that need.

Decades of extensive research since the publication of Hu and Bentler's (1995, 1998, 1999) landmark research has shown fit index cut-off points (e.g., CFI ≥ 0.96, RMSEA < 0.08) have failed both to supplant chi-square as an inferential model fit test and to be reliable measures of model fit. For example, Hu and Bentler's fit index cut-off points will show acceptable fit for structural equation models based on analysis variables with unacceptable internal consistency reliability, as well as models that poorly reproduce analysis variable relationships known to exist in the sample data (Hancock and Mueller, 2011; McNeish and Wolf, 2024). However, despite these dismal empirical realities, fit index cut-off points are universally alive and well in the minds of research editors, reviewers, and authors because nothing else more quickly answers a model fit question and little (if anything) else is available.

The goal of our previous publication (Peugh and Feldon, 2020) was to introduce and demonstrate EQT (Marcoulides and Yuan, 2017), which showed initial promise as a replacement for the poor existing mechanisms to assess model fit quality (Marcoulides and Yuan, 2020, 2023). However, subsequent Monte Carlo simulation testing showed EQT fit statistics (CFI_t and RMSEA_t) produce excessive Type-I (rejecting well-fitting models) and Type-II (accepting poorly fitting models) errors and failed to outperform the chi-square test of model fit in scenarios commonly occurring in applied research (e.g., N ≤ 300, missing data, and skewed data; McNeish, 2023; Peugh et al., 2023; McNeish and Wolf, 2024). Accordingly, we felt it necessary to provide the most complete information available for researchers eager to evaluate the quality of statistical analysis models.

Researchers wondering, “Well, now what?” should be aware that a new fit assessment technique, direct-discrepancy dynamic fit indices (3DFI; McNeish 2023; McNeish and Wolf, 2024) seems to have avoided the pitfalls of both traditional fit indices and the EQT approach. Briefly, McNeish and colleagues operationalized Millsap's (2007, 2013) conjecture that Hu and Bentler's Monte Carlo simulation procedures could accurately and reliably determine the fit of a single data analysis model. Further, rather than absolute cut-off points, the authors followed Marcoulides and Yuan's (2017) example and created interpretational categories or “bins” within which 3DFI simulation results could be placed and from which a model fit determination could be made.

3DFI is currently available either as a Shiny application (https://dynamicfit.app; Wolf and McNeish, 2021), or as an R package (“dynamic,” Wolf and McNeish, 2023), but it is not available for multiple-group or multilevel/mixed linear model analyses. Preliminary Monte Carlo testing shows 3DFI has potential as an inferential model fit test. However, how 3DFI performs in simulations across a wide range of SEM types under various sample sizes, data distributions, number of analysis variables, and missing data conditions—as we used when testing (unsuccessfully) the utility of EQT (Peugh et al., 2023)—is currently an open question. As such, we must repeat the cautionary warning we offered previously for EQT: model fit assessment methods should not be adopted until rigorous empirical Monte Carlo testing establishes both the reliability and validity of the procedure.

REFERENCES

• Hancock, G. R., & Mueller, R. O. (2011). The reliability paradox in assessing structural relations within covariance structure models. Educational & Psychological Measurement, 71(2), 306–324. https://doi.org/10.1177/0013164410384856 Google Scholar
• Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16(3), 319–336. https://doi.org/10.1037/a0024917 Medline, Google Scholar
• Hu, L. T., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural Equation Modeling: Concepts, Issues, and Applications (pp. 76–99). Thousand Oaks, CA: Sage Publications. Google Scholar
• Hu, L. T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453. https://doi.org/10.1037/1082-989X.3.4.424 Google Scholar
• Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118 Google Scholar
• Marcoulides, K. M., & Yuan, K. H. (2017). New ways to evaluate goodness of fit: A note on using equivalence testing to assess structural equation models. Structural Equation Modeling, 24(1), 148–153. https://doi.org/10.1080/10705511.2016.1225260 Google Scholar
• Marcoulides, K. M., & Yuan, K. H. (2020). Using equivalence testing to evaluate goodness of fit in multilevel structural equation models. International Journal of Research & Method in Education, 43(4), 431–443. https://doi.org/10.1080/1743727X.2020.1795113 Google Scholar
• Marcoulides, K. M., & Yuan, K. H. (2023). Testing structural equation model fit in psychological studies: A replication study using equivalence testing. Quality & Quantity, 58, 3417–3433. https://doi.org/10.1007/s11135-023-01796-4 Google Scholar
• McNeish, D. (2023). Generalizability of dynamic fit index, equivalence testing, and Hu & Bentler cutoffs for evaluating fit in factor analysis. Multivariate Behavioral Research, 58(1), 195–219. https://doi.org/10.1080/00273171.2022.2163477 Medline, Google Scholar
• McNeish, D., & Wolf, M. G. (2024). Direct discrepancy dynamic fit index cutoffs for arbitrary covariance structure models. Structural Equation Modeling: A Multidisciplinary Journal, 1–28. https://doi.org/10.1080/10705511.2024.2308005 Google Scholar
• Millsap, R. E. (2007). Structural equation modeling made difficult. Personality and Individual Differences, 42(5), 875–881. https://doi.org/10.1016/j.paid.2006.09.021 Google Scholar
• Millsap, R. E. (2013). A simulation paradigm for evaluating model fit. In M. EdwardsR. MacCallum (Eds.), Current Issues in the Theory and Application of Latent Variable Models (pp. 165–182). New York: Routledge. Google Scholar
• Peugh, J., & Feldon, D. F. (2020). “How well does your structural equation model fit your data?”: Is Marcoulides and Yuan's equivalence test the answer? CBE—Life Sciences Education, 19(3), es5. https://doi.org/10.1187/cbe.20-01-0016 Medline, Google Scholar
• Peugh, J. L., Litson, K., & Feldon, D. F. (2023). Equivalence testing to judge model fit: A Monte Carlo simulation. Psychological Methods, https://doi.org/10.1037/met0000591 Medline, Google Scholar
• Wolf, M. G., & McNeish, D. (2021). Dynamic model fit (1.1.0) [Computer software]. Retrieved January 15, 2024 from www.dynamicfit.app Google Scholar
• Wolf, M. G., & McNeish, D. (2023). dynamic: An R package for deriving dynamic fit index cutoffs for factor analysis. Multivariate Behavioral Research, 58, 189–194. https://doi.org/10.1080/00273171.2022.2163476 Medline, Google Scholar