Common Mistakes when Reporting Q², PLSpredict, and CVPAT
The PLS-SEM predictive assessment framework has changed substantially in recent years: PLSpredict replaced blindfolding and Stone-Geisser Q², then CVPAT supplemented PLSpredict. Many published articles still reflect this transition, from outdated tools to misinterpreted p-values. This article identifies frequent errors and specific corrections.
Start readingPrediction needs observations held out from estimation
- Split observations
Set folds and repetitions
- Fit
Use the training observations
- Evaluate
RMSE, MAE and Q²predict
- Benchmark
Read IA, LM and CVPAT in their roles
Conceptual diagram. Use consistent data splits when comparing prediction benchmarks.
Source: Lê Minh · Conceptual illustration, not software output.1. Outdated Tools
Official SmartPLS technical documentation states that blindfolding was discontinued and removed in SmartPLS 4 because it does not provide out-of-sample predictive assessment (SmartPLS, Blindfolding documentation). Yet many new manuscripts still describe running blindfolding for Q² as a standard step, often following a template published before 2019, when PLSpredict (Shmueli et al., 2019) was not yet widespread.
The timing of methodological milestones partly explains this persistent error. PLSpredict appeared in 2019 and CVPAT in 2021, while SmartPLS formally removed blindfolding only in version 4. Articles following older reporting examples, or theses inheriting earlier cohorts' structure, can easily retain the habit of running blindfolding for Q² even though the current software no longer supports it.
Zhang, Adams, Cheah, Teeroovengadum, and Ramayah's (2026) systematic review of 62 PLS-SEM articles in leading educational leadership and management journals provides specific figures: 17 of 62 articles (27,4%) reported blindfolding-based Q², only 3 (4,8%) reported CVPAT, and PLSpredict appeared explicitly in just one article (Thien and Yeap, 2023). The review covers educational leadership and management rather than all interdisciplinary PLS-SEM literature. It nevertheless provides concrete quantitative evidence of a substantial gap between current recommendations and writing practice in at least one carefully examined field.
Stone-Geisser Q² from blindfolding is calculated within the sample using omission and data reconstruction rather than generating actual out-of-sample predictions. Calling it predictive ability assessment is therefore a methodological inaccuracy rather than merely a wording issue.
2. Terminology and Notation
These errors occur even in articles using PLSpredict and CVPAT. The most common is confusion between Q², the in-sample Stone-Geisser measure from blindfolding, and Q²predict, obtained through out-of-sample cross-validation in PLSpredict. Their similar symbols measure different things. A second error conflates IA, the naive indicator-average benchmark for Q²predict, with LM, the linear regression benchmark for RMSE/MAE classification and CVPAT. The benchmarks serve distinct roles within PLSpredict.
Q² (Stone-Geisser, blindfolding): in-sample; removed from SmartPLS 4. Q²predict (PLSpredict): out-of-sample; compared with IA. IA: naive benchmark using the training-sample mean. LM: linear regression benchmark for RMSE/MAE classification and CVPAT.
3. Interpreting PLSpredict
Three errors recur. First, examining RMSE/MAE without checking for negative Q²predict can lead to classifying a variable that already underperforms the most naive IA benchmark. Second, reporting only RMSE or MAE without justifying the choice ignores potential disagreement under skewed errors (Chai and Draxler, 2014; Willmott and Matsuura, 2005). Third, and most seriously, the win-loss counting rule of Shmueli et al. (2019) is interpreted as a significance test, using terms such as statistically significant superiority even though PLSpredict supplies no p-value for that comparison.
| Common wording | Why it is wrong |
|---|---|
| "The model predicts well because RMSE is below LM for most indicators, with a statistically significant difference." | PLSpredict has no p-value; statistical significance is justified only by also running CVPAT and citing that test result. |
| "Variable X has Q²predict of -0.03, but this is acceptable because it is close to 0." | Any negative Q²predict indicates performance below the most naive IA benchmark. The original guidance contains no acceptable-close-to-0 threshold. |
4. Interpreting CVPAT
Because CVPAT supplies a p-value, authors often quote it without stating the rejected H0 of equal predictive ability, leaving readers unsure what was tested. Another error reports only one benchmark, often LM because its smaller difference sounds more modest and credible, while omitting IA even though SmartPLS 4 calculates both by default. Some authors also intuitively dismiss small absolute loss differences, such as -0.02, although statistical significance depends on the p-value rather than absolute size alone.
Whenever reporting a CVPAT p-value, state the benchmark, IA or LM, the average loss difference, and the hypothesis being rejected. Omitting any of these leaves the interpretation vulnerable when a reviewer asks what supports the claim.
5. The Broader Error: Confusing Explanation with Prediction
Underlying these errors is confusion between explanatory power, assessed through in-sample R², f², and path coefficients, and predictive power, assessed through out-of-sample Q²predict, RMSE/MAE, and CVPAT. Shmueli and Koppius (2011) warned of this distinction before PLSpredict existed. Yet claims that high R² implies high predictive value still appear frequently, even in articles that correctly run PLSpredict and CVPAT in Results but summarize them incorrectly in Conclusions.
6. Overview Table
| Error group | Manifestation | Correction |
|---|---|---|
| Outdated tools | Reporting blindfolding-based Stone-Geisser Q² as predictive evidence. | Use PLSpredict (Q²predict, RMSE, MAE) and CVPAT, explaining that they replace blindfolding in SmartPLS 4. |
| Terminology | Confusing Q² with Q²predict or conflating IA with LM. | Use correct notation and define each benchmark at its first appearance. |
| PLSpredict interpretation | Treating win-loss counts as a statistical test; overlooking negative Q²predict. | Check positive Q²predict first, then use the counting rule of Shmueli et al. (2019) descriptively, without calling it statistically significant. |
| CVPAT interpretation | Quoting a p-value without its hypothesis; reporting one benchmark only. | State H0/H1 and report both IA and LM with their respective loss differences and p-values. |
| General interpretation | Concluding high predictive ability from R² alone. | Distinguish explanation (R² and path coefficients) from prediction (Q²predict and CVPAT) in the conclusions. |
7. Pre-submission Checklist
- Avoid presenting blindfolding or Stone-Geisser Q² as predictive evidence, except in historical methodological comparisons.
- Check positive Q²predict for all main endogenous variables before examining RMSE/MAE.
- Report RMSE and MAE together or explain why only one is used.
- Do not call PLSpredict results statistically significant without accompanying CVPAT results.
- For each CVPAT result, report its loss difference, p-value, and corresponding IA/LM benchmark.
- Do not combine R² and predictive ability into one conclusion.
Conclusion
Most PLS-SEM predictive reporting errors arise because guidance and writing habits have not kept pace with methodological changes over less than a decade, from blindfolding through PLSpredict to CVPAT. A brief pre-submission checklist covering the correct tool sequence, terminology, and certainty of each conclusion is an inexpensive way to prevent most of these errors.
References
Primary Sources
- Shmueli, G., Ray, S., Estrada, J.M.V. and Chatla, S.B. (2019). Predictive model assessment in PLS-SEM: Guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322-2347. https://doi.org/10.1108/EJM-02-2019-0189
- Liengaard, B.D., Sharma, P.N., Hult, G.T.M., Jensen, M.B., Sarstedt, M., Hair, J.F. and Ringle, C.M. (2021). Prediction: Coveted, yet forsaken? Introducing a cross-validated predictive ability test in partial least squares path modeling. Decision Sciences, 52(2), 362-392. https://doi.org/10.1111/deci.12445
- Sharma, P.N., Liengaard, B.D., Hair, J.F., Sarstedt, M. and Ringle, C.M. (2022). Predictive model assessment and selection in composite-based modeling using PLS-SEM: Extensions and guidelines for using CVPAT. European Journal of Marketing, 57(6), 1662-1677. https://doi.org/10.1108/EJM-08-2020-0636
- Shmueli, G. and Koppius, O.R. (2011). Predictive analytics in information systems research. MIS Quarterly, 35(3), 553-572.
- Chai, T. and Draxler, R.R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? Arguments against avoiding RMSE in the literature. Geoscientific Model Development, 7(3), 1247-1250. https://doi.org/10.5194/gmd-7-1247-2014
- Willmott, C.J. and Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30(1), 79-82. https://doi.org/10.3354/cr030079
Review of Reporting Practice
- Zhang, Y., Adams, D., Cheah, J.-H., Teeroovengadum, V. and Ramayah, T. (2026). An assessment of the use of partial least squares structural equation modelling (PLS-SEM) in educational leadership and management research. European Journal of Education, 61, e70600. https://doi.org/10.1111/ejed.70600
Academic note: Statements about blindfolding's removal come directly from official SmartPLS documentation. The 27,4% and 4,8% figures in Section 1 come from Table 6 of Zhang et al. (2026), covering 62 PLS-SEM articles in educational leadership and management. A sentence in the original article's Discussion says only one article applied CVPAT, whereas its own Table 6 reports 3 articles (4,8%). This article does not attempt to resolve that internal inconsistency on the original authors' behalf.