CVPAT: Theory, Hypotheses, and Interpretation in PLS-SEM

Many readers of PLS-SEM reports look only at CVPAT's p-value in the PLSpredict table before deciding whether a model has superior predictive ability. That number is the result of a fully structured hypothesis test involving a loss function, an average difference, and a specific test statistic. This article examines CVPAT's workings so that SmartPLS users can interpret and defend the number under peer review.

PLS-SEM & PREDICTIVE ABILITYPrepared by Lê Minh8 min

Start reading

Prediction needs observations held out from estimation

Split observations
Set folds and repetitions
Fit
Use the training observations
Evaluate
RMSE, MAE and Q²predict
Benchmark
Read IA, LM and CVPAT in their roles

Conceptual diagram. Use consistent data splits when comparing prediction benchmarks.

Source: Lê Minh · Conceptual illustration, not software output.

1. The Gap CVPAT Addresses

PLSpredict (Shmueli et al., 2019) ranks predictive ability by counting indicators for which PLS-SEM's RMSE or MAE is lower than that of a linear regression model (LM). This is a useful heuristic but leaves a gap: it does not establish whether the observed difference is reliable or simply random noise in a particular sample. Two studies may both outperform LM on 7 of 9 indicators, yet their conclusions about good prediction may differ greatly in reliability if one has a sample size of 150 and the other 600.

Liengaard et al. (2021) formulated this issue as a formal statistical hypothesis test, the Cross-Validated Predictive Ability Test (CVPAT). Sharma et al. (2022) subsequently extended the original framework, which compared two competing theoretical models, into a practical procedure comparing a PLS-SEM model with standardized benchmarks (IA and LM), as currently implemented in SmartPLS 4.

How CVPAT differs from PLSpredict

PLSpredict answers a descriptive question: "For how many indicators does PLS-SEM have lower prediction error than LM?" CVPAT answers an inferential question: "Is the prediction error difference large enough relative to its own variation across the sample to reject the possibility that it is merely random?" CVPAT's answer is a p-value rather than a win-loss proportion.

2. The Loss Function: Measuring Prediction Error

CVPAT uses the same cross-validation technique as PLSpredict, with 10 folds repeated 10 times by default in SmartPLS 4, to generate out-of-sample predictions for each observation. For each observation, CVPAT computes a loss value: the mean squared prediction error across all indicators of the endogenous variable under consideration (Liengaard et al., 2021). Lower loss indicates a more accurate prediction for that observation.

Because both models, PLS-SEM and its comparator, produce a separate loss for the same observation, CVPAT can calculate each observation's loss difference and then average those differences across the sample. The original article denotes this average by D̄. It is the average loss difference reported in the SmartPLS CVPAT output.

Why calculate differences for individual observations?

Retaining loss differences at the observation level, rather than only comparing two aggregated RMSE values as in PLSpredict, allows CVPAT to calculate the variance of the differences themselves. Without that variation, one cannot determine whether an observed average difference is stable enough to be statistically significant. This is what PLSpredict does not establish.

3. CVPAT's H0 and H1 Hypotheses

In the original framework of Liengaard et al. (2021), CVPAT compares two models, denoting the estimated model by EM and the alternative model by AM. The null hypothesis (H0) states equal predictive ability, meaning an average loss difference of zero. The alternative hypothesis (H1), in the one-sided form commonly used in practice, states that one model's loss is systematically lower than the other's.

H0 (null hypothesis)loss(model) − loss(comparator) = 0

The two models have equal predictive ability on out-of-sample data. There is no evidence that the PLS-SEM model predicts better or worse than the benchmark.

H1 (one-sided alternative hypothesis)loss(model) − loss(comparator) < 0

The PLS-SEM model has statistically significantly lower average loss than the benchmark, establishing predictive superiority beyond simply counting indicators on which it wins.

In SmartPLS 4's practical implementation, following the extension by Sharma et al. (2022), the alternative model (AM) is replaced by two standardized benchmarks, IA and LM, also used in PLSpredict. CVPAT in SmartPLS therefore runs two parallel tests for each endogenous variable: one against IA and one against LM, each with the hypothesis structure above.

4. The Test Statistic: A Form of Paired t-test

Liengaard et al. (2021) specify a CVPAT test statistic of T = D̄ divided by the square root of (S²/N), where D̄ is the average loss difference described in Section 2, S² is the sample variance of the observation-level loss differences, and N is the sample size. The authors explicitly describe it as a form of paired t-test (Liengaard et al., 2021, p. 367) and note its structural similarity to the Diebold and Mariano (1995) test, familiar from comparisons of prediction accuracy between time-series models.

The central point is that CVPAT applies paired t-test logic to comparing two models' out-of-sample prediction errors. A more negative T, indicating lower PLS-SEM loss relative to the comparator, and a larger absolute value relative to its standard error provide stronger evidence of predictive superiority.

5. The p-value: t Distribution or Bootstrap?

Liengaard et al. (2021) propose two ways to convert T into a p-value. The first compares T with a t distribution with N − 1 degrees of freedom, as in a conventional t-test. The second uses bootstrap resampling of the observed loss differences to estimate the distribution of D̄ empirically, without assuming a t distribution. In their illustrative example, the original authors used 10.000 bootstrap samples to calculate p-values and confidence intervals.

Why this matters for reporting

Public SmartPLS technical documentation does not detail its internal method for calculating CVPAT p-values, such as whether it uses a t distribution or bootstrap and, if applicable, how many bootstrap repetitions. Users should report the p-value and average loss difference directly from SmartPLS output and cite Liengaard et al. (2021) and Sharma et al. (2022) for the methodology, rather than adding an account of internal calculations that they cannot substantiate.

6. IA and LM: Comparators in SmartPLS 4

According to SmartPLS technical documentation, CVPAT uses the same cross-validation approach as PLSpredict, with 10 folds repeated 10 times by default, to generate out-of-sample predictions. It then computes the PLS-SEM model's average loss and compares it with two benchmarks: IA, a naive benchmark, and LM, a more conservative benchmark. SmartPLS uses the convention that a statistically significantly negative average loss difference, meaning lower PLS-SEM loss than the benchmark, supports a conclusion of predictive superiority.

These IA and LM benchmarks are the same as those used in PLSpredict; see "IA, LM, RMSE, and MAE in PLSpredict" in this series. CVPAT can therefore be viewed as a formal statistical test of the comparison that PLSpredict evaluates by counting.

7. Reading an Actual CVPAT Results Table

In Jumani's (2026) study, an extended TPB model of Islamic brand consumption behavior was assessed with CVPAT at the overall model level against both IA and LM.

Table 1
Comparison against	Average loss difference	p-value	Interpretation in terms of H0/H1
Indicator Average (IA)	-0.40	< 0.001	Reject H0; PLS-SEM loss is statistically significantly lower than IA loss
Linear Model (LM)	-0.02	< 0.001	Reject H0; PLS-SEM loss is statistically significantly lower than LM loss, despite a much smaller absolute difference

The Overall row in Table 13 (CVPAT) of Jumani (2026). The absolute difference against LM is much smaller than against IA, as expected because LM is the stronger benchmark. Both nevertheless remain sufficiently small relative to the variation of the loss differences to reach p < 0.001.

p < 0.001The significance level of both CVPAT comparisons, against IA and LM, in Jumani's (2026) model

-0.02The loss difference against LM: small in absolute size but statistically significant because of sample size and the stability of the difference

N − 1Degrees of freedom for comparing T with a t distribution, according to the original formula of Liengaard et al. (2021)

8. Common Mistakes

Table 2
Common wording	Why it is wrong	Better wording
"CVPAT produces a p-value, so it must be more reliable than PLSpredict."	This overlooks the different questions answered by CVPAT and PLSpredict, inferential and descriptive. They should be reported together rather than treated as substitutes.	Present PLSpredict as a detailed descriptive view by indicator and CVPAT as a complementary statistical test at the overall level.
"A small negative loss difference, such as -0.02, is negligible."	The absolute size of a loss difference does not determine statistical significance by itself. It must be considered relative to its variation, reflected in the p-value, rather than judged intuitively.	Interpret the result using the accompanying p-value and/or confidence interval, rather than the absolute loss difference alone.
"CVPAT compares two theoretical models, as SmartPLS does."	The original framework of Liengaard et al. (2021) compares two competing theoretical models, but SmartPLS 4 follows the extension of Sharma et al. (2022), using standardized IA/LM benchmarks instead of a user-selected alternative model.	Cite both sources explicitly: Liengaard et al. (2021) for the theoretical foundation and Sharma et al. (2022) for the practical IA/LM procedure in SmartPLS.

State clearly that CVPAT tests H0 of equal loss against H1 of lower PLS-SEM loss, rather than merely saying that CVPAT is significant.
Report both average loss differences and p-values for both IA and LM, rather than selecting only one benchmark.
Cite Liengaard et al. (2021) for the theoretical foundation and Sharma et al. (2022) for the practical IA/LM procedure.
Do not speculate about SmartPLS's internal p-value calculation without supporting documentation.

Example Results Wording

CVPAT results (Liengaard et al., 2021; Sharma et al., 2022) indicate an average loss difference of [value] between the PLS-SEM model and the [IA/LM] benchmark (p [=/<] [value]), rejecting H0 of equal predictive ability. This statistically supports the predictive superiority previously observed descriptively through PLSpredict.

Conclusion

CVPAT is a fully structured hypothesis test rather than an additional check run after PLSpredict: a loss function measures prediction error for each observation, D̄ represents the average difference, and a paired t-test statistic can be compared with a t distribution or assessed through bootstrap. Understanding this mechanism helps PLS-SEM authors explain why the result matters instead of mechanically copying a p-value from SmartPLS. It also helps them defend predictive results when reviewers ask how the number was calculated.

References

Primary CVPAT sources

Liengaard, B.D., Sharma, P.N., Hult, G.T.M., Jensen, M.B., Sarstedt, M., Hair, J.F. and Ringle, C.M. (2021). Prediction: Coveted, yet forsaken? Introducing a cross-validated predictive ability test in partial least squares path modeling. Decision Sciences, 52(2), 362-392. https://doi.org/10.1111/deci.12445
Sharma, P.N., Liengaard, B.D., Hair, J.F., Sarstedt, M. and Ringle, C.M. (2022). Predictive model assessment and selection in composite-based modeling using PLS-SEM: Extensions and guidelines for using CVPAT. European Journal of Marketing, 57(6), 1662-1677. https://doi.org/10.1108/EJM-08-2020-0636
Diebold, F.X. and Mariano, R.S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253-263.
Shmueli, G., Ray, S., Estrada, J.M.V. and Chatla, S.B. (2019). Predictive model assessment in PLS-SEM: Guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322-2347. https://doi.org/10.1108/EJM-02-2019-0189

Illustrative source

Jumani, Z.A. (2026). Predictive validation of an extended TPB model in Islamic branding: PLSpredict and CVPAT evidence from Muslim and non-Muslim Thai consumers. Journal of Islamic Marketing, ahead-of-print. https://doi.org/10.1108/JIMA-09-2025-0574

Academic note: The description of the test statistic, including the loss function, D̄, T, and degrees of freedom, follows Liengaard et al. (2021), the original article introducing CVPAT. Public SmartPLS technical documentation does not fully describe internal SmartPLS 4 implementation details, such as p-value calculation. This article presents only what can be substantiated from public sources.