Comparing two effects in SEM with clustered data

1. Separate three statistical questions

A study may include several predictors and ask which has the stronger association with an outcome. A common mistake is to inspect two estimates, select the larger one, and describe this as established superiority. Point estimates alone do not support that conclusion.

Gelman and Stern (2006) explain that the difference between a significant result and a nonsignificant result is not itself necessarily significant. A direct contrast is needed. The same reasoning applies when both effects are significant but one estimate is larger.

Based on Gelman and Stern (2006) and Cameron and Miller (2015). SE: standard error.

A robustness table for individual paths does not replace a table testing contrasts. Conversely, a contrast bootstrapped by resampling individuals does not establish that its conclusion survives cluster-level resampling. Identify the question before selecting the relevant output.

2. The illustrative model and setting

Suppose an educational study includes 432 students in 54 classes, with 8 sampled students per class. Students report their learning experiences, while the teacher responsible for each class rates proactive learning. This is a hypothetical design, not a description of a completed survey.

The variables and their roles are hypothetical. These descriptions are not validated measurement scales. Codes are retained across the Vietnamese and English editions.

The model hypothesizes that PF and HT are associated with CD through TT. AT moderates the PF → TT and HT → TT paths. The resulting indirect associations may therefore vary with AT: a model of first-stage moderated mediation. Hayes (2015) presents an index for testing linear variation in an indirect effect.

In this example, AT is an individual-level variable. Shared class membership does not automatically turn it into a class characteristic. Testing a class-level characteristic, or separating within-class from between-class relationships, requires a research question and model that explicitly address those levels.

3. What is a cluster and why does it matter?

A cluster is a group of observations that share data-generating conditions and may consequently be dependent. Here, the cluster is the class. Students in a class share a learning environment and a teacher who rates CD. A variable such as ClassID identifies membership. This structure exists before analysis; it is not an output of a clustering algorithm.

A teacher may rate more strictly than another teacher. Students in a class may also share resources or a timetable. Their errors may therefore be correlated. Cameron and Miller (2015) explain why ignoring within-cluster correlation can make conventional standard errors overstate precision.

Other examples include employees reporting to one manager, patients in one facility, or participants at one study site. Cluster selection should follow the data-generating process. Do not try several grouping variables and retain the one producing favorable p-values. If one teacher covers several classes, or classes share a school, assess the wider dependence structure rather than assuming that class-level adjustment resolves it.

A nonsignificant test of between-class variance does not demonstrate independent observations. Use the ICC as additional descriptive information alongside the study design, rather than turning one cutoff into a universal decision rule for SEM.

4. Cluster adjustment or multilevel modeling?

Cluster-adjusted standard errors and multilevel models do not serve identical purposes. The former primarily changes inference for the pooled relationship being estimated. The latter models structure at multiple levels. They should not be treated as interchangeable options selected according to which yields more attractive results.

Based on Cameron and Miller (2015), Field and Welsh (2007), and Gelman (2006).

If the question concerns variation in the PF–TT relationship across classes, a class-robust SE alone does not answer it. If the target remains a pooled coefficient but dependence may exaggerate precision, cluster-aware inference deserves consideration. Gelman (2006) describes multilevel modeling as incorporating group structure in the model rather than merely adjusting an SE.

Many individuals in a few clusters do not provide the same independent information as the same number spread across many clusters. Report both individuals and clusters. Few clusters or highly unequal cluster sizes require attention to the selected correction and resampling method; no single cluster-count threshold guarantees valid inference for every application (Cameron & Miller, 2015).

One-way cluster-robust inference still relies on independence across clusters at the level being considered. If classes within a school remain dependent, reassess that structure. Class-level adjustment does not automatically resolve every relationship among schools, teachers, and classes.

5. Choose the bootstrap resampling unit

A bootstrap constructs repeated samples from the available data and estimates the target in each sample. The resampling unit is part of the method. Field and Welsh (2007) describe resampling clusters with replacement. In a whole-cluster bootstrap, observations inside a selected cluster are retained together.

The counts 432 and 54 belong only to the hypothetical design. With 8 sampled students in every class, each whole-cluster replicate still has 432 records. Unequal class sizes need not preserve that total.

When a cluster is drawn twice, its copies must be handled as separate draws in the chosen workflow. If the re-estimated model also uses a cluster identifier, check how replicated clusters are relabeled. Otherwise, the implemented procedure may differ from the one described.

The number of replicates, resampling unit, and interval construction method are separate specifications. “5,000 replications” does not reveal whether students or classes were resampled. Percentile, bias-corrected (BC), and bias-corrected and accelerated (BCa) intervals must also be named accurately. BCa adds acceleration to bias correction; not every bootstrap interval is BCa (Efron, 1987).

Increasing the number of replicates does not repair an inappropriate resampling unit. Establish the design first, then examine convergence, inadmissible solutions, usable replications, and the stability of the resulting distribution.

6. Define the quantity being compared

Consider a linear model with two predictors, one shared mediator, and first-stage moderation. The equations omit the measurement model and controls to focus on coefficient interpretation. Coefficients refer to the unstandardized scales in use.

At AT = w, the PF slope is a_PF + u_PFw and the HT slope is a_HT + u_HTw. If b is not moderated, their conditional indirect effects are (a_PF + u_PFw)b and (a_HT + u_HTw)b. Their indices of moderated mediation are u_PFb and u_HTb, expressing linear change in an indirect effect with AT (Hayes, 2015).

Contrast between interaction coefficients

This asks whether the rate at which the predictor-to-mediator slope changes with AT differs across predictors. It does not directly compare the slopes at one particular AT value.

Contrast between indices of moderated mediation

This contrasts changes in the indirect effects as AT changes. The simplified expression applies because both branches share TT and the same b coefficient. Different mediators, different b paths, or second-stage moderation require expressions reflecting those structures.

Contrast between indirect effects at a specified AT value

This is a different question. Similar indices can coexist with different indirect effects at AT = 0 when the a coefficients differ. To state which predictor has the larger indirect effect at low AT, define and bootstrap that particular conditional contrast.

7. Obtain the contrasts in AMOS 28

The following workflow obtains ordinary bootstrap results for the defined targets. Do not split the sample into PF and HT groups: both predictors enter one model. The official user-defined estimand documentation supports functions of model parameters with bootstrap SEs, intervals, and significance tests (Amos Development Corporation, n.d.).

These labels support the example workflow. Constructing product terms and specifying interactions must remain consistent with the selected model. Labels do not replace checking the measurement model or latent interaction specification.

Index_PF = p.intPF * p.b
Index_HT = p.intHT * p.b
Diff_Interaction = p.intPF - p.intHT
Diff_Index = (p.intPF * p.b) - (p.intHT * p.b)

Use parameter labels so AMOS recomputes each target within each bootstrap sample. Do not substitute rounded constants such as 0.140 and 0.100: that would remove the variability of interest. Do not subtract two individual SEs or interval endpoints to construct the SE or interval for a difference.

Retain the model, estimand definitions, and output from the same run. Intervals can differ across bootstrap runs even when point estimates coincide. Report the software version, planned and usable replication counts, interval method, resampling unit, and contrast direction.

8. Read two hypothetical results tables

The following tables contain assumed teaching values, not results from AMOS or another package. They illustrate different questions and a situation where inferential methods do not agree completely. In an empirical analysis, replace every numerical entry with traceable output.

Sensitivity of individual effects

Hypothetical design: 432 students, 54 classes, and 5,000 planned whole-class resamples. MLR: maximum likelihood with robust inference. B is unstandardized. Indirect effects refer to AT = 0; call this the mean only when the actual scaling and centering support that interpretation. All SEs, intervals, and p-values in the example are assumed, not calculated from data.

For PF → CD, the assumed MLR p is .041 while the assumed cluster-bootstrap interval includes zero. For the HT indirect effect, p is .057 while the interval excludes zero. These columns use different inferential procedures and need not agree exactly. In a real analysis, first verify that they concern the same parameter, model, and data. Then report method sensitivity rather than selecting the favorable column.

Do not call an MLR p-value the p-value corresponding to a BC interval from another procedure. Preserve the method labels. This table also cannot establish differences between predictors because it contains no direct contrast.

Direct contrasts

Suppose Table 7 uses ordinary student-level bootstrapping within the same model and run. The illustrative point products are consistent: 0.140 × 0.450 = 0.063 and 0.100 × 0.450 = 0.045. SEs, intervals, and p-values are assumed teaching values, not software output.

PF is numerically larger, but both contrast intervals include zero. The appropriate statement is that the chosen procedure does not provide sufficient evidence of a difference. It does not establish that PF is more effective. To assess robustness of the contrasts themselves, resample whole classes and recompute Diff_Interaction and Diff_Index within each replicate. The individual-effect results in Table 6 do not supply that test.

Failure to detect a difference also does not establish equivalence. Lakens (2017) explains that equivalence testing requires prespecified bounds for differences considered practically negligible and an appropriate testing procedure. A large p-value for a zero-difference test is insufficient.

9. Reporting templates

Replace the placeholders below with actual study information. Methods should identify the unit of analysis, resampling unit, and bootstrapped target. Results should state the estimate, uncertainty, and inferential limits.

If methods disagree, identify the affected path and procedures. Avoid describing all findings as fully robust when an important effect changes conclusion. Neither one p-value nor one interval warrants causal language unsupported by the design.

10. Common mistakes

Based on Gelman and Stern (2006), Cameron and Miller (2015), Hayes (2015), Lakens (2017), and the official AMOS estimand guidance.

11. Before publication

• Identify clusters from sampling, organization, or assessment, and report both N and the cluster count.
• Distinguish individual effects, effect contrasts, and robustness of those contrasts.
• State scales, centering, and contrast direction; verify each estimand against the model.
• Keep the model, data, missing-data handling, and estimation settings aligned when comparing methods.
• Report the resampling unit, interval type, planned and usable counts, and admissibility criteria.
• Retrieve SEs, intervals, and p-values from the correct output sections; preserve their method labels.
• Check cluster-aware robustness for the contrasts themselves when they are central to the study.
• Disclose method-sensitive conclusions and do not interpret a large p-value as proof of equivalence.