Pilot Data in PLS-SEM: What to Assess, Which Thresholds to Use, and How to Avoid Mechanical Item Deletion
A practical workflow from questionnaire pretesting and data screening to preliminary measurement assessment, helping researchers decide when to retain, revise, remove or retest an indicator before the main survey.
A pilot is neither a miniature main study nor an automatically mandatory step in every PLS-SEM project. Its greatest value is identifying problems in question comprehension, collection procedures, response quality and measurement within the particular context. PLS-SEM statistics at this stage provide preliminary diagnostic evidence, rather than the sole basis for purifying a scale.
A pilot identifies problems before the main study
- Comprehension
Check how participants understand items
- Procedure
Assess flow and response time
- Measurement
Inspect distributions and early evidence
- Revision
Record revisions before the main sample
Figure 1. Do not mechanically optimise every main-study cutoff on pilot data.
Source: Lê Minh · Conceptual illustration, not software output.1. Distinguishing pretesting, a pilot and pilot analysis
These three activities are often grouped under running a pilot, although they answer different questions. Without distinguishing them, researchers may expect a small sample to establish reliability, validity, structural relationships and the quality of the whole study simultaneously.
Cognitive pretesting
How do respondents understand terminology, questions, time frames and response options? The focus is cognitive interviewing, paraphrasing and observing sources of confusion.
Pilot survey
Does the complete recruitment, consent, routing, collection and storage procedure work? The focus is operational feasibility and quality.
Preliminary PLS-SEM analysis
Do indicators behave as anticipated in early data? The purpose is diagnostic feedback on the instrument, rather than hypothesis conclusions.
Qualitative pretesting often reveals problems that statistics cannot identify: double-barreled questions, unfamiliar terminology, double negatives, ambiguous time frames or incomplete response options. A low loading alone cannot show whether the cause is item content, translation, respondents or sampling variation.
When the aim is to improve a questionnaire, conduct qualitative pretesting before a quantitative pilot. PLS-SEM does not replace expert content-validity assessment or direct feedback from the target population.
2. Set objectives before opening SmartPLS
A useful pilot starts with a decision question rather than a threshold list. State what decisions the pilot should support before analyzing it. The following five objectives cover much survey research using PLS-SEM:
- Feasibility: completion time, dropout, recruitment channels, eligibility questions and page-routing logic.
- Comprehension: wording clarity, cultural fit, translation and interpretation of response scales.
- Response quality: missing data, unusually fast responses, prolonged straight-lining, repeated patterns or logical contradictions.
- Preliminary measurement diagnosis: outer loadings, reliability, AVE and HTMT for reflective constructs; VIF and outer weights for formative constructs.
- Planning the main study: revising the questionnaire, collection procedures, exclusion rules and main-study sample-size justification.
Attach an action criterion to each objective. For example, if more than 10% of participants misunderstand a term, revise and retest it; if median completion time exceeds the plan, shorten the demographic section; if an essential indicator loads weakly, inspect translation and interview respondents before deleting it.
3. Pilot sample size: no universal number
Numbers such as 30, 50 or 100 matter only in relation to an objective. Perneger and colleagues show that detecting questionnaire problems depends on their prevalence; smaller pretests more easily miss uncommon problems. Johanson and Brooks discuss pilot samples for initial scale development rather than a mandatory standard for all research.
| Main objective | Approach to sample size | What not to conclude |
|---|---|---|
| Cognitive interviews | Recruit purposively to cover groups with different interpretations; work iteratively until no important new content problems emerge. | A few interviews cannot estimate reliability or structural relationships. |
| Process checks and preliminary reliability | A few dozen cases may provide initial signals, but state the purpose and acknowledge substantial uncertainty. | Do not assume n ≥ 30 validates a scale. |
| Full PLS-SEM estimation | Justify the main-study sample separately using power, anticipated effects, predictor numbers and model structure. | A pilot sample is not evidence of adequate power for hypothesis testing. |
Kock and Hadaya's inverse square root method estimates minimum PLS-SEM sample size from the smallest path coefficient to be detected. It is not specifically a pilot-size formula. When applying it to the main study, state the target effect and significance level rather than turning it into a rule that 40–50 observations suffice.
4. A pilot-data assessment workflow
Move from the research question toward revision decisions. This avoids opening SmartPLS, spotting red results and removing indicators before checking the data and their meaning.
Specify objectives and action criteria
Decide whether the pilot addresses feasibility, comprehension, response quality or preliminary measurement. Prespecify conditions for retention, revision, additional collection and retesting.
Check data structure and coding
Verify names, scale ranges, reverse-worded items, branching, missing-value codes and permissible values before statistical analysis.
Screen response quality
Examine missingness, completion times, longstrings, within-person variation, contradictions, duplicates and outliers using several signals together.
Assess the correct measurement model
Do not apply reflective criteria to formative constructs. Assess both statistical evidence and content coverage for every construct.
Integrate evidence and decide
Classify each issue as retain, revise, remove or retest. Changes need substantive and empirical reasons, rather than a single threshold.
Log changes and reassess
Retain question versions, before-and-after findings, approvals and expected implications. Pretest changes that may alter meaning.
5. Screen data quality before PLS-SEM
No single rule, such as zero standard deviation or mean replacement below 5% missingness, resolves data quality. Each signal reveals only part of the problem and must be considered alongside the survey design.
| Check | Signals | Appropriate action |
|---|---|---|
| Validity and logic | Out-of-range values, miscoding, unreversed items or responses inconsistent with eligibility. | Correct traceable entry or coding errors; exclude ineligible cases using prespecified rules. |
| Missing data | Casewise and variablewise rates, missingness patterns and groups more likely to have missing data. | Investigate causes and mechanisms; consider recollection, maximum likelihood or multiple imputation where appropriate. |
| Completion time | Unusually short times relative to the distribution, accounting for devices and question branches. | Treat time as a flag rather than excluding cases using one cutoff alone. |
| Careless responding | Longstrings, low within-person variation, attention checks and contradictory item pairs. | Combine indicators and assess sensitivity before and after exclusion. |
| Duplicates and anomalies | Identifiers, timestamps, identical patterns and univariate or multivariate outliers. | Verify the process before exclusion and document rules and affected case counts. |
Mean replacement reduces variance and can distort correlations. Decisions should reflect the amount and pattern of missingness, relevant variables and analytical objectives. If respondents repeatedly skip a pilot question, examine its design and routing before attempting to fill in values.
6. Assessing reflective measurement
Reflective indicators represent a common latent concept, so assessment concerns indicator reliability, internal consistency, convergent validity and discriminant validity. The following are common reference points, not an automatic deletion algorithm.
| Criterion | Reference point | Interpretation in a pilot |
|---|---|---|
| Outer loading | ≥ 0,708 | The desired level. For 0.40–0.70, consider removal only if reliability or AVE improves without damaging content validity. Below 0.40 generally warrants close examination. |
| Cronbach's alpha, rho_A, rho_C | 0,70–0,95 | Approximately 0.60–0.70 may be acceptable in exploratory research. Very high values may indicate redundant content. |
| AVE | ≥ 0,50 | The construct explains at least half its indicators' variance. If lower, inspect loadings and content before removal. |
| HTMT | < 0.85 or < 0.90 | 0.85 is stricter; 0.90 is often used for theoretically similar concepts. A bootstrap interval that excludes 1 may provide additional evidence. |
Estimates can vary considerably in a small pilot. A loading of 0.66 is not definitive evidence of failure, and 0.72 does not guarantee stability in the main sample. Examine wording, response distributions, inter-item correlations and the sensitivity of CR/AVE to retention or deletion.
If this indicator is removed, does the construct still cover its content domain? An indicator with a modest loading but essential content may need rewriting and retesting rather than deletion.
7. Assessing formative measurement
Formative indicators jointly define a concept. They need not correlate strongly and should not be evaluated using reflective-model criteria such as Cronbach's alpha, composite reliability or AVE. Removing one may change the construct's meaning.
| Criterion | Reference or procedure | Decision |
|---|---|---|
| Content validity | Theory, expert review and target-population feedback | Confirm coverage of the construct's components before inspecting statistics. |
| Convergent validity | Redundancy analysis, path approximately ≥ 0.70 | Compare the formative construct with a defensible global measure or alternative reflective measure. |
| Collinearity | VIF ≤ 3 desirable; 3–5 requires assessment; > 5 serious | Examine overlapping content, coding and sample characteristics rather than deleting indicators solely to reduce VIF. |
| Outer weight | Bootstrap significance and magnitude | If a weight is nonsignificant, also examine its outer loading and content validity. Remove it only with a clear theoretical justification. |
Redundancy analysis helps only when the global measure genuinely represents the same concept. If none exists, add one during questionnaire design rather than attempting to repair the omission after collection.
8. Structural models and method bias
Pilot structural analysis can reveal misspecification, implausible signs, near-duplicate relationships or high predictor collinearity. Nevertheless, paths, p-values, R² and indirect effects from small samples are diagnostic rather than grounds to confirm or reject hypotheses.
Inner VIF and full collinearity VIF
For inner VIF, ≤ 3 is desirable, 3–5 warrants assessment and above 5 signals a serious concern. Full collinearity VIF with a 3.3 threshold is sometimes used to diagnose common method bias statistically, but a value below 3.3 does not prove its absence.
Address common method variance through design: anonymity, separate sources or occasions where feasible, clearer items, fewer cues about correct answers, sensible item ordering and records of potential method sources. Statistical assessment supplements rather than replaces procedural remedies.
9. A retain, revise, remove or retest matrix
A pilot should lead to explainable decisions. These categories move beyond pass/fail labels toward action.
- Retain: Clear content, no unusual response patterns, suitable statistics and preserved conceptual coverage. Reassess the evidence in the main sample.
- Revise: Evidence of problems with interpretation, translation, double-barreled wording, negatives, time frames or response options. Pretest substantial revisions again.
- Remove: Weak statistical performance plus nonessential, redundant or conceptually inappropriate content. Explain the rationale and implications for content coverage.
- Retest: Conflicting or unstable evidence, an overly homogeneous pilot, too few valid cases or revisions that may change meaning. Collect more pilot data or conduct another pretest.
| Indicator | Content evidence | Data evidence | Decision | New version |
|---|---|---|---|---|
| Example: SAT3 | Respondents interpret service in two different ways. | Loading = 0.61; unusually frequent midpoint responses. | Revise, do not remove. | Specify the service and the past 6 months; pretest again. |
| Example: PE2 | Content almost duplicates PE1. | VIF = 5.42; nonsignificant weight. | Consider removal. | Recheck the construct definition and expert opinions. |
A minimum decision log for each revised or removed indicator.
10. Sample academic reporting
Report objectives, participants, recruitment, procedures, diagnostic criteria, changes and sample limitations. Saying only that the questionnaire's reliability and validity were tested is insufficient.
Sample report
Before the main survey, we conducted cognitive pretesting and a pilot survey to assess question comprehension, collection feasibility and preliminary measurement performance. Data were screened using prespecified eligibility, missingness, completion-time and careless-response rules. Reflective constructs were assessed using outer loadings, rho_A, composite reliability, AVE and HTMT; formative constructs were assessed through content validity, redundancy analysis, VIF and outer weights. Statistics served as diagnostic evidence rather than automatic deletion rules. After comparing findings with qualitative feedback and construct definitions, [x] indicators were retained, [y] revised and pretested again, and [z] removed because [reason]. Pilot structural estimates were not used for hypothesis testing.
If assessing common method bias, a cautious statement is: We used procedural measures to limit common method bias and full collinearity VIF as an additional diagnostic. No serious collinearity was identified under the selected threshold; this assessment does not eliminate the possibility of common method bias.
11. Six common mistakes
- Treating a pilot as mandatory in every study: the need depends on instrument novelty, context, translation and operational risks.
- Selecting a sample using one universal rule: sample size must serve a particular detection or estimation objective.
- Deleting indicators solely because loadings are below 0.70: this ignores sampling variation, content validity and implications for CR/AVE.
- Using alpha, CR and AVE for formative constructs: this applies the wrong measurement logic and may remove essential components.
- Replacing all missing values with means: this ignores missingness mechanisms and can distort distributions, variances or correlations.
- Claiming no CMB based on one assessment: no single test proves that common method bias is completely absent.
12. Checklist before the main survey
- Qualitative pretesting, process piloting and preliminary PLS-SEM objectives are distinguished.
- Action criteria were prespecified rather than selected after inspecting findings.
- Pilot size serves its objective and does not substitute for main-study sample justification.
- Coding, reversed items, value ranges, branching and eligibility have been checked.
- Missing-data rates, patterns and causes were examined without automatic mean replacement.
- Careless responses were assessed using multiple signals.
- Reflective and formative constructs were distinguished before selecting criteria.
- Statistics were considered alongside content validity before deletion.
- Retention, revision, removal and retesting decisions are logged with reasons and question versions.
- Meaning-changing revisions have been pretested again.
- The main-study sample size and analysis plan are specified separately.
- The pilot's limitations will be reported transparently.
Conclusion
A useful pilot is not one in which every statistic turns green. It identifies problems early enough to improve instruments and procedures before the main data are collected. PLS-SEM organizes diagnostic evidence, but decisions still require theory, content validity, target-population feedback and recognition of small-sample uncertainty.
Before asking whether an indicator passes a cutoff, ask what it measures, why it behaves this way and whether removing it leaves the concept intact.
References
- Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2022). A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM) (3rd ed.). SAGE.
- Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43, 115–135. https://doi.org/10.1007/s11747-014-0403-8
- Johanson, G. A., & Brooks, G. P. (2010). Initial scale development: Sample size for pilot studies. Educational and Psychological Measurement, 70(3), 394–400. https://doi.org/10.1177/0013164409355692
- Kock, N. (2015). Common method bias in PLS-SEM: A full collinearity assessment approach. International Journal of e-Collaboration, 11(4), 1–10. https://doi.org/10.4018/ijec.2015100101
- Kock, N., & Hadaya, P. (2018). Minimum sample size estimation in PLS-SEM: The inverse square root and gamma-exponential methods. Information Systems Journal, 28(1), 227–261. https://doi.org/10.1111/isj.12131
- Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
- Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590
- Perneger, T. V., Courvoisier, D. S., Hudelson, P. M., & Gayet-Ageron, A. (2015). Sample size for pre-tests of questionnaires. Quality of Life Research, 24, 147–151. https://doi.org/10.1007/s11136-014-0752-2
- Podsakoff, P. M., Podsakoff, N. P., Williams, L. J., Huang, C., & Yang, J. (2024). Common method bias: It's bad, it's complex, it's widespread, and it's not easy to fix. Annual Review of Organizational Psychology and Organizational Behavior, 11, 17–61. https://doi.org/10.1146/annurev-orgpsych-110721-040030
- SmartPLS. (2026). Result color thresholds in SmartPLS reports. https://www.smartpls.com/documentation/functionalities/thresholds/