Research Scale Development: A 9-Step Process from Construct Definition to CFA Validation

1. Learning Objectives and Prerequisites

After this article, you should distinguish new development, adaptation, and revalidation; map conceptual domains; organize expert reviews and cognitive interviews; select EFA and CFA data; and report reliability and validity evidence without turning cutoffs into automatic item-removal mechanisms.

DeVellis and Thorpe (2022) place clear identification of what is to be measured first. Without clear theory, conceptual scope, and score-use purposes, later factor analysis can only organize ambiguous items into an even more ambiguous structure.

• Provide a theoretically grounded conceptual definition.
• Know whether scores serve group description, relationship testing, or individual decisions.
• Review existing scales and usage rights.
• Clearly identify the population, language, and application context.
• Have resources for at least one pilot round and an independent confirmation step.
• Plan to retain all removed items and their removal reasons.

2. First: Develop, Adapt, or Use an Existing Scale Unchanged?

New development is justified only when existing scales do not cover the concept, population, language, or score purpose. Using or carefully adapting a suitable instrument generally preserves comparability and resources. Boateng et al. (2018) recommend confirming that no existing instrument meets the purpose before creating a new scale.

Translation involves more than replacing words. Check conceptual, linguistic, response, and score-use equivalence. Following adaptation, prior evidence does not automatically transfer intact to the new context.

3. Define the construct and map its domain

A working definition specifies what the concept is, its dimensions, its differences from neighboring concepts, and its intended population and context. Clark and Watson (2019) treat clear conceptualization, broad item pools, and comparisons with related concepts as foundations of construct validity.

Record two threats in the domain map: construct underrepresentation and construct-irrelevant variance. An overly narrow domain omits content; an overly broad domain includes items measuring other characteristics. Content validity concerns scale use within a population and context rather than a fixed instrument label (DeVellis and Thorpe, 2022).

4. Decide first: a reflective scale or a formative index?

The item correlations, EFA, CFA and internal-consistency procedures in this guide primarily concern reflective scales, whose items express a common attribute. Sharing a topic does not establish a shared latent cause. Chapter 7 of DeVellis and Thorpe (2022) distinguishes such scales from indices defined by their components. Neither the label scale nor the presence of Likert responses resolves that distinction.

Constructed example: expressions of trust in an adviser’s competence might be hypothesized to reflect one common dimension. Numbers of consultations, available support channels and waiting time are different service components. Combining them into an index makes the chosen components part of its definition. Do not remove a component solely because alpha is low, or force the components into one factor to obtain high internal consistency (DeVellis & Thorpe, 2022).

5. Three phases and nine steps: distinguish the source frameworks

Boateng et al. (2018) describe three phases and nine steps: item development covers domain identification/item generation and content validity; scale development covers pretesting, sampling, item reduction and factor extraction; scale evaluation covers dimensionality, reliability and validity. The illustration below is an integrated teaching map, not a reproduction of the authors’ procedure table.

The nine steps in chapter 5 of DeVellis and Thorpe (2022) divide the work differently: define the target attribute, generate items, choose the format, obtain expert review, conduct cognitive interviews, consider validation items, collect a development sample, evaluate items and optimize length. The fourth edition has eight steps and does not separate cognitive interviewing into its own step (DeVellis, 2017). Equal step counts do not make the frameworks interchangeable; the detailed sections below follow practical decisions.

6. Generate a broad item pool within the construct boundary

The initial pool should be larger than the final scale, giving every domain component several opportunities for representation. Conceptual redundancy can help, but nearly copied wording may generate method correlations and inflate reliability (DeVellis and Thorpe, 2022).

Combine deduction from theory and existing scales with induction from interviews, focus groups, or observation. Each item should address one idea, match language proficiency, specify timing when needed, and avoid experiences unavailable to most respondents. Hinkin (1998) likewise situates item generation within a clear development and psychometric-testing sequence.

• Each dimension has multiple candidate items.
• Items reflect the construct rather than merely a shared topic.
• No pairs differ by only a few words.
• Item specificity matches conceptual specificity.
• Wording suits target respondents.
• Record sources and rights for inherited items.
• Use reversed items only with justification rather than as a trick against mechanical responding.

7. Choose response formats as part of measurement

More response points are not automatically better. Respondents must meaningfully distinguish levels. A midpoint suits genuine neutrality; an even-numbered scale suits necessary directional choice only without misrepresenting genuinely neutral respondents. DeVellis and Thorpe (2022) base midpoint decisions on purpose and question nature, with no universally superior format.

Likert, semantic differential, binary, and visual analog scales yield different information. With multiple collection modes, examine whether identical items and options function equivalently on paper, telephone, and web.

8. Expert review and content-validity evidence

Give experts conceptual definitions and dimension descriptions before item rating. They assess relevance, representativeness, clarity, concision, omissions, and confusion with neighboring concepts. A CVI value cannot replace reasons why items do or do not represent the domain (DeVellis and Thorpe, 2022; Boateng et al., 2018).

Include content experts and measurement-method specialists. Where feasible, involve target-population members to identify overlooked language or experiences. Record final decisions in a disposition table rather than saving disconnected rating files alone.

9. Calculate CVI with explicit denominators, formulas and limits

Under a four-point relevance protocol, ratings of 3 or 4 count as relevant. I-CVI divides the endorsing experts by the experts assessing that item under the protocol: 7 / 8 = .875. S-CVI/Ave averages the I-CVIs; S-CVI/UA is the proportion of items receiving unanimous endorsement. Reporting only S-CVI leaves these different calculations ambiguous (Polit & Beck, 2006).

In the convention discussed by Polit and Beck (2006), three to five experts require item-level unanimity; for six to ten experts, .78 is commonly used. Endorsements are integers: five of six or seven of eight are needed to exceed .78. This is a selected screening convention, not a guarantee of validity. For larger panels or different guidance, identify the applicable source rather than extending a threshold table without justification. For S-CVI/Ave, this source recommends .90 for average relevance; meeting it still requires a substantive content argument.

The constructed example in Figure 2 has four items and eight experts, with endorsement counts of 8, 7, 5 and 8. Their I-CVIs are 1.000, .875, .625 and 1.000; S-CVI/Ave = .875 and S-CVI/UA = .500. The third item warrants review. If it alone represents a subdomain, consider revision or new candidates and obtain fresh evidence rather than automatically removing it (DeVellis & Thorpe, 2022).

# Constructed relevance ratings, rows = items, columns = experts.
ratings <- rbind(
  NT1 = c(4,3,4,3,4,4,3,4),
  NT2 = c(4,3,4,2,4,3,4,4),
  NT3 = c(3,2,3,2,4,3,2,3),
  NT4 = c(4,4,3,4,4,4,4,4)
)
stopifnot(!anyNA(ratings), all(ratings %in% 1:4))
relevant <- ratings >= 3
i_cvi <- rowMeans(relevant)
s_cvi_ave <- mean(i_cvi)
s_cvi_ua <- mean(rowSums(relevant) == ncol(ratings))
data.frame(item = rownames(ratings), votes = rowSums(relevant), I_CVI = i_cvi)
c(S_CVI_Ave = s_cvi_ave, S_CVI_UA = s_cvi_ua)
# Expected: 0.875 and 0.500. Complete ratings only; no missing-value rule inferred.

10. Conduct cognitive interviews before a larger pilot

Cognitive interviews ask target-population members to interpret items, describe what they considered, and explain their response-level choices. This is more direct evidence than a general question about understandability. Willis (2005) provides a practical framework for think-aloud and verbal probing.

Not everyone needs every probe for every item. Divide items into groups to reduce burden while ensuring coverage of all items and important respondent groups. The objective is to identify systematic misunderstanding patterns rather than remove every criticized item.

11. Plan validation measures before collecting the development sample

Step 6 in DeVellis and Thorpe (2022) considers validation items before collecting the development sample. Candidates include measures of related and distinct constructs, external criteria or response tendencies where justified. They do not automatically become items of the scale being developed.

Prespecify the expected direction, timing and type of score relationship, with a theoretical rationale. Selecting items and evaluating them against a criterion created from those same items provides little independent evidence. Check source, permission and respondent burden for each external measure; do not add numerous scales simply to find a small p-value (DeVellis & Thorpe, 2022).

12. Plan development samples for stability, not a mechanical ratio

Samples must belong to the intended population and vary sufficiently on the construct. Excessive homogeneity may make items appear weak because of range restriction. Samples very different from the population may produce structures that do not replicate in actual use (DeVellis and Thorpe, 2022).

No respondent-to-item ratio applies universally. Sample size depends on factor counts, items per factor, communalities, loadings, estimators, categorical data, and complexity. Boateng et al. (2018) describe common rules while recommending sufficiently large samples and independent validation. Justify the intended model through power or precision rather than merely citing a 10:1 rule.

13. Choose Correlations and Estimators According to Data Type

EFA and CFA should not automatically treat all Likert items as continuous. For ordinal items, few categories, or pronounced skewness, polychoric correlations with categorical estimators such as WLSMV are often more appropriate than Pearson correlations and ordinary ML. Flora and Curran (2004) support considering categorical estimation when categories are few and continuous assumptions unreasonable.

With enough categories, modest skewness, and a defensible continuous approximation, ML or robust ML such as MLR may be considered. Neither every Likert item requiring WLSMV nor every 5-point scale allowing ML is a valid universal rule. Consider category counts, distributions, sample size, models, and software implementation. For EFA, Fabrigar et al. (1999) recommend theoretically connected decisions about extraction, factor counts, and rotation.

14. Evaluate items, EFA and CFA as connected decisions

Begin with missingness, distributions, floors/ceilings, item correlations, and corrected item-total correlations. Negative correlations may reflect incorrect reversal, different wording, or different dimensions. Do not remove items merely to increase alpha when they protect important domain content (DeVellis and Thorpe, 2022).

EFA explores dimensionality and loading patterns. CFA tests a specified structure. Clark and Watson (2019) prioritize unidimensionality and construct validity over maximizing internal consistency. High alpha may result from near-duplicate items while important conceptual content remains absent.

15. Lock scoring, reverse coding and missing-response rules

Evidence supporting a structure does not uniquely determine its scoring rule. Item means, sums, CFA factor scores and PLS-SEM construct scores are different calculations. A mean uses equal weights; factor scores depend on the fitted model and scoring method; PLS scores depend on the specification and algorithmic weights. State the calculation, item version and score direction rather than calling every aggregate a latent score (DeVellis & Thorpe, 2022; SmartPLS, n.d.).

This constructed example only demonstrates scoring three items of a hypothetical subscale; it does not establish unidimensionality. For responses from 1 to 5, Q2R is reversed as 6 minus its original response. Compare requiring all three items with allowing at least two. The two-of-three policy is an example assumption, not a general cutoff. Do not turn not applicable into a neutral response; permitted missingness must follow the manual or a justified protocol.

# Constructed teaching data, not an estimated measurement model.
raw <- data.frame(Q1=c(4,3,5,NA,2), Q2R=c(2,1,3,2,NA), Q3=c(5,4,NA,3,2))
items <- raw
items$Q2R <- 6 - raw$Q2R  # Reverse once; retain the raw values.
n_valid <- rowSums(!is.na(items))
mean_complete <- ifelse(n_valid == 3, rowMeans(items), NA_real_)
mean_min2 <- ifelse(n_valid >= 2, rowMeans(items, na.rm=TRUE), NA_real_)
data.frame(n_valid, mean_complete, mean_min2)
# The two-of-three rule is illustrative, not a universal scoring recommendation.

16. Hypothetical Example: Developing Trust in Methodological Advisors

The following example is illustrative; all values were created to demonstrate decisions. The team defines trust through ability, integrity, and benevolence. Theory and qualitative interviews generate 27 items. Expert review retains 22; cognitive interviews revise 6 and remove 2 double-barreled items. The pilot contains 20 items.

EFA in a hypothetical development sample suggests three factors. Two items have similar cross-loadings, but one uniquely covers benevolence content. The team retains that item for revision and retesting rather than immediate deletion. CFA in a hypothetical independent sample gives the revised item a loading of 0,67. The three-factor model outperforms the one-factor model. Describe this as initial support for a three-dimensional structure rather than permanent proof of scale correctness.

17. Read reliability and validity as accumulating evidence

Reliability is necessary but insufficient for validity. Alpha or omega indicate consistency under a model and dataset rather than whether the intended concept is measured. Very high values may indicate duplicated content; low values may reflect multidimensionality, few items, range restriction, or substantial error (DeVellis and Thorpe, 2022).

Validity is an argument about score interpretation and use. Prespecify hypotheses relating scores to similar variables, distinct variables, and external criteria. Boateng et al. (2018) and Clark and Watson (2019) treat dimensional confirmation, reliability, and theoretically grounded relational networks as complementary rather than interchangeable.

18. Ten Errors That Produce Attractive but Hard-to-Defend Scales

19. APA 7 Reporting Template and Prepublication Checklist

Methods should identify item sources, rights, generation or translation, expert composition, I-CVI and S-CVI calculations, cognitive-interview procedures, stage-specific samples, variable types, correlations, estimators, and decision criteria. Hinkin (1998) notes that weak measurement undermines interpretation of relationships in field research; scale development should not be reduced to stating that Cronbach’s alpha was checked.

Appendices should preserve all trial and final items, instructions, response scales, coding, disposition logs, and item-to-domain matrices. If copyright prevents item publication, still describe procedures sufficiently and direct readers to legitimate sources.

• Explain why development or adaptation is necessary.
• Provide sourced definitions, dimensions, and conceptual boundaries.
• The item pool covers the domain and records each item’s source.
• Expert reviews specify rating levels, I-CVI formulas, correctly named S-CVI indices, and qualitative comments.
• Cognitive interviews involve the intended population and revision rounds.
• Describe a relevant, sufficiently diverse development sample.
• Match EFA correlations, estimators/extraction, and rotation to data types.
• Do not decide factor counts using one cutoff.
• Give substantive and statistical reasons for every removed item.
• CFA uses information independent of exploration.
• Reliability suits the purpose and does not replace validity.
• Convergent, discriminant, and criterion evidence follows hypotheses.
• Consider measurement invariance for group comparisons.
• Avoid claiming the scale has been proved completely valid.
• Version item files, codebooks, and disposition logs.