Top of page
Skip to main content

Scale development: when to build a new measure and how to defend the process

Most theses do not need a new scale. When a new measure is justified, development involves three phases and nine steps, from defining the construct to validating it in an independent sample. An omitted step is likely to attract questions at the defence.

The sensible order is to adopt an established measure, then consider adaptation, and develop a new scale only when existing measures fail to cover the construct in the research context. A new scale needs evidence about its content, structure and relationships with other variables, including expert review, cognitive interviews, EFA, independent CFA, reliability and validity.

Start reading

1. Adopt, adapt or develop

Saunders et al. (2023), drawing on Bourque and Clark, distinguish three ways of obtaining survey questions, adopting existing wording, adapting existing questions and developing new ones. The first two take considerably less time, and published measures also support comparisons with earlier research. The UK Data Service question bank contains more than 250,000 questions from completed surveys, so searching existing questions should precede writing new ones.

The same source describes scale development as complex and time-consuming and recommends adopting or adapting existing measures whenever possible. Changes should be limited to what is necessary. Altering wording, response categories or item order may affect established validity. Researchers reusing a measure should check copyright, obtain permission where required and credit its original authors.

Boateng et al. (2018) add a specific requirement. If a measure of a similar construct already exists, researchers must explain why a new instrument is needed and how it differs. Common reasons include inadequate coverage of an important facet in a new context, development for a substantially different population, such as Western consumers rather than Vietnamese household businesses, or a construct too new to have an established measure.

2. Three phases and nine steps

Churchill's (1979) classic marketing framework covers specifying the domain, generating items, collecting data, purifying the measure, collecting further data, assessing reliability and validity, and developing norms. Hinkin (1998) presents a comparable approach for organisational research. Boateng et al. (2018) synthesise these frameworks into three phases and nine steps, a convenient structure for the methods chapter.

Phase 1, step 1. Specify the domain and generate items

Define the construct and its boundaries, then generate a pool considerably larger than the final scale.

Phase 1, step 2. Assess content validity

Experts and members of the target population assess each item's fit with the definition.

Phase 2, steps 3 and 4. Pretest, sample and administer

Conduct cognitive interviews and a pilot survey, followed by a main survey with enough observations for factor analysis.

Phase 2, steps 5 and 6. Reduce items and extract factors

Use item–total relationships and EFA to identify weak items and determine how many factors to retain.

Phase 3, steps 7 to 9. Assess dimensionality, reliability and validity

Run CFA in an independent sample and evaluate internal consistency, convergent, discriminant and criterion-related validity.

The Vietnamese edition of Saunders et al. (2010) uses the corresponding terms for content validity, criterion-related validity and construct validity. The Vietnamese article follows those translations so readers can compare it with the textbook.

3. Define the construct and its content domain

MacKenzie et al. (2011) identify three recurring weaknesses in behavioural and information-systems measurement, inadequate construct definitions, misspecified reflective or formative models, and underuse of validation techniques. All can begin when researchers write items before deciding precisely what they intend to measure.

A workable definition should answer four questions. Does the construct belong to an individual, group or organisation? Is it an attribute, behaviour, perception or intention? Is it unidimensional or multidimensional, and how do its dimensions relate? How does it differ from neighbouring constructs?

4. Generate the initial item pool

Hinkin (1998) distinguishes deductive and inductive item generation. Deduction starts with definitions and theory and produces items covering each facet. Induction starts with respondents, using interviews, focus groups or open questions and coding their responses into items. Boateng et al. (2018) recommend combining the approaches. In Vietnamese research, inductive work can reveal expressions of a construct absent from overseas literature.

≥ 2 times

The initial pool should contain at least twice as many items as the final scale (Boateng et al., 2018).

≤ 20 words

Recommended question length, attributed to Sekaran and Bougie in Saunders et al. (2023).

5 to 7

Independent experts recommended for item review (Boateng et al., 2018).

The surplus is practical. Expert review, pretesting and factor analysis will remove items. A construct intended to have four or five final items should begin with about ten. Starting with four leaves little room for deletion before stable factor estimation becomes difficult.

5. Content validity and expert review

Saunders et al. (2023) discuss establishing content validity through a clear literature-based definition and a panel rating questions as essential, useful but not essential, or unnecessary. These categories originate with Lawshe (1975), who proposed the content validity ratio, CVR, to quantify panel agreement.

CVR = (ne − N/2) / (N/2)

Here ne is the number rating an item essential and N is the total number of experts. CVR ranges from −1 to +1. A positive value means more than half consider the item essential. Lawshe (1975) provides minimum values by panel size, with higher thresholds for smaller panels. Boateng et al. (2018) recommend five to seven independent experts and discuss Cohen's kappa for inter-rater agreement.

The panel should combine theoretical expertise with target-population experience. When measuring digital capabilities among household-business owners, two or three actual owners may identify confusing wording that academics miss.

6. Write questions and choose response categories

Constructs such as satisfaction and loyalty are not directly observable, so multiple item scores are combined. Saunders et al. (2023) note the interval-data assumption behind this practice, citing Murphy and Likert's 1937 work. Negatively worded items therefore need reverse coding before summation or analysis.

Number of response points

Hinkin (1998) reports that alpha increases up to five Likert response points and then largely levels off. Boateng et al. (2018) recommend five categories for unipolar items, such as frequency, and seven for bipolar items, such as strongly disagree to strongly agree. Two- or three-category scales tend to have lower reliability. Saunders et al. (2023) add that the choice also depends on likely measurement error in the particular respondent group.

Table 1. Response-anchor families, adapted from Table 11.3 in Saunders et al. (2023). The supplied article paraphrases these anchors in Vietnamese.
RatingFive-point anchor exampleUse
AgreementStrongly disagree, disagree, neutral, agree, strongly agreeAttitudes, beliefs and perceptions about a statement
Extent or amountNot at all, very little, somewhat, quite a lot, very muchExtent to which a characteristic is present
FrequencyNever, rarely, sometimes, often, alwaysRepeated behaviour
LikelihoodVery unlikely, unlikely, unsure, likely, very likelyIntentions and future behaviour

Item-writing principles

  • Ask about one thing per item. “Employees are friendly and provide fast service” combines two questions.
  • Keep response ordering consistent. Reversing category order partway through can increase errors (Dillman et al., as cited in Saunders et al., 2023).
  • Positively and negatively worded statements may reduce habitual responding, but reverse-worded items must be clear and avoid double negatives.
  • Avoid unfamiliar jargon and wording that suggests a preferred answer.

7. Translation and cultural adaptation

Many measures used in Vietnamese research originate in English. Drawing on Usunier and colleagues, Saunders et al. (2023) distinguish lexical, idiomatic, experiential and grammatical or syntactic equivalence. Experiential equivalence asks whether respondents have the experiences assumed by an item.

Table 2. Three questionnaire-translation approaches, synthesised from Saunders et al. (2023).
ApproachProcedureStrengthLimitation
Direct translationOne person translates from source to target language.Quick and inexpensiveMeaning shifts are difficult to detect without an independent comparison.
Back translationOne person translates into Vietnamese. A second independently translates back into English. The English versions are compared.Helps detect meaning shiftsRequires two proficient translators. Equivalent wording may still sound unnatural.
Parallel translationTwo or more translate independently, then a group compares versions and selects wording.Can produce natural target-language phrasingTime-consuming and requires skilled coordination

Include both original and translated scales in the thesis appendices, as Saunders et al. (2023) recommend. This lets examiners inspect individual items rather than rely on an unsupported statement that translation was careful.

8. Pretesting

Saunders et al. (2023) recommend expert feedback before piloting, both to improve questions and support content validity. Cognitive interviewing then asks respondents to explain how they understand questions while answering them. Boateng et al. (2018) suggest 5 to 15 interviews in two or three rounds, or until no new problems emerge.

For pilot surveys, Saunders et al. (2023) cite Fink's minimum of ten participants for most student projects, whereas Dillman and colleagues suggest 100 to 200 for major surveys. Testing with friends is better than skipping testing, but mainly supports face validity.

Questions for pilot participants, adapted from Bell and Waters as cited in Saunders et al. (2023), include the following.

  • How long did completion take?
  • Were the instructions clear?
  • Which questions were unclear or difficult?
  • Which questions felt uncomfortable?
  • Were important topics omitted?
  • Was the layout easy to follow?
  • What other suggestions do you have?

9. Item reduction and structural assessment

Boateng et al. (2018) review the common heuristic of ten respondents per item and absolute recommendations of 200 to 300 for factor analysis. They cite Comrey and Lee's descriptive scale, 100 is poor, 200 fair, 300 good and 500 very good. This stage needs considerably more data than wording pretests.

Item reduction

Inter-item relationships and corrected item–total correlations below .30 may flag items for review (Boateng et al., 2018). The review describes EFA loadings below .30 as inadequate, loadings of .40 or above as commonly retained, and cross-loadings as potential grounds for removal. Numerical criteria should be weighed against content coverage. Rewriting may be preferable when a weak item uniquely represents part of the definition.

Number of factors

Boateng et al. (2018) discuss scree plots, explained variance, parallel analysis, minimum average partial (MAP) and the Hull method. The article contrasts these with SPSS's familiar eigenvalue-greater-than-one default. Reporting parallel analysis is particularly helpful when the eigenvalue rule conflicts with theory.

Validation in independent data

Assess the EFA structure through CFA in another sample or data collected later (Boateng et al., 2018). Fit guidelines cited in that review include RMSEA ≤ .06, CFI and TLI ≥ .95, and SRMR ≤ .08. The supplied article proposes randomly splitting the main sample for EFA and CFA when collecting another sample is difficult.

10. Reliability and validity

Reliability

Saunders et al. (2023) discuss internal consistency, alternative-form checks and test–retest reliability. Alpha is commonly interpreted against .70. Repeated questions should be used sparingly because they lengthen the questionnaire and may irritate respondents. Test–retest requires surveying the same people twice and can provide supplementary evidence. Boateng et al. (2018) describe .70 as commonly acceptable and .80 or above as preferred.

Validity

Table 3. Validity evidence to report for a new scale.
TypeQuestionCommon assessment
ContentDo items adequately cover the definition?Literature-based definition, expert panel, CVR (Lawshe, 1975)
ConvergentDo items for the same construct converge?Standardised loadings and AVE ≥ .50 (Fornell & Larcker, 1981)
DiscriminantIs the construct distinct from related constructs?Square root of AVE greater than construct correlations (Fornell & Larcker, 1981), HTMT below .85 or .90 (Henseler et al., 2015)
Criterion-relatedDoes the score predict the theoretically expected criterion?Correlation or regression with an independently measured criterion (Saunders et al., 2023)

Henseler et al. (2015) show that Fornell–Larcker and cross-loading assessments often miss discriminant-validity problems and propose HTMT. For a new scale, report HTMT and its bootstrap confidence interval alongside the earlier criterion.

Criterion-related validity is frequently overlooked even though it addresses whether a scale is useful beyond internal consistency. A new organisational-commitment scale should, for example, relate negatively to turnover intention if theory predicts that relationship. An attractive factor structure alone is insufficient.

11. A thesis reporting template

Example for the methods chapter

No existing measure of [construct] adequately addressed [population and context]. A new scale was therefore developed following Boateng et al. (2018). The construct was defined as [definition], with [number] facets, [list], and specified as [reflective/formative]. The initial [number] items were generated from the literature and [number] interviews. [Number] experts assessed content validity, and [number] items below the CVR threshold of [value] were removed. Following [number] cognitive-interview rounds with [number] participants, [number] items were revised. The main survey yielded [N] usable responses, randomly divided into two subsamples. EFA in the first (n = [number]) retained [number] items and [number] factors. CFA in the second (n = [number]) yielded CFI = [value], TLI = [value], RMSEA = [value] and SRMR = [value]. Factor alphas ranged from [value] to [value], AVE from [value] to [value], and all HTMT values were below [threshold]. Scale scores related [direction and magnitude] to [criterion], consistent with theory.

12. Common errors and checklist

Table 4. Common scale-development reporting problems.
Weak practiceProblemBetter practice
“The scale was adopted and adapted from previous research.”Changes and reasons are unspecified.List each revised item, justify the change and append the original.
Initial and final item counts are identical.No room to remove weak itemsStart with at least twice the final number.
“Experts were consulted.”No ratings or change recordReport panel size, rating procedure, CVR and removals.
Deleting items following EFA in a pilot of 50Unstable factor structure in a small sampleUse piloting for wording and run EFA in an adequately sized main sample.
EFA and CFA in the same dataCFA revisits a tuned result.Split the sample or collect another.
Alpha for a formative constructFormative items need not correlate.Assess collinearity and item weights following MacKenzie et al. (2011).

Before submission

  • Search existing scales and justify their inadequacy.
  • Define the construct, dimensions and measurement specification.
  • Document item sources and an initial pool at least twice the final size.
  • Retain expert ratings and agreement measures.
  • Record cognitive-interview findings and revisions.
  • Append source and translated versions where applicable.
  • Justify the factor-analysis sample, considering the cited ten-per-item or 200–300 heuristics.
  • Use factor-retention evidence beyond eigenvalues greater than one.
  • Run CFA independently of EFA.
  • Report reliability, convergent, discriminant (including HTMT) and criterion-related validity.

Conclusion

Scale development is a research project in its own right. With the time available for a master's thesis, adapting an established measure and demonstrating equivalence is often more feasible.

If a new scale is warranted, its value rests on the evidence chain, definition, expert review, cognitive interviews, adequately sampled EFA, independent CFA and criterion assessment. Missing links weaken the case at review.

References

Textbooks and methodological sources

  • Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
  • Churchill, G. A., Jr. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16(1), 64–73. https://doi.org/10.1177/002224377901600110
  • Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121. https://doi.org/10.1177/109442819800100106
  • MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Quarterly, 35(2), 293–334. https://doi.org/10.2307/23044045
  • Saunders, M., Lewis, P., & Thornhill, A. (2010). Phương pháp nghiên cứu trong kinh doanh (Nguyễn Văn Dung, dịch). Nhà xuất bản Tài chính.
  • Saunders, M. N. K., Lewis, P., & Thornhill, A. (2023). Research methods for business students (9th ed.). Pearson.

Additional sources

  • Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.2307/3151312
  • Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135. https://doi.org/10.1007/s11747-014-0403-8
  • Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

This article synthesises the listed academic sources for methodological reference. Numerical guidelines should be interpreted in context and checked against institutional and target-journal requirements.