Data Analysis

Survey Data Analysis: Cleaning, Weighting, Scales and Valid Inference

Q: Can I calculate a mean from Likert data?

For a single item, a mean is strictly inappropriate because the intervals are not known to be equal — report the median and the frequency distribution. For a multi-item scale with acceptable reliability, means and parametric tests are standard practice; state the treatment explicitly in the methods.

Q: What response rate is acceptable?

There is no universal threshold. What matters is non-response bias, not the rate itself. Compare respondents with the population on known characteristics, or compare early and late respondents as a proxy, and report the comparison rather than defending a percentage.

Q: How large should a survey sample be?

For estimating a proportion in a large population at 95% confidence with a 5% margin of error, roughly 385 completed responses are needed; a 3% margin needs about 1,067. For regression or SEM, sample size is driven by the number of parameters, commonly 10-20 cases per estimated parameter.

Q: Do I need weights for a convenience sample?

Weights cannot repair a convenience sample, because the selection probabilities are unknown. Post-stratification to population margins can improve representativeness on those specific variables, but the inference remains non-probability and the limitation must be stated plainly.

GOSPELTRADER Research Desk · 12 September 2026 · 12 min read

Quick answer

Survey data analysis runs in five stages: screen and clean the responses, diagnose and handle missing data, establish the reliability and validity of every multi-item scale, apply design weights if the sample is not a simple random sample, then run tests matched to the measurement level and report effect sizes alongside significance.

1. Screening: decide the exclusion rules before you look at results

Write the exclusion criteria into the analysis plan before examining any outcome, and report how many cases each rule removed. Deciding afterwards turns cleaning into an undisclosed researcher degree of freedom.

• Speeders — completion time below roughly one third of the median; flag and inspect rather than delete blindly.
• Straight-lining — zero or near-zero variance across a reversed-item block.
• Attention checks failed — state the rule and the number removed.
• Duplicates — same respondent ID, or identical response vectors from one IP within a short window.
• Out-of-range or impossible values — an age of 3 on a workforce survey, a 7 on a 5-point item.
• Partial completions — set a completeness threshold (commonly 80% of core items) and apply it consistently.

2. Missing data: the mechanism decides the remedy

Deleting incomplete cases is only defensible when data are missing completely at random, which is rarely true in surveys — sensitive items (income, health, misconduct) are missing for reasons connected to the answer itself.

Mechanism	Meaning	Acceptable handling
MCAR	Missingness unrelated to anything	Listwise deletion acceptable; Little's MCAR test non-significant
MAR	Explained by observed variables	Multiple imputation (m ≥ 20) or FIML
MNAR	Depends on the unobserved value itself	Sensitivity analysis; model the mechanism; state the limitation
Any, < 5% scattered	Minor	Mean/median substitution tolerable for scale items only

3. Reliability and validity of the instrument

A construct score means nothing until the instrument is shown to measure it consistently and distinctly. Report reliability for every composite, not just an overall figure for the questionnaire.

Reliability
  Cronbach's α ≥ 0.70 acceptable      (α inflates with item count)
  McDonald's ω — preferred when items load unequally
  Composite reliability (CR) ≥ 0.70

Convergent validity
  Average variance extracted  AVE = Σλ²ᵢ / k  ≥ 0.50
  Standardised loadings λ ≥ 0.50 (0.70 preferred)

Discriminant validity
  Fornell-Larcker: √AVE of a construct > its correlation with every other
  HTMT ratio < 0.85

Sampling adequacy before factor analysis
  KMO ≥ 0.60 and Bartlett's test of sphericity significant

4. Weighting and complex designs

If the sample was stratified, clustered, or drawn with unequal probabilities, unweighted estimates are biased and unweighted standard errors are too small. Use survey-design procedures, not ordinary ones.

A design weight is the inverse of the selection probability; post-stratification then rakes the sample to known population margins such as age, gender and region. Report the weighted and unweighted N, the variables used for raking, and the design effect.

Design weight      w_i = 1 / π_i        (π_i = probability of selection)

Design effect      DEFF = Var_complex / Var_SRS
Effective sample   n_eff = n / DEFF

Software: SPSS Complex Samples, R {survey} svydesign(), Stata svyset,
Python {samplics}. Do not run ordinary regression on weighted survey data.

5. Choosing the test for Likert and categorical data

A single Likert item is ordinal; a summated multi-item scale with acceptable reliability is conventionally treated as continuous. Say which treatment you adopted and why, because the test follows from that decision.

Question	Single ordinal item	Summated scale
Two independent groups	Mann-Whitney U	Independent samples t-test
Three or more groups	Kruskal-Wallis H	One-way ANOVA
Association between two measures	Spearman's rho	Pearson's r
Two categorical variables	Chi-square, Cramér's V	—
Predicting a scale outcome	Ordinal logistic regression	Multiple linear regression

6. Common method bias and reporting

When predictors and outcome come from the same self-report instrument, some shared variance is artefactual. Test it — Harman's single-factor test at minimum, a marker-variable or unmeasured latent method factor approach if the study is inferential — and report the result rather than ignoring it.

The results section states: the population and sampling frame, the achieved and usable N, the response rate, exclusions with counts, the missing-data strategy, reliability per scale, whether weights were applied, and every test with its effect size and confidence interval.

Frequently asked questions

Can I calculate a mean from Likert data?

For a single item, a mean is strictly inappropriate because the intervals are not known to be equal — report the median and the frequency distribution. For a multi-item scale with acceptable reliability, means and parametric tests are standard practice; state the treatment explicitly in the methods.

What response rate is acceptable?

There is no universal threshold. What matters is non-response bias, not the rate itself. Compare respondents with the population on known characteristics, or compare early and late respondents as a proxy, and report the comparison rather than defending a percentage.

How large should a survey sample be?

For estimating a proportion in a large population at 95% confidence with a 5% margin of error, roughly 385 completed responses are needed; a 3% margin needs about 1,067. For regression or SEM, sample size is driven by the number of parameters, commonly 10-20 cases per estimated parameter.

Do I need weights for a convenience sample?

Weights cannot repair a convenience sample, because the selection probabilities are unknown. Post-stratification to population margins can improve representativeness on those specific variables, but the inference remains non-probability and the limitation must be stated plainly.

Want this analysis done for you?

We deliver cleaning, assumption testing, modelling and a reporting-ready write-up.

Start a project