Research Methods
Five survey design mistakes that invalidate research findings
GOSPELTRADER Research Desk · 20 August 2026 · 7 min read
Sampling, wording and scale errors that surface too late — and the checks that catch them before data collection begins.
Analysis cannot repair a badly designed instrument. By the time responses are in, the errors below are permanent — which is why the design stage deserves more time than the analysis.
1. Convenience sampling described as representative
Sampling whoever is reachable is legitimate when stated. It becomes misconduct when the write-up generalises to a population the sample never covered.
2. Double-barrelled questions
"Is the service fast and affordable?" cannot be answered by a respondent who thinks it is fast and expensive. Every item should test exactly one construct.
3. Unbalanced scales
Response options must be symmetrical around a neutral point. Three positive options against one negative option guarantees a favourable result that means nothing.
4. No pilot
A pilot of twenty respondents exposes ambiguous wording, broken skip logic and completion fatigue while it is still cheap to fix.
- • Pre-register the sampling frame and target size
- • Pilot the instrument and record where respondents hesitate
- • Compute reliability for multi-item scales before analysis
- • Document every deviation from the original plan
5. Choosing the test after seeing the data
The analysis plan should be written before collection. Selecting tests once the results are visible turns hypothesis testing into storytelling.