Survey design

From Wikiversity

Overview This learning resource is about how to design surveys (or questionnaires) for systematically obtaining data from human beings. Before designing a survey, the research purpose, research questions and/or hypotheses should be clear. Survey types include paper-based, online, face to face, and via phone. Surveys may be used to collect quantitative (at any level of measurement) and/or qualitative data. Survey questions may be open-ended or closed-ended, and they may be objective or subjective. Response formats include dichotomous, multichotomous, ranking, semantic differential, and Likert scales. Sampling involves the process of gathering representative data using the survey from a target population. Collected survey data is then usually entered into a computer file and subjected to quantitative and/or qualitative data analysis. Surveys are commonly used in disciplines such as psychology, health, marketing, sociology, governance, and demographics. The main advantage of survey research is its efficiency in gathering a large amount of data. The main disadvantages relate to the difficulty in developing reliable and valid survey measures and representative sampling. In summary, a well-conducted survey research project should exhibit: Clarity of the research purposes, research questions and hypotheses) Careful, iterative development of well-worded questions, using appropriate response formats and A well-designed and implemented sampling method Contents 1 Survey development steps 2 Modes of administration 3 Survey structure 4 Types of questions 5 Design principles 6 Response formats 7 Pre-testing and piloting a survey 8 Sampling 9 Biases 10 Readings 11 See also 12 References 13 External links Educational level: this is a tertiary (university) resource. Completion status: this resource is ~33% complete. Go to the workshop. Feel free to make suggestions...

[edit] Survey development steps

1. Before designing a survey: Clarify research purpose, develop research questions and research proposal, including:
   1. Determine research design: Experimental, quasi-experimental, non-experimental
   2. Determine sampling method
   3. Conceptualise and describe the target constructs (independent and dependent variables) and make sure they are operationally defined
2. To develop an initial draft survey:
   1. Create separate sections for each main purpose/research question/hypothesis
   2. Within each section, brainstorm ways data about topic/question could be obtained and draft items (questions) which you expect can provide a reliable and valid measure of the target constructs; items may also be obtainable from previous surveys. Start off with lots more possible questions/items (based on the operational definitions) than will actually be used; this way, you can cull and refine, using only the best items
   3. For each consider, brainstorm
   4. Add an informed consent statement, a coversheet, and an instructions page
3. Get the draft survey critically reviewed by others, then redraft etc.
4. Get assistance with high quality word-processing skills (if you don't have them) to tweak the essay so that it looks professional
5. Pre-test the survey (on convenient others), redraft etc.
6. Pilot test the survey (on target population), redraft etc.
7. Use the survey in a major study

[edit] Modes of administration

What are the strengths and weaknesses of the following modes of administration?

Hard-copy (paper and pencil)

Structured interviews: Face to face

Online

Structured interviews: Telephone

[edit] Survey structure

1. Cover letter
2. Informed consent
3. Ethics complaints
4. Sections containing survey questions
   1. Personal details / demographics
   2. One section per major topic
5. Debrief information

[edit] Types of questions

It is surprisingly difficult to develop a "good" survey question or item. Consider each of the following aspects of survey questions, their pros and cons, and with examples:

1. Objective vs. subjective
2. Close-ended vs. open-ended
3. Leading and loaded questions
4. Positive-, negative-, and double-negative-wording

[edit] Design principles

Jenkins and Dillman (1995^[1]) suggest these general self-report survey design principles:

1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired navigational path for respondents to follow when answering the questionnaire.
2. When established format conventions are changed in the midst of a questionnaire use prominent visual guides to redirect respondents.
3. Place directions where they are to be used and where they can be seen.
4. Present information in a manner that does not require respondents to connect information from separate locations in order to comprehend it.

[edit] Response formats

It is important to understand the implications of response formats on levels of measurement in survey design and quantitative data analysis.

Some commonly used response formats include:

• Dichotomous: e.g., Yes or No
• Multi-chotomous: e.g., Yes, No, or Maybe
• Multiple response: e.g., Tick all that apply
• Likert scale: Equally-spaced intervals, usually 3 to 9 intervals
• Graphical rating: Can mark any point on a continuous scale
• Semantic differential: Put two words at opposite ends of a scale with interval marks
• Idiographic: Use symbols/pictures instead of words and numbers
• For more info see: Rating scale (Wikipedia)

[edit] Pre-testing and piloting a survey

Pre-testing

1. Have a few people you know look over the survey and fill it out; ask for their feedback and suggestions and make relevant changes

Pilot testing

1. Arrange for a small group from the target population to complete the survey; analyse their responses, ask for their feedback, and make relevant changes

[edit] Sampling

There are a dazzling array of possible sampling strategies. It is worth considering their strengths, weaknesses, and applicability to your specific situation:

1. Random sampling
2. Systematic random sampling
3. Stratified sampling
4. Clustering sampling
5. Convenience sampling

It is important to understand the purpose of sampling, which is to permit generalization and do so with a tolerable margin of error.

[edit] Biases

Several biases may influence the reliability and validity of results, including:

1. Social desirability bias
2. Order effect
3. Fatigue effect
4. Novelty effect

[edit] Readings

Recommended readings about how to design questionnaires:

Online articles

1. Creative Research Systems (2008). Survey design: How to begin your survey project.
2. Frary, R. B. (1996). Hints for designing effective questionnaires. ERIC Digest.
3. Leung, W. (2001). How to design a questionnaire. Student BMJ, 9, 171-216.
4. Pollograph (2008). Designing a survey.
5. StatPac (c. 2007). Questionnaire design considerations.

Book chapters

1. Fowler, F. J., Jr. (2002). Designing questions to be good measures. In In F. J. Fowler, Survey research methods (3rd ed.) (pp. 76-103). Thousand Oaks, CA: Sage.
   • Google Books (Note: pp. 80-81, 87-89, 91-92, 95, 97, 100-102 are missing)
   • html (earlier version, full text)
   • Or contact the instructor for a full copy
2. Nardi, P. (2006). Developing a questionnaire (Ch 4). In Doing survey research: A guide to quantitative methods (2nd ed.). Boston, MA: Pearson. UC-eReserve

Books

1. Alreck, P. L., & Settle, R. B. (2004). The survey research handbook (3rd ed.). Boston: McGraw-Hill/Irwin.
2. Backstrom, C. H., & Hursh-César, G. (1981). Survey research (2nd ed.). New York: Wiley.
3. Dillman, D. A. (2007). Mail and internet surveys: The tailored design method 2007 Update with new internet, visual, and mixed-mode guide (2nd ed.). Hoboken, NJ: Wiley.
4. Frazer, L., & Lawley, M. (2000). Questionnaire design & administration: A practical guide. Brisbane, Australia: John Wiley.

[edit] See also

Wikiversity

• Online surveys
• Readings
• Survey design workshop
• Survey design introduction and overview presentation
• Survey research
• Human research ethics

Wikibooks

• Survey/Questionnaire
• Introduction to Psychology/Surveys

Wikipedia

• Questionnaire
• Statistical survey

[edit] References

1. Spector, P. E. (1994). Using self-report questionnaires in research: A common on the use of a controversial method. Journal of Organizational Behavior, 15, 385-392.

[edit] External links

• Survey Design I (Lecture; Neill, 2008, University of Canberra)
• Survey Design II (Lecture; Neill, 2008, University of Canberra)
• Survey design (Google Group)