Survey design

From Wikiversity
Jump to: navigation, search
Sciences humaines.svg Educational level: this is a tertiary (university) resource.
Progress-0500.svg Completion status: this resource is ~50% complete.
Crystal kthememgr.svg Go to the workshop

Overview[edit]

This learning resource is about how to design surveys (or questionnaires) in the social sciences.

Surveys are commonly used in disciplines such as psychology, health, marketing, sociology, governance, and demographics.

Survey research is an efficient way of gathering data to help address a research question. The main challenge is developing reliable and valid measures and sampling representative data.

Survey design is critical in determining the quality of research. The potential for poor design is vast - whether intentionally on the part of the researcher or unintentionally. For example, watch this [http://www.youtube.com/watch?v=G0ZZJXw4MTA 2 min. episode of Yes, Minister] about politicians trying to get the poll results they want.

Before designing a survey[edit]

It can be very tempting to press ahead with designing a survey. But first, be clear about the purpose of the study and the research methodology.

Designing a survey? Don't put the cart before the horse. Develop a proposal first, then design the survey.

Before designing a survey, develop a research proposal which clearly explains the:

  1. research purpose
  2. research questions
  3. hypotheses
  4. Research design: Experimental, quasi-experimental, non-experimental
  5. Sampling method
  6. Target constructs - operationally define the:
    1. (independent variables
    2. dependent variables

Have the research proposal peer reviewed and modify as appropriate. Before designing a survey, it is helpful, and generally recommended, to clearly establish a research proposal and to get this proposal peer-reviewed (and/or reviewed by a supervisor). Investment in developing the proposal is generally returned many-fold.

The seven Ps apply to survey design: Prior preparation and planning prevents piss-poor performance.

Poor research results and conclusions emerge from poor data, which is often due to poor survey design. Hence, a well-conducted survey research project should exhibit:

  • clarity in the project's purposes (and specific research questions and hypotheses)
  • careful development of well-worded questions with appropriate response formats and/or
  • a well designed and implemented sampling method

Designing a survey[edit]

Survey questions may be:

  1. open-ended or
  2. closed-ended

They may also be:

  1. objective or
  2. subjective.

When designing the response format, it is critical to consider the level of measurement).

Choices of response formats include:

  1. dichotomous
  2. multichotomous
  3. ranking
  4. semantic differential
  5. Likert scales

Initial draft survey[edit]

  1. Create separate sections for each main purpose/research question/hypothesis
  2. Within each section, brainstorm ways data about topic/question could be obtained and draft items (questions) which you expect can provide a reliable and valid measure of the target constructs; items may also be obtainable from previous surveys. Start off with lots more possible questions/items (based on the operational definitions) than will actually be used; this way, you can cull and refine, using only the best items
  3. For each consider, brainstorm
  4. Add an informed consent statement, a coversheet, and an instructions page
  5. Get the draft survey critically reviewed by others, then redraft etc.
  6. Get assistance with high quality word-processing skills (if you don't have them) to tweak the essay so that it looks professional
  7. Pre-test the survey (on convenient others), redraft etc.
  8. Pilot test the survey (on target population), redraft etc.
  9. Use the survey in a major study

Survey structure[edit]

  1. Cover letter
  2. Informed consent
  3. Ethics complaints
  4. Sections containing survey questions
    1. Personal details / demographics
    2. One section per major topic
  5. Debrief information

Types of surveys[edit]

Types of surveys are:

  1. Hard copy
  2. Electronic
  3. Face to face
  4. Telephone


What are the strengths and weaknesses of the following modes of administration?

Hard-copy (paper and pencil)

Structured interviews: Face to face




Online




Structured interviews: Telephone

Types of questions[edit]

It is surprisingly difficult to develop a "good" survey question or item. Consider each of the following aspects of survey questions, their pros and cons, and with examples:

  1. Objective vs. subjective
  2. Close-ended vs. open-ended
  3. Leading and loaded questions
  4. Positive-, negative-, and double-negative-wording

Types of data[edit]

Surveys can be used to collect:

  1. quantitative and/or
  2. qualitative

data.

Response formats[edit]

It is important to understand the implications of response formats on levels of measurement in survey design and quantitative data analysis.

Some commonly used response formats include:

Layout[edit]

Jenkins and Dillman (1995[1]) suggest these general self-report survey design principles:

  1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired navigational path for respondents to follow when answering the questionnaire.
  2. When established format conventions are changed in the midst of a questionnaire use prominent visual guides to redirect respondents.
  3. Place directions where they are to be used and where they can be seen.
  4. Present information in a manner that does not require respondents to connect information from separate locations in order to comprehend it.

Pre-testing and piloting a survey[edit]

Pre-testing
  1. Have a few people you know look over the survey and fill it out; ask for their feedback and suggestions and make relevant changes
Pilot testing
  1. Arrange for a small group from the target population to complete the survey; analyse their responses, ask for their feedback, and make relevant changes

It is important to understand the purpose of sampling, which is to permit generalization and do so with a tolerable margin of error.

Biases[edit]

Several biases may influence the reliability and validity of results, including:

  1. Social desirability bias
  2. Order effect
  3. Fatigue effect
  4. Novelty effect
  5. Demand characteristics

After designing a survey[edit]

Once a survey is designed, gather representative data] via sampling, then conduct data analysis.

Sampling

There are a dazzling array of possible sampling strategies. It is worth considering their strengths, weaknesses, and applicability to your specific situation:

  1. Random sampling
  2. Systematic random sampling
  3. Stratified sampling
  4. Clustering sampling
  5. Convenience sampling

Summary[edit]

In summary, a survey research project should exhibit:

  1. Clarity of research purposes, research questions and hypotheses
  2. Well-worded survey questions, using appropriate response formats and
  3. An appropriate sampling method

Readings[edit]

A tower of used books - 8447.jpg

These recommended readings explain survey (questionnaire) design in more detail:

Online articles
  1. Author unknown (nd). Smart survey design.
  2. Creative Research Systems (2008). Survey design: How to begin your survey project.
  3. Frary, R. B. (1996). Hints for designing effective questionnaires. ERIC Digest.
  4. Leung, W. (2001). How to design a questionnaire. Student BMJ, 9, 171-216.
  5. Pollograph (2008). Designing a survey.
  6. StatPac (c. 2007). Questionnaire design considerations.
Book chapters
  1. Fowler, F. J., Jr. (2002). Designing questions to be good measures. In In F. J. Fowler, Survey research methods (3rd ed.) (pp. 76-103). Thousand Oaks, CA: Sage.
    • Google Books (Note: pp. 80-81, 87-89, 91-92, 95, 97, 100-102 are missing)
    • html (earlier version, full text)
    • Or contact the instructor for a full copy
  2. Nardi, P. (2006). Developing a questionnaire (Ch 4). In Doing survey research: A guide to quantitative methods (2nd ed.). Boston, MA: Pearson. eReserve
Books
You should be able to find some books about survey research and survey design in your university library.
  1. Alreck, P. L., & Settle, R. B. (2004). The survey research handbook (3rd ed.). Boston: McGraw-Hill/Irwin.
  2. Backstrom, C. H., & Hursh-César, G. (1981). Survey research (2nd ed.). New York: Wiley.
  3. De Vaus, D. (2002). Surveys in social research (5th ed.) London: UCL Press.
  4. Dillman, D. A. (2007). Mail and internet surveys: The tailored design method 2007 Update with new internet, visual, and mixed-mode guide (2nd ed.). Hoboken, NJ: Wiley.
  5. Frazer, L., & Lawley, M. (2000). Questionnaire design & administration: A practical guide. Brisbane, Australia: John Wiley.

See also[edit]

Wikiversity
Wikibooks
Wikipedia

References[edit]

  1. Jenkins, C. R., & Dillman, D. A. (1995). Towards a theory of self-administered questionnaire design.
  1. Spector, P. E. (1994). Using self-report questionnaires in research: A common on the use of a controversial method. Journal of Organizational Behavior, 15, 385-392.

External links[edit]