Survey research and design in psychology/Assessment

From Wikiversity
Jump to navigation Jump to search

Assessment

Overview[edit]

  1. There are 9 online quizzes (5% each; total 45%), a data collection and entry exercise (10%) and a lab report (45%).
  2. All assessment is optional. Non-completed assessments will be awarded 0.
  3. A final overall mark of 50% or higher is required to Pass the unit. Partial final marks will be rounded up.
  4. Summary of assessment items:
Item Weight Description Generic skill Learning outcome Due date Late submissions
Quizzes 45% 9 x online quizzes worth 5% each Problem solving Survey research and design; interpret correlation-based statistics, including EFA, reliability, and MLR. Mon 09:00 AM of Week 14 Not accepted

Data collection & entry

10%

Collect and enter data for 5 cases Analysis and inquiry Conduct survey-based research in psychology. Mon 09:00 AM of Week 4 (data file) / Mon 09:00 of Week 6 (hard copies) Not accepted

Lab report

45%

APA style lab report with EFA and MLR Analysis and inquiry Use SPSS; Interpret correlation-based statistics. Communicate in writing the results of survey-based psychological research. Mon 09:00 AM of Week 13 Accepted up to 7 days late with -5% per day penalty

Quizzes[edit]

  1. Topics: There are 9 online quizzes, each worth 5% (total 45%), with the following topics:
    1. Survey research
    2. Survey design
    3. Descriptives and graphing
    4. Correlation
    5. Exploratory factor analysis
    6. Psychometrics
    7. MLR 1
    8. MLR 2
    9. Power and effect size
  2. Availability:
    1. Quizzes will be available until Mon 09:00 Week 14.
    2. UCLearn quizzes are only available to enrolled University of Canberra students.
  3. Late submissions: Not accepted
  4. Assessed knowledge: Quizzes assess knowledge of concepts covered in lectures, tutorials, and readings.
  5. Quiz content:
    1. Each quiz consists of 10 multiple choice questions drawn randomly from a larger test bank.
    2. The questions from each quiz correspond directly to a single lecture and its related readings.
    3. Some quizzes are harder than others (e.g., the lowest mean is typically for Quiz 05: Exploratory factor analysis).
  6. Question style: Some questions have more than one correct answer.
    1. In order to get full marks, all correct the answers must be selected and no incorrect answers must be selected. For example, if a question has 5 answer choices, 2 of which are correct (each correct answer would be worth 50%, adding to +100%) and 3 of which are incorrect (each incorrect answer would be worth -33% each, adding to -100%). The mark awarded is the total of the marks for each of the answer options selected. However, no negative total marks will be awarded for a question, so the lowest mark that can be awarded for a question is 0.
    2. The reason for including negative marks for incorrect answers for multiple response question types is that, otherwise, all answer options could be selected to achieve a full mark, even though some incorrect answers were selected. So, there needs to be some penalty for selecting incorrect options.
  7. Time limits: Quiz time limits vary between 10 and 20 minutes, depending on the complexity of the questions.
  8. Academic integrity: Quizzes are to be completed independently (i.e., without assistance of others), in your own time.
  9. Open book: Quizzes are open book - recommended resource materials include lecture, tutorial, and reading notes.
  10. Number of attempts: Each quiz can only be attempted once.
  11. Feedback: Answers to the quiz questions are shown immediately after quiz submission, but are no longer accessible once the feedback window is closed. Take any notes for study purposes prior to closing the feedback screen.
  12. Bonus marks: A bonus quiz mark is awarded for identifying an error or significant improvement to a quiz question - email details to the unit convener. For example, a student who got 7/10 on Quiz 1, emailed the unit convener to correctly point out a spelling error in that quiz, resulting in a bonus mark being added and the student's new Quiz 1 mark becoming 8/10.
  13. Practice quizzes: Some practice quizzes are available:
    1. Quiz 0 (a practice quiz) can be attempted as many times as you like. Use this quiz to test and make sure that you are familiar with how the quiz system works on the computer you are using.
    2. Several practice quizzes are available on Wikiversity, including:
      1. Research types
      2. Research purposes
      3. Correlation
      4. Exploratory factor analysis
      5. Psychometric instrument development
      6. Multiple linear regression
    3. Pearson - see External link sections on the lecture pages

Data collection and entry[edit]

This assessment exercise provides hands-on involvement in real world survey data collection and data entry. It requires collecting data using a survey, entering data, submitting an electronic data file, and submitting the hard copy surveys in an accurate, timely fashion as part of a larger class project. The specific steps are to:

  1. Collect 5 cases of survey data by following the survey administration guidelines.
  2. Enter the survey data by following the data entry guidelines.
  3. Submit:
    1. Electronic data file via UCLearn by Monday 09:00 AM of Week 4 - no late submissions accepted
    2. Hard copy surveys as a single bundle with a coversheet by Monday 09:00 AM of Week 6 - no late submissions accepted - see hard copy survey guidelines

Important note: In order to earn any marks, it is necessary to submit both electronic and hard copy data on time. No marks will be provided if either the electronic or hard copy submissions are missing or late.

Lab report[edit]

Overview[edit]

  1. Present an independently-developed APA style lab report which examines:
    1. the psychometric properties of a multi-item, multi-factorial measure (either time management or time perspective) using exploratory factor analysis and reliability analysis (guided by a research question)
    2. an explanatory model involving at least three predictors and one dependent variable using multiple linear regression (guided by hypotheses - one per predictor).
  2. Use the class-aggregated data file from the data collection and entry assessment exercise, based on the .
  3. For further information, see the general marking criteria and the detailed marking criteria which explains what to include for each section.

Data download, screening, & recoding[edit]

  1. Data download: TSQFUS1_2018_All_Unscreened
  2. Data screening
  3. Data recoding

Marking process[edit]

  1. Lab reports will be evaluated according to the general and detailed marking criteria.
  2. For reports submitted by the original due date, marks and feedback should be available three weeks of submission. Marking of reports submitted late or with extension may take longer than three weeks from the date of submission.
  3. Availability of marks and feedback will be notified via UCLearn.

General marking criteria[edit]

Within each of the lab report sections, marking will reflect these general criteria:

  1. 70%: Quality of execution of the task
    (e.g., review literature and develop hypotheses, describe method, summarise results, discuss findings)
  2. 10%: Quality of written expression
    (structure and headings, flow, sentence and paragraph structure, spelling, and grammar)
  3. 10%: Contribution to a cohesive, meaningful report
    (i.e., a story that makes sense - with sections that are not disjointed)
  4. 10%: APA style

Detailed marking criteria[edit]

These detailed guidelines describe the requirements for each section of the lab report:

Overview of sections[edit]

This is an overview of the requirements and weighting for each section:

Criteria Description % Suggested word count
Title/Abstract Succinct, specific title. Abstract covers purpose, method, findings, and conclusions. 5% 10-15, 160-200
Introduction Establishes the problem, reviews theory and research, develops research question(s) and hypotheses 10% 400-700
Method Describes method and design, including Participants, Materials, and Procedure 15% 500-700
Results Screens (5%) and analyses data using EFA, internal consistency, descriptive statistics and correlations for the composite scores (20%), and MLR (20%) 45% 900-1500
Discussion Summarises and interprets the findings, considers implications, and makes recommendations 25% 500-750

The word count ranges per section are suggestions only. The only restriction is an upper limit on the overall word count.

Cover sheet[edit]

  1. p. 0
  2. Download via Lab report cover sheet - click File - Download as - Select file type

Running head[edit]

  1. APA style

Title page[edit]

  1. p. 1 (note that the running head (short title and p#) should appear differently on p. 1 than on subsequent pages)
  2. Follow APA style (i.e., title, author, institution) except:
    1. use student ID number instead of name (for blind marking)
    2. do not include author note or non-APA style material such as: Word count, Unit name, Date, Tutor name etc.
  3. Title
    1. ~10-15 words
    2. Does the title convey the content or purpose of the report? Should reflect the psychometric and hypothesis-testing aims of the study.
    3. Should be succinct, yet specific to the study (e.g., mentions key constructs)?
    4. Catchy? Memorable?
    5. APA style notes: Capitalise the first letters of the title; no full-stop
  4. Institution name

Abstract[edit]

  1. p. 2
  2. ~160-200 words
  3. Abstract heading (APA style - should not be in bold)
  4. Summarise (avoiding excessive detail)
    1. Purpose of the study
    2. Method, including the sample (after data screening) and sampling method
    3. Key findings, including:
    4.  % of variance explained in the EFA and identified factors
    5. R2 for the MLR and the relative contribution and meaning (consider strength, direction, and significance) of each predictor
    6. Conclusions about theory and method, with key recommendation(s)

Key words[edit]

  1. APA style
  2. 3 to 5 terms

Introduction[edit]

  1. pp. 3-
  2. Introduce the topic and concisely explain the study's purpose(s).
  3. Provide a critical overview of relevant past research and identify key issues to be addressed in this study.
  4. Only review constructs which are analysed in the Results - e.g., see possible topics.
  5. Use citations to key background literature - Wikiversity (see readings) lists some starting references, but use of additional references is recommended.
  6. Avoid describing methodological details about the current study, such as measurement tools - this belongs in the Method.
  7. Provide logically-derived and clearly-stated research question(s) and/or hypotheses (can be null and/or alternative). The derivation of the questions and hypotheses should be supported by theoretical argument and citations.
    1. Firstly, specify a research question about the underlying factor structure of one of the multidimensional constructs in the (i.e., either time perspective or time management) e.g., "How many distinct dimensions (factors) of X are there, what are they, and which items best represent these factors?"
    2. Secondly, make a hypothesis about the extent to which each of the independent variables (IVs) predict a dependent variable (DV) (→ Multiple linear regression (MLR)) (choose any suitable variables, or composite of several variables, from the ) data file e.g., "It is hypothesised that [IV1], [IV2], and [IV3] will each positively predict [DV]".

Method[edit]

  1. Clearly explain how the study was conducted in sufficient detail to allow a replication study, but without extraneous detail.
  2. Key marking criteria: Is the study replicable? Is sufficient detail provided for a "naive person" (say, someone in Japan in 20 years time) to be able to fully replicate the study?

Participants (5%)

  1. Provide a one to two paragraph descriptive overview of the participants in the final (after data screening) sample.
  2. Consider which of the available data can be summarised in order to provide an insightful description.
  3. Advanced option: You may wish to compare the sampled data with population statistics for UC students (e.g., see UC at a glance and Annual Reports).

Measures (5%)

  1. Briefly summarise the development of the survey instrumentation.
  2. For the measures used to collect the data which is analysed in the Results, describe e.g.
    1. type of questions
    2. response format
    3. any reverse-scoring and meaning/direction of high/low scores
    4. Do not describe measures which are not used in the current study.
    5. Optional: Use a table to help present the proposed factors (e.g., label, definitions and example items)

Procedure (5%)

  1. Sampling:
    1. What was the target population and the sampling frame?
    2. What sampling technique was used?
  2. Administration:
    1. Briefly summarise and provide an APA style reference for the Survey administration guidelines
    2. Where and how did you collect data?
    3. How long did participants typically take to complete the survey?
    4. Refusal rate? (for the surveys you administered)
    5. Procedural anomalies? (e.g., explain any unanticipated responses or unplanned occurrences)

Results[edit]

  1. The analysis should proceed through three basic steps:
    1. Data screening - Summarise in one to two paragraphs how the data was screened and what changes were made. Is enough detail provided for the same steps to be followed by someone else? However, avoid excessive detail (e.g, CaseID number are meaningless to a reader).
    2. Sample size assumptions do not belong in this section - instead, this would belong in the section(s) for the corresponding analyses.
    3. Psychometric instrument development
      1. Conduct EFA of a multidimensional construct (either the time perspective items or the time management items).
      2. For each extracted factor, provide reliability analysis (internal consistency - Cronbach's alpha), composite score descriptive statistics, and correlations between factors.
    4. Multiple linear regression - Conduct a MLR with at least three IVs to address one hypothesis per IV
  2. Communicate the depth of your understanding by using your own words; avoid writing results in a robotic (mindless) manner (e.g., avoid overparaphrasing a specific sample write-up)
  3. Most statistics should be rounded to two decimal places unless there is particularly useful information communicated by including a third decimal place (e.g., when reporting exact p values).
  4. Scope and depth of analysis
    1. Additional analyses may be presented. However, it is quite possible to gain maximum marks by conducting one of each of the required analyses. If additional analyses are presented, then they must be clearly related to the research question and hypothesis(es).
    2. In marking, some account will be taken of the scope of the analysis undertaken. Where a more advanced analysis is appropriate (given the research questions(s) and/or hypothesis(es)) and is well conducted, this could represent higher quality work than a simpler analysis. However, there's much also to be said for parsimony (keep it simple and get it right) by focusing on doing a good job of fulfilling the minimum requirements. The best reports are usually not the most complex ones. If in doubt, go with analyses which meet the minimum criteria, which relate to the research question and/or hypotheses, and which you are confident about accurately conducting, interpreting and presenting.

Data screening (5%)

  1. Summarise what was done to check and correct errors in the data - see data screening
  2. Statements like ""All analyses were conducted using SPSS version 23." are unnecessary.

Psychometric instrument development (20%)

  1. Report the results of an EFA of a multidimensional survey construct (either time perspective or time management)
    1. The minimum requirement is to report psychometric analysis (EFA, composite scores, correlations between composite scores, and internal consistency) for one set of items (time perspective or time management). However, it may also be of interest to conduct psychometric analysis of another set of items (e.g., stress) in order to develop other composite scores for further analysis. In this case, present one EFA (of either time perspective or time management) in full and briefly summarise the results of the other EFA, perhaps with relevant output in an appendix.
  2. Indicate the type of EFA used (Type of extraction? Type of rotation?)
  3. Explain the extent to which EFA assumptions were met, but not excessively (e.g., one indicator of factorability is quite sufficient; more is redundant)
    1. Sample size (incl. cases:variables ratio)
    2. Linearity (e.g., check at least some scatterplots, particularly for bivariate outliers or non-linear relations)
    3. Factorability of the correlation matrix (either examine via item correlations, anti-correlation matrix diagonals, or a measure of sampling adequacy (KMO or Bartlett's) - but do not report all of these as they are redundant)
  4. Focus on the final model but summarise the steps taken to get there (e.g., How many factors were extracted initially? What models/factors structures were examined? To what extent was the expected structure evident?)
  5.  % of variance explained (for the initial and final model(s))
  6. Label and describe each factor
  7. Which items were retained and/or dropped and the reasons why
  8. Table of factor loadings (sorted by size) and communalities (for the final model)
  9. Reliability analysis (Internal consistency - (Cronbach's alpha)) for each factor
  10. Calculation of composite scores to represent each factor
  11. Table of descriptive statistics for the composite scores
  12. Table of correlations between composite scores

Multiple linear regression (20%)

  1. Report the results of a MLR with at least three predictors - can use any variables in, or derived from, the supplied data set (if they meet the assumptions for MLR)
  2. Reiterate the purpose (research question and/or hypotheses) of the MLR
  3. Mention the type of MLR (e.g., standard, hierarchical, or stepwise)
  4. Describe the IVs and DVs, and any manipulations of the variables (e.g., recoding or creating an interaction term). If not already clear from the Method, clarify the direction of scoring.
  5. Explain the extent to which assumptions were met (e.g., sample size, multicollinearity, multivariate outliers)
  6. Present the correlations between the items (can be part of the MLR coefficients table - see sample write-ups for examples). Demonstrate understanding of the directions of any relationships (e.g., if there is a positive correlation between X and Gender, what does this mean? Are higher values of X associated with males or females?)
  7. Report amount of variance explained (R2 and Adjusted R2 (and the R2 change at each step if a hierarchical MLR is being conducted), along with inferential tests (F(df), p)
  8. Report significance, size, direction and relative contribution of each IV. Make sure to explain what the direction of the relationships mean in plain English.
  9. Table showing the correlations and MLR coefficients, including B for intercept & IVs and Beta (β), and the statistical significance (e.g., t, p), and semi-partial correlations squared (sr2) for each IV and explain the direction and size of the results.
  10. Consider the shared and unique percentages of explained variance.

Discussion[edit]

  1. Build on the introduction to explain the results and what they mean in a balanced manner.
  2. Demonstrate breadth and depth of understanding of the results and their implications. Avoid merely summarising the results without providing additional critical commentary.
  3. Critically review the strengths and weaknesses of the study's methodology and make practical suggestions for how it could be improved, e.g.,
    1. Validity and reliability of the measures?
    2. Statistical power?
    3. Appropriateness of the sampling technique?
    4. Generalisability of the findings?
  4. Provide tangible recommendations for future research and practical implications.

References[edit]

  1. Is the reference list complete (i.e., none missing and all cited)?
  2. Does the lab report make effective use of a core set of relevant, high-quality, peer-reviewed, citations? This involves citing and meaningfully discussing appropriate references (as opposed to just dumping citations without explanation), particularly in the Introduction and Discussion.
  3. Reference the instrumentation and the survey administration guidelines - do not copy the guidelines into the Appendices.
  4. Use APA style, including for electronic sources. Include DOIs where relevant.

Appendices[edit]

  1. Appendices are not needed - they are optional.
  2. Appendices are for additional detail which is relevant to understanding the main body, but which would break the flow of the main report e.g., the correlations between the items used in the factor analysis.
  3. Appendix content does not need to follow APA style but should be well organised, with clear labeling.
  4. Each appendix should be referred to at least once in the main body.

Style[edit]

  1. Follow APA style except:
    1. add a cover sheet
    2. use single-spacing (for electronic submission; double-spacing is a relic from paper submission days - it allowed room for hand-written comments)
    3. do not include author name on the title page (to facilitate blind marking)
    4. include any tables or figures in situ (i.e., embed them without the main body of the report rather than putting them after the references)
    5. formatting of any appendices does not need to follow APA style
  2. Example (owl.english.purdue.edu)
  3. The most important aspects of APA style for this lab report are:
    1. Times New Roman 12 pt font
    2. Running head with page numbers
    3. Page margins
    4. Left-justify body text
    5. Captioning and layout of tables
    6. Right alignment of statistics in tables
    7. Citations and referencing (including for electronic references and use of DOIs)
    8. Symbols, abbreviations, and formatting of statistical symbols (including M, SD, skew, kurt, N, n, F, t, p, r, R, R2, sr2)

Word count[edit]

  1. There is no minimum word count.
  2. Maximum word count: 2800 words + 10%
  3. Count everything using a word processor from the beginning of the Introduction (i.e., not Title page, Abstract etc.) to end of Discussion, including all text, headings, footnotes, citations, tables and figures etcetera (i.e., but not the sections after the Discussion, such as References and Appendices).
  4. Penalty for exceeding maximum word count: Markers will ignore words beyond the maximum (i.e., most likely resulting in a reduced mark for the Discussion).
  5. Word counts provided in the section overview are suggestions only.

Independent writing and plagiarism[edit]

It is necessary to demonstrate independent thinking and writing in order to satisfy the lab report learning outcomes. In other words, avoid plagiarising from these samples or other guides. Some sample write-ups are provided to give ideas. Also seek out other examples (e.g., see the readings). Then write up your results in your own words.

If you are in some doubt about whether you've demonstrated sufficiently independent writing, then upload the assignment to get a text-matching analysis via Urkund. Text-matching for this assignment between about 10% and 30% is reasonably common and probably indicates no particular cause for concern. However, it is recommended that drafts with text-matching scores above approximately 30% should be reviewed and rewritten in order to more effectively demonstrate independent understanding and writing.

Sample write-ups[edit]

Attention
  • Sample write-ups are provided as guides. However, it is strongly recommended not to overly rely on any single write-up. Try to look across these, and other, examples (e.g., journal articles).
  • Reports need to be independently written in order to demonstrate your achievement of the learning outcomes. If a submitted report largely replicates the wording an example write-up, it is arguably plagiarism and is much less impressive than a report which is independently written.
  • Bear in mind that none of the examples are perfect (e.g., see "red ink" and feedback comments, although not everything that can be improved has been commented on).
  • Finally, note that the requirements of the exercise are slightly different each year, so ensure the report you submit addresses the current lab report guidelines and marking criteria.

Some previous High Distinction (HD) reports:

  1. Example 2007 HD lab report (.pdf) - Marking and feedback sheet (.xls)
  2. Example 2013 HD lab report (.pdf)
  3. Example 2016 HD lab report (with qualitative analysis for G students and feedback) (.pdf)

Sample write-ups for specific analyses. Note that these samples are longer than is likely to be able to be presented in the lab report within the word count limit.

  1. Exploratory factor analysis (.docx)
  2. MLR (.doc)
  3. Some of the readings also provide examples of APA style write-ups for specific analyses.

Submission[edit]

  1. Submit the lab report electronically via the unit's UCLearn site.

Late submissions[edit]

  1. Standard UC policy applies:
    1. 5% penalty per day (including weekends)
    2. no submissions accepted or marked after 7 days

Academic integrity[edit]

  1. Emerging academics have a responsibility to uphold University standards on ethical scholarship. Good scholarship involves building on the work of others and use of others’ work must be acknowledged with proper attribution made. Cheating, plagiarism, and falsification of data are dishonest practices that contravene academic values. The Academic Skills Centre provides opportunities to enhance student understanding of academic integrity.
  2. Participants are expected to submit independent work on the assessment items.
  3. Note the University of Canberra policy on plagiarism.

Extensions[edit]

  1. Participants are expected to work on the assessment items throughout semester. Extensions will only be granted in exceptional circumstances. Early communication of problems is strongly advised. Participants should assess within the first few weeks of semester whether they have a reasonable likelihood of being able to complete the unit and should consider withdrawing by the census date if not keeping up.
  2. Extensions will not be granted for:
    1. Workload (e.g., study load and/or paid or voluntary workload) - such problems should be anticipated and withdrawal from the unit is recommended if workload is a problem.
    2. Technical problems (e.g., lost/corrupted/damaged storage media, software/internet access problems, and viruses) - it is strongly recommended that you keep multiple and regular backups of lab report drafts, data, syntax, and output files.
    3. Undocumented issues.
  3. Extension requests should be submitted via email to the unit convener from the requester's UC student email address and include:
    1. First and last name
    2. Unit number and assessment item(s) an extension is requested for
    3. Length of extension request
    4. Reason for extension request
    5. Documentary evidence with correct page orientation - for example:
      1. A medical certificate (from a medical, dental, or health practitioner) which provides:
        1. Registered provider number, provider's contact details, qualifications, and signature
        2. Date(s) of consultation
        3. Duration of incapacity to study (must be during the current study period and prior to the due date)
      2. A death notice or other appropriate documentation for bereavement.
      3. A Reasonable Adjustment Plan from Inclusion and Welfare at the University of Canberra
  4. For more information:
    1. Unit outline
    2. UC Assessment Policy
    3. UC Assessment Procedures
    4. Student Grievance Resolution Policy