Meta-analysis is a systematic technique for reviewing, analysing, and summarising quantitative research studies on specific topics or questions.
This page provides information and resources about how to conduct a meta-analysis. The target audience includes post-graduate students conducting a meta-analysis or beginning researchers in meta-analysis. These pages could also be used by students involved in research methods coursework.
Introduction[edit | edit source]
- A meta-analysis is quantitative technique for conducting a "study of studies".
- Use of meta-analysis has flourished, particularly in the social, health, and medical sciences, since it was developed in the 1970s
- Meta-analysis was initially developed in response to controversy over traditional, subjective literature review methods (specifically, at the time, those used in to review the psychotherapy outcome studies).
Lecture slides[edit | edit source]
- Practical meta-analysis (Lecture slides; Wilson, 1999)
- How to do meta-analysis (Lecture slides; Basu, 2005)
- Meta-analysis: combining information(Lecture Slides; LeBauer, 2010) — includes outline of three approaches to inferences: moment matching, maximum likelihoo, and Bayesian.
How to do a meta-analysis[edit | edit source]
- Meta-analysis involves analysing the summary data from many studies. It can be performed by hand, using a spreadsheet and formulae, using scripts, syntax or macros with generic statistics software packages, or by using dedicated meta-analysis software packages.
- Before starting, identify a clear question(s), e.g., "What are the outcomes of psychotherapy?"
- Questions can also involve the effect of independent variables, e.g., "Are the outcomes of psychotherapy similar for males and females?"
- Read other related meta-analyses to get a feel for the kinds of questions asked.
- Make sure that any independent variables (IVs) and dependent variables (DVs) are very clearly defined.
- Because of the importance of establishing a well-defined question and variables, developing a peer-reviewed proposal for a meta-analytic study is strongly recommended.
- It can be helpful to identify several similar or related meta-analytic studies as models for your meta-analytic study. Consider the strengths and weaknesses of their methodologies.
- Establish clear criteria for selection of studies, e.g., does it need to be published in a peer-reviewed journal, or will you also accept theses and non-peer reviewed papers (e.g., conference papers)?
- Conduct an exhaustive and systematic literature search, recording your steps along the way (important for the Method - must allow replication)
- Create a "coding sheet" - this is the list of fields (variables) you want to extract from each study, and how each of the variables are to be coded - get this peer-reviewed, otherwise you will limit the potential/quality of your analyses
- Enter the data - one study per row, but note that there may be multiple outcomes and/or groups of interest for each study, in which case each of these will receive their own row in the database, with a column to code which type of outcome was measured.
- Analyse the data using spreadsheet formulae, or by writing syntax commands for a generic statistics package, or by using a dedicated meta-analysis software package (with in-built meta-analysis tools).
Effect sizes[edit | edit source]
- Central to understanding meta-analysis is an understanding of effect sizes.
- The chief value of effect sizes in the context of meta-analysis is that they provide a way to standardise effects across studies using different measures, allowing for common analysis.
- There are many possible effect sizes, but essentially there are two commonly reported types in meta-analysis:
- Correlational: e.g., r (product-moment correlation)
- Mean differences: e.g., Cohen's d, Hedge's g, etc.
Limitations[edit | edit source]
- An important limitation of meta-analysis is that its results can only be as good as the original data is valid.
- Meta-analysis can only analyse the role of independent variables in explaining variance in dependent variables if sufficient data is provided in the original studies.
- "Apples and oranges" effect - i.e., there is a risk/tendency in meta-analysis to average/mash together disparate effects.
- Can lack in qualitative insight (e.g., as may be more likely to be contributed by an expert conducting a traditional literature review).
Example meta-analytic studies[edit | edit source]
- Hattie, J., Biggs, J., Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66, 99-136.
- Hattie, J., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and Outward Bound: Out-of-class experiences that make a lasting difference. Review of Educational Research, 67, 43-87.
- Purdie, N., Hattie, J., Carroll, A. (2002). A review of the research on interventions for attention deficit hyperactivity disorder: What works best? Review of Educational Research, 77, 61-99.
Software[edit | edit source]
Comparison table[edit | edit source]
Some dedicated meta-analysis software includes:
|Name||URL||License||$Cost||Trial or Demo?||Version||Notes|
|MetaXL||http://www.epigear.com||Shareware (no commercial redistribution, no derivatives)||Free||No||3.0||Can run bias adjusted (quality effects) meta-analyses and the IVhet model (alternative to the RE model) in addition to conventional (IV, RE, MH, Peto) models. This software creates forest and funnel plots and in addition has implemented the Doi plot and LFK index which are new tools for publication bias assessment.|
|MIX||http://www.meta-analysis-made-easy.com||Proprietary||0-210||Yes||2.0||Student and academic licenses available|
|Cochrane's Review Manager (RevMan)||http://community.cochrane.org/tools/review-production-tools/revman-5/about#http://www.cc-ims.net/RevMan||Proprietary||Free for academic use, 730 GBP with discount for 4+ copies for commercial use.||Yes||5||For organising reviews; for MA, see |
|Meta-Analysis Calculator||http://www.lyonsmorris.com/lyons/maCalc/index.cfm||CC0||0||N/A||3.5||The Meta-Analysis Calculator is a web based application that conducts a Hunter-Schmidt (2004) type Meta-Analysis correcting for variation due to sampling and reliability. A future edition will allow for range restriction corrections. The results and the effect-size data set can be exported in Excel format and downloaded to the user's system.|
Alternative (software[edit | edit source]
Non-dedicated, generic statistics software which can be used for conducting meta-analysis include:
Other comparisons/lists[edit | edit source]
Tasks[edit | edit source]
References[edit | edit source]
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Sage. Thousand Oaks, CA.
See also[edit | edit source]
[edit | edit source]
- Practical meta-analysis effect size calculator (Wilson)
- Meta-analysis (psychwiki.com)
- Meta-analysis research methodology (Wilderdom)
- Meta-analysis stuff (Wilson)
- Meta-Analysis:Methods of Accumulating Results Across Research Domains (Online Paper)
MetaXL is free, but copyrighted software. MetaXL may be installed on as many computers as the user owns or employs, and can be freely distributed to third parties. However, modification of the installation software and distribution of the software to third parties for a fee are strictly prohibited. Please contact Epigear (firstname.lastname@example.org) if you have any questions about this license arrangement.