9

I am a big proponent of reproducible data analysis. In particular, I like when researchers share documents in formats like Sweave and knitr which weave statistical output (e.g., text, tables, and graphs) into a statistical report.

I previously asked for examples of reproducible research of any kind on Stats.SE and for advocating articles in psychology. However, at present I'm particularly interested in complete examples of reproducible meta-analysis.

Meta-analyses involve a number of steps. Summary statistics and study information is extracted from source studies. Various transformations and steps are applied to the data (e.g., corrections for reliability, conversion from one statistic to another, etc.). Various models are tested; tables and graphs are produced. Some journals are requiring that researchers supply tables of data used (e.g., references used, summary statistics). However, I can see meta-analysis as an area which could benefit from a more comprehensive reproducible approach: (1) it would permit greater inspection of specific methods used; (2) researchers could more easily build on the analyses used.

Thus, my questions:

  • Are there any complete examples of reproducible meta-analysis preferably in psychology or a related discipline?
  • Is there any published advocacy for reproducible meta-analysis?
Community
  • 1
Jeromy Anglim
  • 30,741
  • 11
  • 93
  • 221

2 Answers2

2

Update (2023): A lot has happened in 10 years. I have published a couple of open meta-analyses myself. Matthew Jané has put together the more comprehensive compilation of open science meta-analyses (which include links to mine and many others): https://matthewbjane.github.io/opensynthesis/

Old Answer (2014): I have started to see a fair bit of discussion about reproducible meta-analysis.

Tim Churches seems to have a github repository with a few examples of meta-analyses in R.

See in particular the public health example:

Jeromy Anglim
  • 30,741
  • 11
  • 93
  • 221
0

In reference to your first question, not a lot of published standards, I suggest some standard guideline on my site, but that's just that, suggestions. You might want to check out the Cochrane Collaboration though they have produced a set of standards for use in health care related meta-analysis.

You also may want to check an early-ish study by Shadish et al (1997) as an example:

Shadish, William, R., Matt, George, E., Navarro, Ana, M., Single, Gregory, Crits-Christoph, Paul, Hazelrigg, Anthony, J., Lyons, Larry C., et al., (1997). Evidence that therapy works in clinically representative conditions. Journal of Consulting and Clinical Psychology, 65, 355-65.

Seanny123
  • 8,853
  • 3
  • 25
  • 61
Larry C. Lyons
  • 121
  • 3
  • Thanks. Your answer speaks to the actual method of doing meta-analysis, as opposed to sharing the sharing of underlying data or scripts that do various steps (e.g., transformations of original effect sizes; combining of effect sizes within studies, correction formulas, etc.). Any thoughts on the reproducibility aspect? – Jeromy Anglim Feb 09 '18 at 01:46
  • Given the same data set and using the same conversion routines etc., you should get the same results. One potential issue is with rater reliability - variation due to those scoring the studies having slightly differing interpretations of the study factors. You should always report inter rater reliability. Alternatively where a discrepancy is discovered the project lead should go over the discrepancies with both raters and resolve them on a case by case basis, and note that in the study report. – Larry C. Lyons Feb 09 '18 at 20:36
  • As for sharing the study data that is a given. There are a number of sites on the web that you can use, such as my personal favourite Github. You should also publish as appendicies the list of studies used the meta-analysis, and your study rating questionnaire.

    As for transformations those are pretty standard. Any good stats textbook will show you those. For the web based application I wrote, the conversion javaScripts are freely available.

     – Larry C. Lyons Feb 09 '18 at 20:40
  • These script are based on the formulas found in the following

    Abramowitz, M. and Stegun, I. (1964). Handbook of mathematical functions with formulas, graphs, and mathematical tables. Washington, DC: United States Government Printing Office.

    Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd Ed). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Hunter, J. and Schmidt, F. (2015). Methods of meta-analysis: Correcting error and bias in research findings. 3fd Ed. Beverly Hills CA: Sage.

     – Larry C. Lyons Feb 09 '18 at 20:40
  • The Cochrane Collaboration also details its methods and procedures and again their conversion and accumulation methods are publically available.

    Here are a couple of other links to some of my work, first a web based paper that explains the methodology and conversion techniques used, http://www.lyonsmorris.com/MetaA/index.htm

    and the application based on the paper. http://www.lyonsmorris.com/ma1/index.cfm

    and the javascript used to do the conversions from study statistics to a common metric http://www.lyonsmorris.com/ma1/js/ESConversions.js

    hth larry

     – Larry C. Lyons Feb 09 '18 at 20:49