The Meta-Analyst’s Checklist

Intro

• Research domain - Define Constructs
• Research question(s) to be addressed
• Confirm popular opinion
• Controversy in the literature
• Problems with previous meta-analyses
• Theoretical justification of choice of moderators

Data Collection

Sources

• Conference proceedings/abstracts
• Dissertations
• Electronic databases searched; include terms, keywords
• PychInfo
• PsychArticles
• ScienceDirect
• ERIC
• Web of Science
• ABI Inform
• Medline
• Dissertation Abstracts
• Etc.
• Hand searches of literature – a must include because you get stuff you don’t get in electronic databases
• Listservs
• Other sources of documents (e.g., key researchers/informants)
• Reference sections of articles
• Unpublished manuscripts
• Websites of authors in the area

Eligibility: Inclusion/Exclusion Criteria

• Country
• Language

• Measures allowed
• Characteristics of the measures (e.g., minimum number of items, reliability, etc.)
• E.g., NEO vs. big five generic
• Specific measures (e.g,. facet vs. general satisfaction)

• Participants (e.g., all male)
• Publications (type included)
• Journal (e.g., APA journal)
• Book vs. article, etc.
• Published vs. not vs. subsequently published
• Study design (e.g., specific control groups, repeated measures, etc.)
• Time frame

• Required information
• Effect size
• Moderator
• Sample size
• Other (e.g., biological verification)

Coding Studies

• Choice of effect size
• Assumptions (e.g., if only p value reported) for derivation
• Transformations from one effect size to another
• Code-book description
• Consensus method
• Description of raters
• Rater training, N of raters for each judgment
• Rater Agreement/reliability
• Alpha/intraclass correlation
• Kappa
• Percent agreement
• Test-retest

Data Analysis

Focal variables

• Bias reduction (if needed; available for r and d)
• Choice of effect size
• Choice of Fixed-, Random- and Mixed-effects
• Homogeneity analysis (Q test)
• Imputed data (e.g., substitute mean for missing reliability)
• Outliers
• Weights
• Inverse variance
• Reliability
• Range Restriction
• Sample-size
• Unit?

Moderators

• Categorical
• Continuous
• Homogeneity analysis after modeling moderators
• Choice of models (fixed vs mixed)

Results

• (Weighted) Mean ES
• CI for mean

• CI for variance (?)

• Descriptive stats (k, Ns, Means, SDs, etc.)
• Descriptions of the measures (e.g., names, reliability estimates of raw variables)
• Optional - Failsafe N/ filedrawer – report choice of min for ES
• Graphic representation of ES distribution(s) [some choices]
• Boxplot
• Funnel plot
• Dot (forest) plot
• QQ plot
• Scatterplot
• Stem-leaf

• RE Variance Component
• Results of moderator analysis
• Between & within values of Q (homogeneity)
• Report means
• Report regression coefficients/data
• Table of description of the distributions by levels
• Sensitivity of results to choices, e.g., outliers
• Variance of ES

Threats to Validity

• Apples & Oranges
• Dependent Effect Sizes
• File drawer
• Methodological Rigor - GIGO
• Power/small k (small N)
• Unanswerable questions (e.g., correlational ESs, causal question)

Discussion

• Discussion of credible threats / study limits
• Future research e.g., other moderators, what are gaps based on what is known
• Implications consistent with data and analysis
• Meaning of results in terms of study (e.g., BESD)
• Proper interpretation of the effect size (d, r, OR, etc.)
• Results clearly linked to questions