1 Basics. What is meta-analysis? Descriptions of common goals, its relation to a literature review, and introduction to data collection and analysis. We will also explore using Microsoft Excel to store data for upload to R for data analysis, for example, using the McLeod data. Also a simple example of running metafor using the built-in dataset mcdaniel1994.dat.
2 Preliminary Calculations and Statistics. All the studies must share a common metric for effect size before we can meta-analyze. We will discuss effect size definitions, how to calculate effect sizes from different bits of information the authors may provide, how to convert from one effect size to another. We will also anticipate running meta-analyses by computing a mean effect size with metafor. Then we will compute some empirical sampling distributions for r and for d using R. Install & run metafor script file. Correlation Monte Carlo. SMD Monte Carlo.
3 Data Collection and Weighted Means. In the first half, we discuss ways to collect data for a meta-analysis. Search process document. Search record document in Excel. In the seond half, we describe the logic of applying weights for combining effect sizes to get an overall summary estimate. As an example, we examine the Schmidt-Hunter 'bare bones' method of computing an overall effect size for correlations.
4 Fixed-effects and Random-effects. In the first half, we discuss Hedges' computational methods for finding the mean. We introduce the random-effects variance component and its effect on weights for the effect sizes. In the second half, we discuss the conceptual and computational difference between fixed and random effects in meta-analysis.
5 Heterogeneity. This module introduces the heterogenity concepts including the test for homogeneity, tau, and I-squared. We also consider the impact of heterogeneity on the confidence interval and the prediction interval. Sheet for CI and PI. Sheet for correlations.
6 Moderators. In this module, we will consider moderators, both categorical and continuous, and models with multiple independent variables. The basic idea is weighted regression. Example calculations for categorical moderator. SAT ANOVA.
7 Dependencies Meta-analysis programs (including metafor) typically assume that each effect size is statistically independent of the other effect sizes; otherwise there are dependencies in the data. Sometimes this is not the case (e.g. the dependent variable is measured twice on the same peopele. How should we handle this? Example sheet. Exercise sheet.
8 Graphs. In this module, we will consider graphs that a popular in meta-analysis, including the funnel plot and the forest plot. In the second half, we will consider vote-counting as a precursor to meta-analysis. R_code. R_data.
9 Availability Bias and Sensitivity Analysis. This module describes the forest plot sorted by effect size precision, the funnel plot assymetry test (trim and fill was described under plots), identifying and removing outliers, leave-one-out analysis.
10 Psychometric Meta-analysis. This method, also known as the Hunter-Schmidt method, is commonly used in industrial and organizational psychology. We will review the bare-bones calcuations, apply metafor to calculate results that are approximately (but not exactly) what you would get with this method, and look at the method when corrections for reliability, range restriction, or both are applied. If you want to use their method exactly, I provide an Excel sheeet you can use.
11 Rater Reliability12 When you collect data, you need some measure of the reliability of the coding, even for obvious things like sample size.
12 Criticicms and Best Practice. Although the volume and tone of criticsms of meta-analysis has softened over the years, there can be problems with inferences made from meta-analyses. We will examine criticims and also current views on best practice that are helpful in making good inferences and getting published.
13 Methods Research Meta-analysis is still fairly new as a statistical technique, and methods are still evolving. We will examine a small sample of current research on methods.