(Sample) Size Matters

Sample Size Matters

On this blog and others, on twitter (@mwkraus), at conferences, and in the halls of the psychology building at the University of Illinois, I have engaged in a wealth of important discussions about improving research methods in social-personality psychology. Many prominent psychologists have offered several helpful suggestions in this regard (here, here, here, and here).

Among the many suggestions for building a better psychological science, perhaps the simplest and most parsimonious way to improve research methods is to increase sample sizes for all study designs: By increasing sample size researchers can detect smaller real effects and can more accurately measure large effects. There are many trade-offs in choosing appropriate research methods, but sample size, at least for a researcher like me who deals in relatively inexpensive data collection tools, is in many ways the most cost effective way to improve one's science. In essence, I can continue to design the studies I have been designing and ask the same research questions I have been asking (i.e., business-as-usual) with the one exception that each study I run has a larger N than it would have if I were not thinking (more) intelligently about statistical power.

How has my lab been fairing with respect to this goal of collecting large samples? See for yourself:

The data I report here is from all 67 studies I have published, in press, or under review at psychology journals. I collected sample size data for each of these studies and in the figure below, I organized mean sample sizes by publication/submission year. The correlation between year and sample size is (r = .40). What you see in the figure are fairly consistent mean sample sizes that hover between 120 to 155 from when I started publishing in empirical journals between 2009 & 2012. My sample size averages grew to n = 223 in 2013 and n = 375 in 2014.

What happened in 2013 that led to this sizable boost in sample size in my laboratory? For my family, 2013 was a year of huge changes (e.g., new baby, moved to a new state, started a new job). In my lab, we also made the decision to "go big or go home" when it comes to sample size in lab or online studies. We made this decision for several reasons:

(1)The University of Illinois subject pool is large and allows for the possibility to collect large laboratory samples as long as you're willing to wait a few extra weeks.

(2)I am a professor now and have what is known in biz as a start-up ($$$$) that allows me to pay for larger samples online with relative ease.

(3)The University of Illinois as a cultural group favors high-quality research methods--and stresses the importance of larger sample sizes. When the people who decide your promotion and tenure care about large N, you care too. 

(4)Social-Personality psychology has continued to focus on research methods and research integrity, in part, because our guild has been shaken by several high-profile instances of research fraud. 

(5)I started writing (currently unfunded) research grants and this forced me to think more deeply about power analyses and effect size.

(6) I'm on twitter and engage in regular arguments (usually respectfully) with people about research methods. These arguments have forced me to consider the issues more carefully than I otherwise would have.

(7) High profile journals like Psychological Science and PSPB have changed their submission guidelines to improve research methods, and in particular, ask researchers to describe their decision-making about sample sizes.

Together, these factors have provided the social context necessary to make collecting larger samples both a logical choice--in that large samples offer a better chance of obtaining findings that are real and replicable, and a self-benefiting one--in that my career outcomes are decided, in part, by improving my methods.

Another way to look at these methodological improvements is to examine the true effect size I can detect with the average sample size I collected in each year, with 80% power. I present effect sizes in terms of correlations because that's pretty easy to interpret. The red bar shows the average effect size in 100 years of social-personality psychology at r = .21 (Richard et al., 2003). Note, that the average sample size of the studies I published in 2010 was not large enough to detect even the average effect in social psychology (that's bad planning and a complete disregard of effect size and power). Compare that with 2013 & 2014, where my studies can now detect smaller correlations of .17 and .13 respectively.*

In 2014 I feel more confident about my research methods and the potential conclusions that can be drawn from the results reported in my studies. This confidence is almost entirely influenced by the methodological improvements I have made in my own research (at minimal cost to myself and my productivity) by simply collecting larger samples.

Social-personality psychologists help people understand their social lives in fundamental ways. As social-personality psychology looks ahead to the future, I hope we can build a better, more replicable science. We're doing it already in my lab by increasing N!


* In thinking about how large an N is enough, there isn't a one-size-fits-all answer unfortunately. First, larger N is always better, designs being equal, when the goal is estimating an effect with precision. Second, a more complicated design (4 vs. 2 experimental conditions) requires a larger N, and degrees of non-independence change power analyses considerably.