Behind the Scenes of Psychological Research: What Does the Future Hold?

Since starting this blog, I’ve told you about interesting and hopefully useful research findings. But today I wanted to take a step back and share with you a bit about what is going on behind the scenes of psychological research. We assume that findings we are telling you about—those which are published in peer-reviewed journals—are true, but it turns out that is not always the case. Recently, it has come to light that the way we conduct our studies may be leading us to find “significant results” more often than they truly exist. And even more extreme than problems with methodology is the harsh reality that some researchers may be simply making up their findings. So today, I wanted to share with you some of my personal experience behind the scenes of science, as well as three great suggestions I’ve heard for reforming our scientific ways.

As a child, I was constantly pestering my parents with questions. I wanted to know the meaning of life, where the universe ended, and what it meant to be happy. In college, I discovered that there was this thing called “research” which would allow me to systematically answer the questions that were constantly cropping up. And it turns out I could potentially make a living doing so. Sign me up! I eagerly started my PhD program with a bunch of other starry-eyed first years who knew they were also going to help make the world a better place through science. We took classes on interesting subjects and planned and executed research projects. And then reality started to sink in – we couldn’t just ask and answer the questions we were interested in, we had to ask the right questions (the questions that reviewers would think were important), and we had to answer them in the right way (with significant results). Disillusionment began to set – were we trying to discover the mysteries of the human condition and change the world, or were we trying to have the best looking CV? And this is the big question I have grappled with ever since. I do not believe that these two aims are mutually exclusive, but I do think that as things currently stand these two end goals produce different paths. I also believe that the system that is currently in place is evolving in a way that promotes the latter goal, rather than the former. This was not a system created through intelligent design, but a slow process that evolved and adapted over time in response to new technology, population growth, and changing social norms. But it turns out that we can have a hand in the future of our field. We don’t have to sit passively by and watch as good science goes out the window. Many good researchers are turning their attention momentarily away from questions about how to understand the human condition and instead asking questions about how to best study the human condition (see Michael’s attempt to do so with his personal p-curve analysis and just a few of the other great blog posts on it here, here, and here).

So how do we go about putting into place a system that promotes good science and true discovery as the end goal? Many good ideas have been thrown out there, and I applaud them all. Below I describe three that have really stuck out in my mind.

First, psychologist Brian Nosek made a revolutionary suggestion in a talk I attended last year – we need to do away with journals. Why? The simplest explanation comes from an Editor who recently rejected a paper I was a co-author on. In the rejection letter, we were told “Please understand that my decision was rendered with the recognition that the page limitations of the journal dictate that only a small percentage of submitted manuscripts can be accepted.  We receive more than 750 submissions per year, but only publish approximately 125 papers each year.  Papers without major flaws are often not accepted by [this journal] because the magnitude of the contribution is not sufficient to warrant publication.” That’s right, perfectly good papers that could help advance science are being rejected because of page limits. Well, if we started publishing everything online, as Nosek suggested, we would no longer have to deal with these space constraints. As long as the science is sound, a paper would be published. Online publications also mean instant access, rather than waiting for the quarterly issue to get printed and mailed out to subscribers. He also suggested that we don’t have separate online journals and instead have one site through which we submit all papers. No more trying to decide which journal is the best fit for your interdisciplinary work, and no more guessing how “good” your research is when trying to decide where to submit it, just submit to the one portal where it will be tagged and sent to the appropriate reviewers. If we need to differentiate the breadth and rigor of different papers, we can create a rating system. Other fields seem to be moving in this direction, and I think it is time we did too.

Second, just as the stars in my eyes were beginning to dim slightly in the early stages of graduate school, an older and wiser graduate student griped that our field really needed to reform our article submission process. Instead of submitting a paper in its entirety, we should just submit the introduction and methods. Reviewers would then judge the paper based on the soundness of the theory and the methods, rather than the significance of the results. This transition to a focus on methods would ensure that we conducted sound science rather than worrying about flashy results. And if we no longer need to worry about paper-based space constraints, there would be no concern about a paper with null results taking room from a paper with significant effects. This older and wiser graduate student rocked my world when she made this suggestion, but I’ve read blog posts by other researchers who make similar suggestions, so perhaps the time for this change is not so far off.

Third, social neuroscientist Matthew Lieberman suggested that we recognize the importance of replication, and instead of rejecting papers that attempt to replicate, encourage and institutionalize this process. First and second year graduate students would be trained in research methods by taking on one of the biggest studies of the previous couple of years and embarking on a replication project. If students all over the world took on this task, we could meta-analyze all of the results and within a few years we would have a good sense of which results were robust and which were a function of Type I error (or something more pernicious). This approach to replication would also help put the emphasis back on strong methodology.

Each of these suggestions offers a different solution for helping tackle the problems of conducting modern day science, but they all have one thing in common – they help get rid of the pressure to produce unbelievable results (in the aforementioned paper that was rejected, one of the criticisms was that the findings were not counterintuitive enough) and in doing so create a system in which there is little incentive to cheat and a lot of incentive to do good work. One frustration I’ve had is that input does not always equal output. We work hard running good studies, but if they aren’t novel enough (likely because someone else got their work in a little more quickly), or the results don’t turn out as we’d hoped, we may find ourselves with a lot put in and little to show for it. In a system where the focus is on rigorous methods, replicable results, and there is no need to limit publications for the sake of page constraints, then our output is going to be a much better reflection of what we put in.

The devil is of course in the detail, and I’m sure that there are many potential problems with each of these suggestions, but I do think they are a step in the right direction towards something new and great. And as someone who still loves asking questions and seeking answers, I look forward to being part of it.

Article on False Positives: Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant Psychological Science DOI: 10.1177/0956797611417632