As a child, I was constantly pestering my parents with questions.
I wanted to know the meaning of life, where the universe ended, and what it
meant to be happy. In college, I discovered that there was this thing called “research”
which would allow me to systematically answer the questions that were
constantly cropping up. And it turns out I could potentially make a living
doing so. Sign me up! I eagerly started my PhD program with a bunch of other
starry-eyed first years who knew they were also going to help make the world a
better place through science. We took classes on interesting subjects and planned
and executed research projects. And then reality started to sink in – we couldn’t
just ask and answer the questions we were interested in, we had to ask the right questions (the questions that
reviewers would think were important), and we had to answer them in the right way (with significant results).
Disillusionment began to set – were we trying to discover the mysteries of the
human condition and change the world, or were we trying to have the best
looking CV? And this is the big question I have grappled with ever since. I do
not believe that these two aims are mutually exclusive, but I do think that as
things currently stand these two end goals produce different paths. I also
believe that the system that is currently in place is evolving in a way that
promotes the latter goal, rather than the former. This was not a system created
through intelligent design, but a slow process that evolved and adapted over
time in response to new technology, population growth, and changing social
norms. But it turns out that we can have a hand in the future of our field. We
don’t have to sit passively by and watch as good science goes out the window. Many
good researchers are turning their attention momentarily away from questions
about how to understand the human condition and instead asking questions about
how to best study the human condition (see Michael’s attempt to do so with his
personal p-curve analysis and just a few of the other
great blog posts on it here, here, and here).
So how do we go about putting into place a system that promotes
good science and true discovery as the end goal? Many good ideas have been
thrown out there, and I applaud them all. Below I describe three that have
really stuck out in my mind.
First, psychologist Brian Nosek made a revolutionary
suggestion in a talk I attended last year – we need to do away with journals. Why? The simplest explanation
comes from an Editor who recently rejected a paper I was a co-author on. In the
rejection letter, we were told “Please
understand that my decision was rendered with the recognition that the page
limitations of the journal dictate that only a small percentage of submitted
manuscripts can be accepted. We receive more than 750 submissions per
year, but only publish approximately 125 papers each year. Papers without
major flaws are often not accepted by [this journal] because the magnitude of
the contribution is not sufficient to warrant publication.” That’s right,
perfectly good papers that could help advance science are being rejected
because of page limits. Well, if we started publishing everything online, as
Nosek suggested, we would no longer have to deal with these space constraints.
As long as the science is sound, a paper would be published. Online
publications also mean instant access, rather than waiting for the quarterly
issue to get printed and mailed out to subscribers. He also suggested that we don’t
have separate online journals and instead have one site through which we submit
all papers. No more trying to decide which journal is the best fit for your
interdisciplinary work, and no more guessing how “good” your research is when
trying to decide where to submit it, just submit to the one portal where it
will be tagged and sent to the appropriate reviewers. If we need to
differentiate the breadth and rigor of different papers, we can create a rating
system. Other fields seem to be moving in this direction, and I think it is
time we did too.
Second, just as the stars in my
eyes were beginning to dim slightly in the early stages of graduate school, an
older and wiser graduate student griped that our field really needed to reform
our article submission process. Instead of submitting a paper in its entirety,
we should just submit the introduction and methods. Reviewers would then judge
the paper based on the soundness of the theory and the methods, rather than the
significance of the results. This transition to a focus on methods would ensure
that we conducted sound science rather than worrying about flashy results. And
if we no longer need to worry about paper-based space constraints, there would
be no concern about a paper with null results taking room from a paper with
significant effects. This older and wiser graduate student rocked my world when
she made this suggestion, but I’ve read blog posts by other researchers who
make similar suggestions, so perhaps the time for this change is not so far
off.
Third, social neuroscientist Matthew Lieberman suggested that we
recognize the importance of replication, and instead of rejecting papers that
attempt to replicate, encourage and institutionalize this process. First and
second year graduate students would be trained in research methods by taking on
one of the biggest studies of the previous couple of years and embarking on a
replication project. If students all over the world took on this task, we could
meta-analyze all of the results and within a few years we would have a good
sense of which results were robust and which were a function of Type I error
(or something more pernicious). This approach to replication would also help
put the emphasis back on strong methodology.
Each of these suggestions offers a
different solution for helping tackle the problems of conducting modern day
science, but they all have one thing in common – they help get rid of the
pressure to produce unbelievable results (in the aforementioned paper that
was rejected, one of the criticisms was that the findings were not
counterintuitive enough) and in doing so create a system in which there is
little incentive to cheat and a lot of incentive to do good work. One
frustration I’ve had is that input does not always equal output. We work hard
running good studies, but if they aren’t novel enough (likely because someone
else got their work in a little more quickly), or the results don’t turn out as
we’d hoped, we may find ourselves with a lot put in and little to show for it.
In a system where the focus is on rigorous methods, replicable results, and
there is no need to limit publications for the sake of page constraints, then
our output is going to be a much better reflection of what we put in.
The devil is of course in the
detail, and I’m sure that there are many potential problems with each of these
suggestions, but I do think they are a step in the right direction towards
something new and great. And as someone who still loves asking questions and
seeking answers, I look forward to being part of it.
Article on False Positives: Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant Psychological Science DOI: 10.1177/0956797611417632
Great stuff! My thoughts exactly.
ReplyDeleteGlad to hear it! Thanks for reading,
DeleteAmie
Nice Blog!! Was a good read. looking for more information about this .waiting for your next blog.thanks
ReplyDeleteI do believe that what the average american and the average student at a normal college is reading is sometimes false, and to me as a learner i do not want to be learing something that isn't even true. I don't want to learn psuedoscicence!!
ReplyDelete