The review process is pretty simple for most journals. An editor--usually an influential scholar in the field--reads the manuscript and assigns the manuscript to 3-4 researchers whom he/she thinks can provide an expert evaluation of the research. These anonymous reviewers are then asked to provide critical evaluations of the research in written comments. These comments are then returned to the author along with a written final decision from the editor. In this final decision, the editor typically takes the reviewer comments into consideration, and decides whether the paper is worthy of publication, should be revised, or rejected.
 |
| Dawson obviously crying over the comments of Reviewer #2 (source) |
[A quick note about psychology reviews: I don't know anything about other fields, but a colleague at UC San Francisco once told me that unlike other science fields, psychologists are the harshest critics of new research, adding poignantly, "They eat their young."]
What I've done below is reproduce some of my "favorite" quotes from these reviews. When you are reading these, try to imagine me crying into a cup of english breakfast tea!
Most Hilarious
"In an otherwise tidy data story this is a bit of a turd in the punch bowl." -- Reviewer 2
This review comes from a paper examining how people from lower socioeconomic backgrounds react with more hostility to teasing behavior relative to people from upper socioeconomic backgrounds. What makes this review especially hilarious to me is that I actually know the person who wrote this comment, and I now think of his/her voice whenever I read it!
In retrospect, it's hard to argue this comment. In this paper, we had a consistent set of results in all but one case--wherein our hostile reactivity hypothesis did not replicate using one of the measures of socioeconomic status. This one aberration was the proverbial "turd" in the punchbowl of our research.
 |
| Punch: better without turd (source) |
Most Condescending
"Your line of research is promising but it is not yet mature (or sophisticated?) enough at the theoretical level to warrant publication." -- Editor
This review comes from my first big research project. In the paper we were wading into the waters of social class research for the very first time. Imagine being a second year graduate student and receiving the first (ever) review of a manuscript that you have basically spent the last 1.5 years working on. To put it simply, this one stung--especially because it came from an editor whom I still respect a great deal (unlike the reviewers, the editor is not anonymous). In retrospect, the argument wasn't yet mature enough in this version of the paper.
The one thing that makes me feel better about the comment is that we finally did get the paper published at this journal (albeit with new data, and a different editor who appreciated our mature arguments!).
 |
| I like to think of the editor's words being said by this guy (source) |
Most Clever/Hurtful
"Inattention to and highly selective reporting of the larger epidemiological literature, coupled with these outlandish claims, guarantees that this healthy baby will surely be tossed out with the bathwater." -- Reviewer 3
This review is from the same paper on socioeconomic status and hostile reactivity as above (a different reviewer though). In the paper, we did not do a good enough job--in the eyes of the reviewer-- of effectively characterizing the literature linking socioeconomic status to health. It's a clever comment, but typically, my bathwater gets drained out of the bath tub, so I'm not sure how the baby fits in there!
 |
| Where are you going with that baby!!! (source) |
Most Flattering
"It is an absolute pleasure to read a manuscript that does so many things well." -- Reviewer 1All reviewer comments are not negative, and I don't want to leave you with that impression. Some researchers are genuinely impressed with your ideas. After all, we do some things well!
I'd love to hear about the feedback you receive in your own work. Let us know about it in the comments. If we get a lot of responses, this may become a regular feature on the blog!
Kassirer JP, & Campion EW (1994). Peer review. Crude and understudied, but indispensable. JAMA : the journal of the American Medical Association, 272 (2), 96-7 PMID: 8015140
One of the best goes back to the 18th century: Samuel Johnson : "Your manuscript is both good and original. But the part that is good is not original, and the part that is original is not good."
ReplyDeleteThat's a great one! I would have used it as a lead-in had I known about it. Thanks for reading!
ReplyDeleteUnless I am missing something fundamental, this result is not surprising at all. (That was a comment from a member of the Editorial board who missed all the points of the paper...)
ReplyDeleteAside from commenting that the prose (in Methods and Results) has somewhat of an impenetrable character, this Reviewer has a number of substantive comments that reduce enthusiasm for the work.
Of course, some are more positive than those 2. But scientists can be harsh sometimes...
Thanks for these. I love "the prose has somewhat of an impenetrable character." That's a wonderful phrase!
ReplyDeleteThanks for reading.