Friday Fun: My "Favorite" Comments From Anonymous Reviewers

This Friday I thought it might be nice to provide a window into the manuscript review process in psychology. In short, after a researcher (1) designs a handful of studies (typically without much success), (2) finally discovers a finding that is interesting (months to years later), and (3) survives what is (typically) a hyper-critical vetting from among his/her research peers within the home University, it is now time to take on the empirical review process!

The review process is pretty simple for most journals. An editor--usually an influential scholar in the field--reads the manuscript and assigns the manuscript to 3-4 researchers whom he/she thinks can provide an expert evaluation of the research. These anonymous reviewers are then asked to provide critical evaluations of the research in written comments. These comments are then returned to the author along with a written final decision from the editor. In this final decision, the editor typically takes the reviewer comments into consideration, and decides whether the paper is worthy of publication, should be revised, or rejected.

Dawson obviously crying over the comments of Reviewer #2  (source)

[A quick note about psychology reviews: I don't know anything about other fields, but a colleague at UC San Francisco once told me that unlike other science fields, psychologists are the harshest critics of new research, adding poignantly, "They eat their young."]

What I've done below is reproduce some of my "favorite" quotes from these reviews. When you are reading these, try to imagine me crying into a cup of english breakfast tea!




Most Hilarious
"In an otherwise tidy data story this is a bit of a turd in the punch bowl." -- Reviewer 2

This review comes from a paper examining how people from lower socioeconomic backgrounds react with more hostility to teasing behavior relative to people from upper socioeconomic backgrounds. What makes this review especially hilarious to me is that I actually know the person who wrote this comment, and I now think of his/her voice whenever I read it!

In retrospect, it's hard to argue this comment. In this paper, we had a consistent set of results in all but one case--wherein our hostile reactivity hypothesis did not replicate using one of the measures of socioeconomic status. This one aberration was the proverbial "turd" in the punchbowl of our research.

Punch: better without turd (source)

Most Condescending
"Your line of research is promising but it is not yet mature (or sophisticated?) enough at the theoretical level to warrant publication." -- Editor

This review comes from my first big research project. In the paper we were wading into the waters of social class research for the very first time. Imagine being a second year graduate student and receiving the first (ever) review of a manuscript that you have basically spent the last 1.5 years working on. To put it simply, this one stung--especially because it came from an editor whom I still respect a great deal (unlike the reviewers, the editor is not anonymous). In retrospect, the argument wasn't yet mature enough in this version of the paper.

The one thing that makes me feel better about the comment is that we finally did get the paper published at this journal (albeit with new data, and a different editor who appreciated our mature arguments!).

I like to think of the editor's words being said by this guy (source)

Most Clever/Hurtful
"Inattention to and highly selective reporting of the larger epidemiological literature, coupled with these outlandish claims, guarantees that this healthy baby will surely be tossed out with the bathwater." -- Reviewer 3

This review is from the same paper on socioeconomic status and hostile reactivity as above (a different reviewer though). In the paper, we did not do a good enough job--in the eyes of the reviewer-- of effectively characterizing the literature linking socioeconomic status to health. It's a clever comment, but typically, my bathwater gets drained out of the bath tub, so I'm not sure how the baby fits in there! 

Where are you going with that baby!!! (source)

Most Flattering

"It is an absolute pleasure to read a manuscript that does so many things well." -- Reviewer 1

All reviewer comments are not negative, and I don't want to leave you with that impression. Some researchers are genuinely impressed with your ideas. After all, we do some things well!

I'd love to hear about the feedback you receive in your own work. Let us know about it in the comments. If we get a lot of responses, this may become a regular feature on the blog!

Kassirer JP, & Campion EW (1994). Peer review. Crude and understudied, but indispensable. JAMA : the journal of the American Medical Association, 272 (2), 96-7 PMID: 8015140