“No data, yet combines two or more of the worst phonological theories, resulting in an account that is far more complicated and assumption-laden than the simple if typologically odd pseudo-example given.”
I received this review on an abstract I submitted recently. I’ve gotten plenty of bad reviews in the sense of them being negative, but I’ve never gotten one that was so unprofessional, and that made it so clear that the reviewer hadn’t engaged with the abstract in anything but the most superficial fashion. Because I didn’t think this reviewer was doing their job, I was moved to complain about it. I did so as follows:
“I’ve never complained about a conference review before, but this is one’s beyond the pale. I don’t want you to do anything about it, but I had to tell you I’m pretty shocked by it.”
The conference organizer reported that the program committee agreed that the review was unprofessional, and that this reviewer, along with another who had engaged in “soapboxing or axe-grinding”, would not be included in the list of reviewers passed on to the next year’s organizer.
I was pleased with this outcome, and I thought I’d tell this story because this seemed like a good way of improving the quality of reviewer pools that others might usefully adopt. I’d also be happy if this contributed to a general discussion of what the expectations are for reviews, and how we can make them better.
I think we probably all have horror stories of one form or another. At one point, Greg Hickok and David Poeppel were toying with the idea of blogging the reviews they have received (see, for example, http://www.talkingbrains.org/2008/10/mirror-neuron-review-reviews-to-see.html), and the PLoS One idea of continued comment after publication is an interesting idea, but one that doesn’t seem to be used much.
I wonder though, whether this outcome will change the reviewer’s behavior at all. It seems to me that just to not be asked back next year doesn’t convey a specific enough message to them. I know that “no fly lists” of bad reviewers exist, but I think that editors or organizers need to also tell these people that they wrote an unacceptable review, and that they’re being put on the blacklist.
Good point Bill. I admit that I did have the thought that the author might see this post and find out that the review had been judged unacceptable, and that there were consequences.
I’ll pass along your suggestion to the conference organizer in question. Another possibility would be that the reviews themselves could be published on the conference website along with the grounds for them being judged unacceptable, so that future reviewers would have an idea of what’s not OK. (Positive guidelines would be a good idea too, of course.)
Joe, this is a belated comment (well, I just found this article).
Do you remember when I was still a student, I got a reviewer who called me a liar arguing that I must have been fabricating data (of course s/he did not cite evidence for that claim)? That was a truly traumatic experience. I think Andries told me that he once received a review saying that “it is sad to see that the field is corrupted so much so that I would have to review a bad article like this”. A student of mine recently received a pretty badly-written review, containing a statement like “the field will not forgive you…”. I think having an online forum to share this sort of bad reviews could be really useful.
I also find this online article useful:
http://www.psychologicalscience.org/index.php/publications/observer/2007/april-07/twelve-tips-for-reviewers.html
I think they sent it to me when I reviewed for a psychology journal.
Thanks for this Shigeto! I hearby withdraw my claim to the worst review ever!