We are asking reviewers to rate each abstract numerically on four dimensions. These are:
- Is the work methodologically sound? 1 means unsound; 7 means impeccable.
- Does the work have substantial theoretical implications for psycholinguistics, linguistics, or psychology? 1 means little or unclear theoretical import; 7 means exceptionally notable theoretical import.
- Is the work methodologically or theoretically innovative? 1 means not original and not innovative; 7 means exceptionally original and innovative
- Overall score. Please take into account the originality and quality of the work as well as its relevance to the field. Use the following scale: 1 (poor), 2 (fair), 3 (“meh”), 4 (good), 5 (very good), 6 (excellent) to 7 (outstanding). A rating of “7” should be reserved for submissions that are innovative, methodologically impeccable, and theoretically significant. Please use the entire scale.
The first question is exactly the same as previous years. The second question was elaborated a bit, because ‘theoretically significant’ is likely to mean different things to different reviewers. The third question was previously framed as “Is the work timely and innovative.” We decided we did not want to ask reviewers to rate the ‘timeliness’ of the work: focusing on innovation, rather than trendiness, seemed more important to evaluating the abstracts. Our goal was to promote some more consistent criteria in the responses to this question across reviewers.
Another question that has been asked in previous years is ‘Is the work of broad interest to the CUNY community’. We have dropped this question entirely, for two reasons. First, it was not obvious to us in discussion that this is a great criterion for deciding (e.g.) what should be a talk, and what should be a poster. We wanted to develop a reviewing process whereby innovative, sound, impactful and non-mainstream work had a good opportunity to rise to the top. Second, it seemed to us that reviewers varied widely in how they responded to this question, in ways that seemed orthogonal to our goal of providing a solid, scientifically engaging program. For example, we heard many reviewers in past years who admitted that they were not happy giving honest low marks on this question in response to outstanding but non-mainstream work, knowing that it might ruin the chances of this work getting a talk.
Finally, as in previous years, reviewers will be asked to provide detailed narrative comments to back up, and elaborate on, their ratings.