Choosing reviewers

One of the major projects behind the scenes of organizing CUNY is getting the review process organized and running. One major, early part of this is to decide who gets to review abstracts. The CUNY tradition in this regard has been informal; organizers have simply kept tabs on who is an active, contributing member of the community, and included them as part of a long list of reviewers who are tapped year after year to review abstracts. This list tends to get passed down from organizer to organizer, with small modifications along the way.

This year, we decided to expand the set of reviewers in a more systematic way. The main reason for this was a Twitter discussion started by a number of junior, post-PhD researchers in the CUNY community around this time last year. The question that they asked was: “How do you become a CUNY reviewer?” It became clear to us that this process was totally mysterious from their perspective: you never were asked until one day, you were. We had a chance to talk to a number of junior researchers who had not been reviewing for CUNY, despite having been active members of the community, and get their thoughts on this.

As a result of these talks, we decided to make some changes to reviewer recruitment. We had several constraints we tried to respect. The list of reviewers could be expanded only so much while keeping the review process manageable, and we wanted our process of selecting new reviewers to be objective and transparent, rather than relying on personal connections. In light of these constraints, the newly-invited reviewers for this year’s CUNY will include:

  • The existing list of CUNY reviewers, passed down from previous years, and
  • All post-PhD researchers who were first author on one (non-invited talk) in the last five years.

This will add about 50 new potential reviewers to the list of 167 that we have inherited from last year’s CUNY.

We want to be clear that in expanding the list in just this way, we do not mean to imply that only the people who presented a talk in the last five years are valuable, contributing members of the community. That is obviously false. In fact, our major problem in trying to decide on reviewers was that there are too many valuable members of our community! When we considered a number of different objective criteria that we could imagine for making this cut (e.g., by including first authors on recent posters), we consistently ran into the problem of increasing the reviewer base by several hundred reviewers, to a completely unwieldy size. Our reviewer selection criterion was a compromise that allowed us to objectively expand this pool, while keeping the overall number of reviewers manageable.

Leave a Reply

Your email address will not be published. Required fields are marked *