CUNY reviews are due soon, so we thought it might be interesting to describe the process by which we assigned the abstracts to reviewers. We tried to distill this complicated process down to a single (long) day, which was made possible with a little organization and pre-planning. When the big day came, we booked a room in the Linguistics department from morning until night, ordered a whole bunch of Dunkin’ Donuts, and hunkered down:

We were lucky enough to a ton of psycholinguistic brainpower we could apply to the task. We had support from our lovely and talented students who graciously took time out of their breaks to pitch in. Special thanks go to Anissa Neal, Bethany Dickerson, Alexander Göbel, Erika Mayer, Michael Wilson, and Kuan-Jung Huang. And faculty from all over our university and others pitched in: Mara Breen,  Charles Clifton,  Brian Dillon,  Lyn Frazier,  Jennie Mack,  Shota Momma, and Adrian Staub. We even had little baby Toma Negishi (seen on Shota’s lap) helping us out!

We wanted to keep the actual abstract assignment process to a single day, and to do this, we did a good amount of prep work before the big day so that we were ready to hit the ground running. A week before our meeting date, we created a master Google Sheet that contained information and links to all 432 abstracts, which we were able to download from our conference management software (Soft Conf). Our goal was to have an ‘abstract manager’ from our group for each abstract. Given the number of people pitching in, that worked out to around 33 abstracts. In advance of the meeting, everyone got a chance to claim to abstracts they were most interested in, with students getting first dibs. This way, everyone involved was responsible for assigning reviewers to abstracts that fell somewhere close to their areas of interest, with the idea being that that would put the abstract manager in the best possible position to assign reviewers from the reviewer list.

The actual reviewer assignment process relied on a central Google Spreadsheet that we all worked on in parallel, so that everyone could see everyone else’s reviewer assignments. Our sheet tracked the overall assignment load to each reviewer so that people could work in parallel, but still see how many abstracts were being assigned to any given reviewer. If you would like to use our Google Sheet set-up for your own purposes, you can find a pared down version of it here. It had a couple of bells and whistles that facilitated things, such as a pivot table that tracks the number of abstracts assigned to reviewers, and regular expressions plus conditional formatting in Google Sheets to automatically identify if an author was assigned to their own abstract.

Over the course of a (long) day, we hung out together and assigned reviewers together by jointly editing the sheet. Everyone entered their proposed reviewers from a master list of 173 reviewers into the spreadsheet to assign them to the abstracts. We continuously edited the proposed assignments in order to ensure an equitable distribution of abstracts to reviewers. We found it was especially helpful to do this all together at once together in the room. Not only was it more fun, but we were able to draw on each others’ knowledge when we weren’t sure if a given reviewer was a good fit for a given abstract. One thing we found was very difficult was avoiding conflicts of interest beyond the obvious ones; the last couple of hours involved a close editing of all the assignments in an attempt to ensure that people from the same lab were not reviewing each other’s abstracts (and that e.g. someone wasn’t reviewing an abstract of their former postdoc).

Lastly, Mara Breen deserves special acknowledgement for managing all of the abstracts that had UMass authors associated with them. This was something we had no idea how to handle. The solution we came up with was to give Mara responsibility for all of the UMass affiliated abstracts. Mara worked on a separate sheet to assign reviewers to those, so that the authors of those abstracts remained unaware of who was reviewing their work. This was done to make sure that the reviewers assigned to  a UMass abstract could expect to have the same amount of anonymity afforded to all other reviewers, which is an important part of the process.