Program Selection Process for ISCA 2007

Brad Calder

January 29th, 2007

Disclaimer: I do not at all claim that the below is the best way to run a review process. The below is what I felt would work well for me, and what happened at the PC meeting.

General Stats for ISCA 2007

Some Important Points

A Rant I have

Review Process

I had 4 PC members review every paper, and asked for one external review per paper. This resulted in a PC load of around 25 papers to review for each PC member. I asked for one external reviewer chosen by myself. This was an external reviewer I knew to be an expert on the subject, or an expert I determined from reading the related work for the paper.

Rebuttal Process

After the review deadline, the authors were given 700 words to provide a response to the reviews. After some nagging for a couple of reviewers, 100% of the PC reviews and all of the external reviews were turned in with 1.5 days left to go in the rebuttal process.

Grading Papers

After the rebuttals were entered, each PC member entered a grade for the paper. This grade seemed to correlate very well with how the PC member voted for the paper during the PC meeting (accept or reject).

The PC grade was a value from 1 (Strong Reject) to 6 (Strong Accept), with the following options:

PC members were told to only use "Strong Accept" if they are willing to champion the paper for an accept at the PC meeting, and that they should only use "Strong Reject" if they are willing to really argue against the paper from being accepted.

I asked them to use "Shepherd Accept" if they think there are items that authors can fix, but must be fixed for the paper to be published through the enforcement of shepherding. This should primarily be used if they gave the paper an overall merit ranking of borderline, poor, or reject, and after reading the rebuttal and the other reviews you feel certain items still must be addressed before publishing and they can be addressed with the enforcement of shepherding.

The PC members were instructed to read in detail all of the reviews and the authors response. When entering a grade, it reflected the fact that they have read all of the reviews and considered the arguments from the reviewers not on the PC.

My student Michael Van Biesbrouck added to the CRP review system the ability to rank the papers 4 ways, with the numerical ranking being shown for each paper. The rankings were:

I definitely saw a difference for some papers between the grade and the other overall merit options. Hopefully for reasons such as the rebuttal, etc. Out of the top 35 ranked papers by grade only 1 was rejected.

Grouping Papers for PC Meeting Discussion Order Based on Grades

After grading I broke the papers into 4 groups using the CRP system (these correspond to the W, X, Y, Z categories in the CRP system I had my students added back when I was program chair for MICRO 2003), which I have not seen anyone use since ;-), but I used it again for ISCA. The 4 groups are:

Schedule for the PC meeting

I put up something similar to the below on the board before the PC meeting started, so that everyone knew the schedule. The below is what happened and about when it happened at the PC meeting (note the papers were graded on a scale of 1 to 6, and see above for the grades): Also, after a break in the morning (around 10:00am), we rejected all of the papers in Group Z (Probably Reject).

Discussing a Paper

The following is the process I took when discussing a paper. In using a timer, I kept the discussion of almost every paper under 10 minutes, and almost all papers were kept to 6 to 8 minutes.

For every paper, I assigned a lead reviewer, and sent this out before the PC meeting. The lead reviewer was always the most positive person on the paper. The lead reviewer was to spend a minute or two to summarize the paper, its strengths, and the external reviewers opinion and any important points from the rebuttal. Then I asked the additional PC members who reviewed the paper to comment on the strengths and weaknesses. I tried to pick a discussion order so that we started and ended on a positive note. So if a paper had 2 PC members in favor and 2 against, I would have the lead reviewer start and the 2nd PC member in favor be the 3rd or 4th one to offer their opinion. When it looked like there was consensus, or it looks like there wasn't anything new being said, or time has run out, I then took a vote.

Determining to Shepherd a Paper

Before taking the vote, there were a bunch of papers that had a novel idea or a neat contribution the community would benefit from, but there were some issues with the paper that would prevent it from being accepted straight out. For every paper in this category, I checked to see if it could be saved with shepherding. This consisted of (1) verifying that there was a sold contribution to be had from the paper, (2) seeing if the problems could be fixed during shepherding, and (3) identifying exactly what a shepherd would have the authors fix in order for the paper to be finally accepted. I tried to make sure I gave this same option to every paper that at least one person on the PC wanted to accept.

The main issue with shepherding is when does shepherding actually change the paper so that it is actually a different paper from the submission. Some people will think I took the shepherding a little too far in this respect, and I will be the first two say that maybe I did on a few papers. All of the papers being shepherded had at least one person advocating the paper, and most of the reviewers saw a contribution, but the paper would be rejected without some enforced changes via shepherding.

Voting on a paper

For the initial vote, only the PC members who I have asked to read and review the paper voted on the paper.

Papers that were not a candidate for shepherding were put to a vote of accept or reject. Papers that were a candidate for shepherding were put to a vote of Accept without Shepherding, Accept with Shepherding, and Reject.

If the PC reviewer vote is in favor to accept by at least 1 vote for the paper, then I took the paper. For example, even if there is only 1 reviewer voting to accept, but 3 other reviewers are abstaining then I accepted the paper.

If the PC reviewer vote was tied (and there is at least one accept vote), then I asked the whole PC to vote. This only occurred a handful of times. During the voting process I always asked for accept votes before reject votes. I found it interesting that a few times when I asked the whole PC to vote, there would be a tie via the PC vote. When this occurred, since the reject vote was last, a couple of times this prompted a delayed reject vote to be cast by someone to break the tie. Leaving the vote at that, didn't seem right, since if the accept vote was last, there might have been a delayed accept vote cast to try to break the tie. I therefore, re-did the accept vote, when there was a tie breaking delayed reject vote cast, and repeated the process until the votes stabilized.

PC Papers and Conflicts of Interest

We discussed the PC papers in the afternoon (see the above schedule). Each PC member was given a view of all of the papers with their conflicts removed.

Papers in which PC members had a conflict with will were marked as an X, and no information was shown. For each of these papers I had the PC members leave the room.

Mark Hill and Margaret Martonosi lead the discussions for the papers that I had a conflict with. Thanks!

Acknowledgments

Thanks to all of the people who submitted papers to ISCA, all of the PC members (they all did a ton of work), and the external reviewers.

Many thanks go to Michael Van Biesbrouck for spending over 100 hours on the CRP system getting it in shape and adding features needed for this review process.