Disclaimer: I do not at all claim that the below is the best way to run a review process. The below is what I felt would work well for me, and what happened at the PC meeting.

General Stats for ISCA 2007

• Number of PC members: 33
• Number of Submissions: 204
• Number of Reviews: 1000+
• Total Number of Papers Accepted: 46 (22%) (This includes Shepherd Accept, which are conditionally accepted)
• Number of Papers Discussed at PC Meeting: 75
• Number of Papers with Shepherd Accepts: 16 (35% of accepted papers)
• Total Number of PC Papers Accepted (includes Shepherd Accept): 16 (35% of accepted papers)
• Number of PC Papers that were Shepherd Accept: 4
• Number of PC member reviews per paper: 4
• Number of Reviews per paper: 5

Some Important Points

• Assigning External Reviewers - Following Mark Hill's 2005 exampled I assigned external reviewers myself, but I only assigned one external reviewer (Mark Hill assigned 2 external himself), and instead had 4 PC reviews per paper. This allowed me, with the blind review process, to try and find an expert external reviewer for each paper.
• Rebuttal Process - I am positive that some papers would not have been accepted without the rebuttal process. For those papers still not accepted after the rebuttal process, hopefully going through the rebuttal process was still a helpful experience. For my rejected papers, it has always been helpful, focusing us on what we need to do to next.
• Grading Process - After all of the author responses were in, I had the PC members grade each paper (see more information on this below). All of the PC members had their grades turned in on time. The benefits of this I believe are:
  • It requires all PC members to read the rebuttals and the reviews before the meeting, and adjust there views for the papers. There were a handful of papers that improved their ranking quite a bit when the final grade was entered.
  • It puts the papers in a discussion order that is aligned with the PC member's view of the paper after reading the rebuttals and the other reviews.
  • Starting off on a positive note - We accepted the first 11 papers (non-PC papers) very quickly, since the PC members all had an accept grades for those papers. This helped start the PC meeting out on a very positive note, and we had 11 papers accepted in about 30 minutes. In fact, we discussed 22 papers before rejecting the first paper. In the top 35 ranked papers by grade, only 1 paper was rejected.
• Assigning Lead Reviewers - Before the PC meeting, I assigned each PC member a list of papers they were in charge of at the PC meeting. They were in charge of summarizing what the paper is about, discussing the strengths of the paper, describing any important points in the rebuttal, and summarizing the external review. The PC member assigned was always the most positive person on the paper. This helped get each paper off on a strong and positive note, as well as keep the PC meeting moving.
• Shepherd Accept Vote Before Determining the Shepherd - I asked for PC members to vote to accept a paper with shepherding before determining who the shepherd would be. For every Shepherd Accept paper, I was able to find a PC member to shepherd the paper. In fact, it seemed like it was much easier to find a shepherd after the paper was Shepherd Accepted than before voting on the paper. In addition, it seemed like a couple of papers would have been rejected if I had forced a PC member to volunteer to be the shepherd before hand. It was much easier to twist arms (only had to do it a few times) by first agreeing that the paper should be accepted with shepherding.

A Rant I have

• Number of Papers a PC Member is Allowed to Submit - The SIGARCH guidelines recommend that we should not allow PC members to submit more than 2 papers. I firmly believe that this recommendation should not be followed. We should not make it a requirement that PC members can submit at most 2 papers. An argument I have heard in the past for enforcing this is that it reduces the amount of discussion time on PC papers. Not all PC member papers have to be discussed. At our meeting we only discussed the PC member papers above the same threshold applied to non-PC member papers.

  Out of the 75 papers discussed at the PC meeting, 23 (31%) of those papers were PC papers. Only 1 PC member had three papers discussed. 5 PC members had 2 papers discussed, and 13 PC members had 1 paper discussed. (Note, there were sometimes multiple PC authors on a single paper).

  The main thing limiting PC members to 2 submissions does is make some of the best people in our field turn down being on program committees. Some people have more than 2 projects or collaborations, and they choose to turn down being on PCs that enforce this limit because they need to submit at least 3 papers.

  To be in the spirit of the SIGARCH guidelines, I instead limited each PC member to at most 2 accepted papers. In the end, this limitation had no effect on the results of this PC meeting. Without this limitation, there still would not have been any PC members with more than 2 papers accepted for ISCA 2007.

Review Process

Rebuttal Process

Grading Papers

The PC grade was a value from 1 (Strong Reject) to 6 (Strong Accept), with the following options:

• Strong Accept (6), I will Champion this paper
• Weak Accept (5), but I will not Champion this paper
• Shepherd Accept (4), only accept if the paper is shepherded
• Needs more discussion at PC meeting to grade (3)
• Weak Reject (2), but I will not argue against accepting the paper
• Strong Reject (1), I will argue against accept

PC members were told to only use "Strong Accept" if they are willing to champion the paper for an accept at the PC meeting, and that they should only use "Strong Reject" if they are willing to really argue against the paper from being accepted.

I asked them to use "Shepherd Accept" if they think there are items that authors can fix, but must be fixed for the paper to be published through the enforcement of shepherding. This should primarily be used if they gave the paper an overall merit ranking of borderline, poor, or reject, and after reading the rebuttal and the other reviews you feel certain items still must be addressed before publishing and they can be addressed with the enforcement of shepherding.

The PC members were instructed to read in detail all of the reviews and the authors response. When entering a grade, it reflected the fact that they have read all of the reviews and considered the arguments from the reviewers not on the PC.

My student Michael Van Biesbrouck added to the CRP review system the ability to rank the papers 4 ways, with the numerical ranking being shown for each paper. The rankings were:

• Grade - Average Grade
• OM_ALL - Average Overall Merit
• OM_RL - Average Overall Merit removing the lowest score
• OM_RLH - Average Overall Merit removing the lowest and highest score

Grouping Papers for PC Meeting Discussion Order Based on Grades

• W - Probably Accept - These are all of the papers with only accept grades. We had 16 such papers for ISCA 2007.
• Z - Probably Reject - These are all of the papers with only reject grades. We had 79 such papers.
• X - Definite Discuss - These are the papers above a given grade threshold that we discussed at the PC meeting. All papers above this threshold were discussed. I set this threshold to >= 3.0, and we had 53 papers in this Group X that had an average grade >= 3.0. This threshold also played an important part in determining which PC papers to automatically discuss. Any PC paper with an average grade >= 3.0 was automatically discussed.
• Y - Discuss only with Champion - This is the remaining papers, and a paper (including PC papers) can be discussed if there is a champion.

Schedule for the PC meeting

• 7:30 - 8:00 Give everyone the material on their laptop. Each PC member was given a copy of all of the papers and their reviews for which they did not have a conflict with.
• 8:00 - 8:40 - Discuss non-PC papers in Group W (Probably Accept). This is important to give a quick summary of what we are accepting and the reason why we are accepting it. This took 2-3 minutes per paper in this group.
• 8:40 - 12:10 - Discuss non-PC papers in Group X (Definite Discuss) in grade order. Discuss papers with an average grade >= 3.25.
• 12:10 - 12:30 - Short lunch break
• 12:30 - 4:00 - Discuss PC member papers with an average grade >= 3.0 . PC member papers were discussed in rank order in terms of grade, with the PC member leaving the room when we got to their paper.
• 4:00 - 5:30 - Discuss remaining non-PC papers with grade >= 3.0 .
• 5:30 - 6:30 - Discuss any papers in Group Y, if there is a champion wanting to discuss the paper. Any PC member can bring up a paper that was not yet discussed if they want to champion it for accept. We got 1 accept (a shepherd accept) out of this group of papers.

Discussing a Paper

For every paper, I assigned a lead reviewer, and sent this out before the PC meeting. The lead reviewer was always the most positive person on the paper. The lead reviewer was to spend a minute or two to summarize the paper, its strengths, and the external reviewers opinion and any important points from the rebuttal. Then I asked the additional PC members who reviewed the paper to comment on the strengths and weaknesses. I tried to pick a discussion order so that we started and ended on a positive note. So if a paper had 2 PC members in favor and 2 against, I would have the lead reviewer start and the 2nd PC member in favor be the 3rd or 4th one to offer their opinion. When it looked like there was consensus, or it looks like there wasn't anything new being said, or time has run out, I then took a vote.

Determining to Shepherd a Paper

The main issue with shepherding is when does shepherding actually change the paper so that it is actually a different paper from the submission. Some people will think I took the shepherding a little too far in this respect, and I will be the first two say that maybe I did on a few papers. All of the papers being shepherded had at least one person advocating the paper, and most of the reviewers saw a contribution, but the paper would be rejected without some enforced changes via shepherding.

Voting on a paper

Papers that were not a candidate for shepherding were put to a vote of accept or reject. Papers that were a candidate for shepherding were put to a vote of Accept without Shepherding, Accept with Shepherding, and Reject.

If the PC reviewer vote is in favor to accept by at least 1 vote for the paper, then I took the paper. For example, even if there is only 1 reviewer voting to accept, but 3 other reviewers are abstaining then I accepted the paper.

If the PC reviewer vote was tied (and there is at least one accept vote), then I asked the whole PC to vote. This only occurred a handful of times. During the voting process I always asked for accept votes before reject votes. I found it interesting that a few times when I asked the whole PC to vote, there would be a tie via the PC vote. When this occurred, since the reject vote was last, a couple of times this prompted a delayed reject vote to be cast by someone to break the tie. Leaving the vote at that, didn't seem right, since if the accept vote was last, there might have been a delayed accept vote cast to try to break the tie. I therefore, re-did the accept vote, when there was a tie breaking delayed reject vote cast, and repeated the process until the votes stabilized.

PC Papers and Conflicts of Interest

Papers in which PC members had a conflict with will were marked as an X, and no information was shown. For each of these papers I had the PC members leave the room.

Mark Hill and Margaret Martonosi lead the discussions for the papers that I had a conflict with. Thanks!

Acknowledgments

Thanks to all of the people who submitted papers to ISCA, all of the PC members (they all did a ton of work), and the external reviewers.

Many thanks go to Michael Van Biesbrouck for spending over 100 hours on the CRP system getting it in shape and adding features needed for this review process.