Disclaimer: I do not at all claim that the below is the best way
to run a review process. The below is what I felt would work well for me, and what
happened at the PC meeting.
General Stats for ISCA 2007
- Number of PC members: 33
- Number of Submissions: 204
- Number of Reviews: 1000+
- Total Number of Papers Accepted: 46 (22%) (This includes Shepherd Accept, which are conditionally accepted)
- Number of Papers Discussed at PC Meeting: 75
- Number of Papers with Shepherd Accepts: 16 (35% of accepted papers)
- Total Number of PC Papers Accepted (includes Shepherd Accept): 16 (35% of accepted papers)
- Number of PC Papers that were Shepherd Accept: 4
- Number of PC member reviews per paper: 4
- Number of Reviews per paper: 5
Some Important Points
- Assigning External Reviewers - Following Mark Hill's 2005
exampled I assigned external reviewers myself, but I only assigned one
external reviewer (Mark Hill assigned 2 external himself), and instead
had 4 PC reviews per paper. This allowed me, with the blind review
process, to try and find an expert external reviewer for each paper.
- Rebuttal Process - I am positive that some papers would
not have been accepted without the rebuttal process. For those papers
still not accepted after the rebuttal process, hopefully going through
the rebuttal process was still a helpful experience. For my rejected
papers, it has always been helpful, focusing us on what we need
to do to next.
- Grading Process - After all of the author responses were in, I
had the PC members grade each paper (see more information on this below).
All of the PC members had their grades turned in on time.
The benefits of this I believe are:
- It requires all PC members to read the rebuttals and the reviews
before the meeting, and adjust there views for the papers. There
were a handful of papers that improved their ranking quite a bit
when the final grade was entered.
- It puts the papers in a discussion order that
is aligned with the PC member's view of the paper
after reading the rebuttals and the other
reviews.
- Starting off on a positive note - We accepted the first 11 papers
(non-PC papers) very quickly, since the PC members all had an accept
grades for those papers. This helped start the PC meeting out on a very
positive note, and we had 11 papers accepted in about 30 minutes. In
fact, we discussed 22 papers before rejecting the first paper.
In the top 35 ranked papers by grade, only 1 paper was rejected.
- Assigning Lead Reviewers - Before the PC meeting, I
assigned each PC member a list of papers they were in charge of at the
PC meeting. They were in charge of summarizing what the paper is
about, discussing the strengths of the paper, describing any important
points in the rebuttal, and summarizing the external review. The PC
member assigned was always the most positive person on the paper.
This helped get each paper off on a strong and positive note, as
well as keep the PC meeting moving.
- Shepherd Accept Vote Before Determining the Shepherd - I
asked for PC members to vote to accept a paper with shepherding before
determining who the shepherd would be. For every Shepherd Accept
paper, I was able to find a PC member to shepherd the paper. In fact,
it seemed like it was much easier to find a shepherd after the paper
was Shepherd Accepted than before voting on the paper. In addition,
it seemed like a couple of papers would have been rejected if I had
forced a PC member to volunteer to be the shepherd before hand. It
was much easier to twist arms (only had to do it a few times) by first
agreeing that the paper should be accepted with shepherding.
A Rant I have
- Number of Papers a PC Member is Allowed to Submit - The
SIGARCH guidelines recommend that we should not allow PC members to
submit more than 2 papers. I firmly believe that this recommendation
should not be followed. We should not make it a requirement that PC
members can submit at most 2 papers. An argument I have heard in the
past for enforcing this is that it reduces the amount of discussion
time on PC papers. Not all PC member papers have to be discussed. At
our meeting we only discussed the PC member papers above the same
threshold applied to non-PC member papers.
Out of the 75 papers discussed at the PC meeting,
23 (31%) of those papers were PC papers. Only 1 PC member had three
papers discussed. 5 PC members had 2 papers discussed, and 13 PC
members had 1 paper discussed. (Note, there were sometimes multiple PC
authors on a single paper).
The main thing limiting PC members to 2 submissions does is make
some of the best people in our field turn down being on program
committees. Some people have more than 2 projects or collaborations,
and they choose to turn down being on PCs that enforce this limit
because they need to submit at least 3 papers.
To be in the spirit of the SIGARCH guidelines, I instead limited each
PC member to at most 2 accepted papers. In the end, this limitation
had no effect on the results of this PC meeting. Without this
limitation, there still would not have been any PC members with more
than 2 papers accepted for ISCA 2007.
Review Process
I had 4 PC members review every paper, and asked for one external
review per paper. This resulted in a PC load of around 25 papers to
review for each PC member. I asked for one external reviewer chosen
by myself. This was an external reviewer I knew to be an expert on
the subject, or an expert I determined from reading the related work
for the paper.
Rebuttal Process
After the review deadline, the authors were given 700 words to provide
a response to the reviews. After some nagging for a couple of
reviewers, 100% of the PC reviews and all of the external reviews were
turned in with 1.5 days left to go in the rebuttal process.
Grading Papers
After the rebuttals were entered, each PC member entered a grade for
the paper. This grade seemed to correlate very well with how the PC
member voted for the paper during the PC meeting (accept or reject).
The PC grade was a value from 1 (Strong Reject) to 6 (Strong Accept),
with the following options:
- Strong Accept (6), I will Champion this paper
- Weak Accept (5), but I will not Champion this paper
- Shepherd Accept (4), only accept if the paper is shepherded
- Needs more discussion at PC meeting to grade (3)
- Weak Reject (2), but I will not argue against accepting the paper
- Strong Reject (1), I will argue against accept
PC members were told to only use "Strong Accept" if they are willing
to champion the paper for an accept at the PC meeting, and that they
should only use "Strong Reject" if they are willing to really argue
against the paper from being accepted.
I asked them to use "Shepherd Accept" if they think there are items
that authors can fix, but must be fixed for the paper to be published
through the enforcement of shepherding. This should primarily be used
if they gave the paper an overall merit ranking of borderline, poor, or
reject, and after reading the rebuttal and the other reviews you feel
certain items still must be addressed before publishing and they can
be addressed with the enforcement of shepherding.
The PC members were instructed to read in detail all of the reviews
and the authors response. When entering a grade, it reflected the
fact that they have read all of the reviews and considered the
arguments from the reviewers not on the PC.
My student Michael Van Biesbrouck added to the CRP review system the
ability to rank the papers 4 ways, with the numerical ranking being
shown for each paper. The rankings were:
- Grade - Average Grade
- OM_ALL - Average Overall Merit
- OM_RL - Average Overall Merit removing the lowest score
- OM_RLH - Average Overall Merit removing the lowest and highest score
I definitely saw a difference for some papers between the grade and
the other overall merit options. Hopefully for reasons such as the
rebuttal, etc. Out of the top 35 ranked papers by grade only 1 was
rejected.
Grouping Papers for PC Meeting Discussion Order Based on Grades
After grading I broke the papers into 4 groups using the CRP system
(these correspond to the W, X, Y, Z categories in the CRP system I had
my students added back when I was program chair for MICRO 2003), which
I have not seen anyone use since ;-), but I used it again for ISCA.
The 4 groups are:
- W - Probably Accept - These are all of the papers with only accept grades.
We had 16 such papers for ISCA 2007.
- Z - Probably Reject - These are all of the papers
with only reject grades. We had 79 such papers.
- X - Definite Discuss - These are the papers above a given grade
threshold that we discussed at the PC meeting. All papers above this
threshold were discussed. I set this threshold to >= 3.0, and we
had 53 papers in this Group X that had an average grade >= 3.0.
This threshold also played an
important part in determining which PC papers to automatically discuss.
Any PC paper with an average grade >= 3.0 was automatically discussed.
- Y - Discuss only with Champion - This is the remaining
papers, and a paper (including PC papers) can be discussed if there is a champion.
Schedule for the PC meeting
I put up something similar to the below on the board before the PC
meeting started, so that everyone knew the schedule. The below is what
happened and about when it happened at the PC meeting (note the papers
were graded on a scale of 1 to 6, and see above for the grades):
- 7:30 - 8:00 Give everyone the material on their laptop. Each PC
member was given a copy of all of the papers and their reviews for which
they did not have a conflict with.
- 8:00 - 8:40 - Discuss non-PC papers in Group W (Probably Accept).
This is important to give a quick summary of what we are accepting
and the reason why we are accepting it. This took 2-3 minutes per paper
in this group.
- 8:40 - 12:10 - Discuss non-PC papers in Group X (Definite Discuss)
in grade order. Discuss papers with an average grade >= 3.25.
- 12:10 - 12:30 - Short lunch break
- 12:30 - 4:00 - Discuss PC member papers with an average grade >= 3.0 .
PC member papers were discussed in rank order in terms of grade, with the
PC member leaving the room when we got to their paper.
- 4:00 - 5:30 - Discuss remaining non-PC papers with grade >= 3.0 .
- 5:30 - 6:30 - Discuss any papers in Group Y, if there is a champion
wanting to discuss the paper. Any PC member can bring up a paper that was not
yet discussed if they want to champion it for accept. We got 1 accept
(a shepherd accept) out of this group of papers.
Also, after a break in the morning (around 10:00am), we rejected all of the papers in
Group Z (Probably Reject).
Discussing a Paper
The following is the process I took when discussing a paper. In using
a timer, I kept the discussion of almost every paper under 10 minutes,
and almost all papers were kept to 6 to 8 minutes.
For every paper, I assigned a lead reviewer, and sent this out before
the PC meeting. The lead reviewer was always the most positive person
on the paper. The lead reviewer was to spend a minute or two to
summarize the paper, its strengths, and the external reviewers opinion
and any important points from the rebuttal. Then I asked the
additional PC members who reviewed the paper to comment on the
strengths and weaknesses. I tried to pick a discussion order so that
we started and ended on a positive note. So if a paper had 2 PC
members in favor and 2 against, I would have the lead reviewer start
and the 2nd PC member in favor be the 3rd or 4th one to offer their
opinion. When it looked like there was consensus, or it looks like
there wasn't anything new being said, or time has run out, I then took
a vote.
Determining to Shepherd a Paper
Before taking the vote, there were a bunch of papers that had a novel
idea or a neat contribution the community would benefit from, but
there were some issues with the paper that would prevent it from being
accepted straight out. For every paper in this category, I checked to
see if it could be saved with shepherding. This consisted of (1)
verifying that there was a sold contribution to be had from the paper,
(2) seeing if the problems could be fixed during shepherding, and (3)
identifying exactly what a shepherd would have the authors fix in
order for the paper to be finally accepted. I tried to make sure I
gave this same option to every paper that at least one person on the
PC wanted to accept.
The main issue with shepherding is when does shepherding actually
change the paper so that it is actually a different paper from the
submission. Some people will think I took the shepherding a little
too far in this respect, and I will be the first two say that maybe I
did on a few papers. All of the papers being shepherded had at least
one person advocating the paper, and most of the reviewers saw a
contribution, but the paper would be rejected without some enforced
changes via shepherding.
Voting on a paper
For the initial vote, only the PC members who I have asked to read and
review the paper voted on the paper.
Papers that were not a candidate for shepherding were put to a
vote of accept or reject. Papers that were a candidate for
shepherding were put to a vote of Accept without Shepherding, Accept
with Shepherding, and Reject.
If the PC reviewer vote is in favor to accept by at least 1 vote for the paper,
then I took the paper. For example, even if there is only 1 reviewer
voting to accept, but 3 other reviewers are abstaining then I accepted
the paper.
If the PC reviewer vote was tied (and there is at least one accept
vote), then I asked the whole PC to vote. This only occurred a handful
of times. During the voting process I always asked for accept votes
before reject votes. I found it interesting that a few times when I
asked the whole PC to vote, there would be a tie via the PC vote.
When this occurred, since the reject vote was last, a couple of times
this prompted a delayed reject vote to be cast by someone to break the
tie. Leaving the vote at that, didn't seem right, since if the
accept vote was last, there might have been a delayed accept vote cast
to try to break the tie. I therefore, re-did the accept vote, when
there was a tie breaking delayed reject vote cast, and repeated the
process until the votes stabilized.
PC Papers and Conflicts of Interest
We discussed the PC papers in the afternoon (see the above schedule).
Each PC member was given a view of all of the papers with their
conflicts removed.
Papers in which PC members had a conflict with will were marked as an
X, and no information was shown. For each of these papers I had the
PC members leave the room.
Mark Hill and Margaret Martonosi lead the discussions for the papers
that I had a conflict with. Thanks!
Acknowledgments
Thanks to all of the people who submitted papers to ISCA, all of the
PC members (they all did a ton of work), and the external reviewers.
Many thanks go to Michael Van Biesbrouck for spending over 100
hours on the CRP system getting it in shape and adding features needed
for this review process.