Reviewing Papers

By Steven Swanson (swanson@cs.uscd.edu)

Peer review is an essential part of the research process, since it is how the research community decides what gets published. This means that it decides, in part, the fate of research projects and people careers. Give the high stakes, provide thoughtful, thorough reviews is a shared responsibility of all members of the community.

To help prepare you for this responsibility, I will ask you to review papers from time to time. This web page provides information about how that process will work, an idea of what a good review looks like, and guidance on how to create one.

Throughout this process, the Golden Rule applies: Keep in mind the reviews you have received and how annoying (or worse) it feels to receive poorly reasoned, inaccurate, sloppy, rude, or otherwise poor reviews that made you feel like your work had not received a fair shake. Don't do that to someone else.

Nuts and Bolts

When I request your help on a review, I'll provide you with a pdf of the paper, a review form of some sort, and a due date. The due date is when the review is due, but I'll need to look it over and potentially add my own comments, so your review will need to be complete one week earlier. Don't let that deadline slip.

Read over this document and the articles it links to. Read the paper (probably twice) and fill out the review form to the best of your ability. This means a couple of sentences devote to each of summarizing the paper, describing it's strengths and describing its weaknesses. In the "detailed comments" section (or something similar), you'll provide a detailed justification for your decision, including the reasons you think the paper is good and could be improved. This is usually between 2 and 5 paragraphs, depending on the paper. Finally, there is sometimes a section for questions that the authors should address in their rebuttal. If there is a question you have about the paper and it's possible that the author's response would change your mind about the paper, put it down. Be careful what you put here, because the authors are going to spend time and space responding to it.

There are a few things you should pay especially close attention to in the paper. The first is related work. Novelty is a critical aspect of research, and you need to be sure that what they have done is novel. If you are not an expert in the field that the paper focuses, you will need to do some additional reading to get up to speed. This could mean looking at up to 10 and reading (or at least skimming) 3-4 of the papers prominently cited in the paper and looking through their bibliographies for other papers that might overlap with the paper under review. You should also go to IEEExplore and the ACM Portal and type in some likely key words (The Portal is also great for quickly navigating through the bibliographies of the cited papers). All of this will take time, of course, and you could reasonably expect to spend 2-3 hours writing such a review. It might also be useful to ask someone who's an expert in the area for suggestions about related work, but be careful, you must not reveal any of the ideas in the paper under review. I will do my best to give you papers roughly in your area, but you may still need to do some of this digging.

What you send back to me should be in good enough shape to the members of the PC (i.e., it needs to be well-written, spellchecked, etc.). We'll set up a time to go over the review, discuss the paper, and what you thought of it. In most cases, I'll submit it along with any additional comments

Remember this:  The authors won't know who you are, but the PC will, so the quality of your review reflicts upon you (and me, since we are writing the review together).

It is critical that you hold the paper in confidence. The authors take a bit of a risk in submitting papers since it necessarily share unpublished ideas.

Reviewing the Paper

Read the following (Really. There's a lot too this process.):

Together they do a good job of laying out the referee's task, the questions, you should ask, and how you should approach writing reviews. I've provided a few notes below to augment it:

  1. Look for reasons to accept the paper rather than to reject it. There seems to be a trend in our field toward more negative reviews, and often they rely on critiques of the methodology or results as grounds for rejecting paper. Certainly, those are important, but the idea in the paper is usually its real contribution. If the idea is new and exciting, don't reject the paper because they used Spec2000 instead of Spec2006.
  2. Praise things that excite you. If you thought the paper was really interesting, say so. The paper will probably excite other people.
  3. Highly-polished papers don't necessarily deserved to be published. There are many well-written papers on that don't offer much in terms of insights, new knowledge, or excitement. We don't need any more of those.  Put another way: Being boring is a reason to get rejected.
  4. If they have missed related work, provide a citation. If there is an important piece of related work they should have cited but did not, you must provide a citation. It's not acceptable to say "I read something like this once..."
  5. Avoid inflammatory language like:
    • "Besides there is no novelty in the paper."
    • "Nothing significant can be concluded from the given experiments."
    • "But gives no idea about"
    Instead:
    • "It is difficult to identify the novelty in the paper."
    • "It is hard to draw significant conclusions from the given experiments"
  6. Don't make criticisms personal. Don't talk about "they" "you" or "the authors" unless your comment
    is praise. Otherwise it comes across as too inflammatory. Instead, talk about "the paper" or "the work".
  7. Out of scope papers.  Occasionally you review a paper that seems completely out of place in the conference at hand.  If you think this is the case, let me know.

Thanks to Michael Taylor for his comments and additions.