• It is good to review papers because:
  • paper reviews are the quality control of the field. When reviewing a paper, you are making a contribution to your profession.
  • you will better understand the criteria by which papers are evaluated and this will help you write better papers
  • it keeps you up-to-date on developments and trends in the field.
• Conference papers
  • are usually distributed to members of the Program Committee by the Program Chair(s). Distribution from there can take several forms. For example, each paper might go to more than one (3?) PC member. The individual PC members then distribute their papers to reviewers they select - one reviewer per paper.
  • are usually rated on a scale, such as 1-5, reviews averaged, and all papers listed in numeric order for the Program Committee or Program Chair. This list is usually broken down into 'accept', 'reject', 'stuff between'. Papers in this last category are then discussed individually by the PC and rejected or accepted on a paper by paper basis.
  • are usually reviewed as an all or nothing proposition - it's either accepted or it's not. Some conferences, however, allow a response by the author to clear up misunderstandings by the reviewers (usually not allowing additions to the paper).
  • are typically much shorter, less in depth, less revolutionary than journal papers. However, some conferences, such as SIGGRAPH, have standards as high as any journal.
  • are usually under pretty strict deadlines. The conference date is set and that dictates when decisions have to be made and reviews have to be in. At most, you might have is a couple of days leeway, but you should stick to the deadline if at all possible. Of course, if you have to be late, let the primary reviewer know as early as possible in the process.
• Journal papers:
  • are distributed by the Editor - each paper to one member of the Editorial Board (EB). The EB member then selects some number (3 in the case of TVCG) of reviewers to review the paper. The EB member collects the reviews and makes a recommendation to the Editor.
  • are reviewed in a limited-iteration process. Usually, you can recommend
    • reject, revise and resubmit, or some other variation that requires a new submission
    • a major revision - that will be re-evaluated by the original reviewers.
    • a minor revision - that will be checked by the EB member.
    • accept
    Usually only one major and one minor revision are allowed.
  • should make some quantum-leap contribution to the field, not just some incremental improvement to an existing technique - this, of course, is subjective.
  • usually have no real hard deadline for review unless it is targeted for a special issue. The editors try to move papers along at a reasonable pace, but if your schedule gets crazy, you can often get extensions. As always, ask for an extension as early as possible.
• Doing the review
  • Reviewing a paper should take several hours - but feel free to set limits on yourself.
  • Consider space limitations on the paper when considering what is and is not included in the paper itself (as opposed to the paper just citing references). It's often not possible to explain everything in a single paper and pointers must be given to some technical details.
  • Another tricky thing to evaluate is how much background knowledge the author is assuming the reader has. Typically the paper should be a written for a graduate student in the area.
  • Organize your comments from the general to the specific
  • Direct criticisms to the paper, not the author
  • Make your criticisms constructive and try to be encouraging - remember this is someone's career.

• Overall evaluation of the paper
  • Is it worth reading? Does it make sense to be a paper?
    • Is the problem important/significant to the readership? Is the solution non-trivial, non-obvious?
    • does the author know about existing techniques?
    • Is there a central, well-developed focus to the paper?
  • From the information provided in the paper, could you implement it? If not, why not?
    • Are important constants specified?
    • Is the technical presentation easy to follow?
    • Are there sufficient pointers (e.g. references) to 'black boxes' that are used in implementing the approach?
  • Evaluation, conclusion, contributions
    • Are there reasonable experiments to test the idea?
    • Is its performance compared against competing techniques? Are alternatives evaluated?
    • What is claimed in the 'Introduction'? Are claims supported by results?
    • Are the contributions of this paper clearly identified?
    • In the conclusion, are the limitations of the approach discussed?
    • If a paper extends previous work, is it clear what this paper contributes?
• Evaluating the presentation:
  • Are the mechanics of the presentation sufficient?
    • Are there problems with the use of English? Does it get in the way of understanding the paper?
    • Does it flow logically? Do you get lost in the description?
    • Does it give you a good high-level description as well as details?
    • Does it read well? Is it too verbose? Is it succinct and to the point? Is it well-writtten?
  • Are the equations presented well?
    • Some papers seem to try to blow the reader away by being overly-mathematical - if you can't follow the equations then it's not well-written - don't be intimidated by complex equations - you should still be able to follow the mathematical development if not the details. On the other hand, just because it doesn't have equations, doesn't mean it's a bad paper.
    • When possible, check the equations for correctness. If you can't because of time constraints, then note that in the review to the Editor so (s)he knows whats been checked and what hasn't. But be willing to spend some time reviewing the math - up to a point.
• Supporting material
  • Is the approach well-founded and justified?
    • If applicable, is there a 'theory' that is used as the foundation? Is there some foundation from another discipline?
    • Are important design decisions justified?
    • Are the references sufficient to find out about components used in the approach?
  • Are the references appropriate?
    • spend some time referring back to the references - make sure they are well formatted and that a citation in the text refers to something reasonable.
    • Suggest when a reference is needed - when you'd like to know the background for something brought up in the text

Some Resources

• Siggraph Workshop Paper Grinder
• http://eetd.lbl.gov/EA/Buildings/ALAN/PUBLICATIONS/how.to.review.html
• http://www.csee.usf.edu/~murphy/Students/reviewing.htm
• http://www.eecs.harvard.edu/~mema/courses/cs264/reviews.pdf
• http://www.eng.unt.edu/ian/guides/postscript/referee.ps