Review Guidelines for TEST 2007
Schedule
- XXX to YYY: Individual review.
- Reviewers do not communicate with each other about the submissions. Please enter all the assigned reviews by the deadline.
- XXX to YYY: Discussion phase.
- Reviewers can see each other's reviews and discuss them using the review web site. Hopefully, we can decide (almost) everything by YYY.
- XXX: Program-Committee meeting.
- XXX to YYY: Wrap up.
- Time to check the comments again before they are sent to the authors.
Assignments
Most submissions are assigned to three reviewers (except PC-member papers that are assigned to more reviewers). Of course, you are allowed (and encouraged) to review papers that were not assigned to you, but please do so only after you reviewed your assigned lot. The chair(s) may also ask you to review additional submissions if the need arises.
External reviewers
You are encouraged to ask others outside the committee to help evaluate the papers. External readers should apply the usual confidentiality rules for conference submissions, in particular, not to distribute the paper to other people. It may be necessary to adjust an outside referee's score to make it compatible with your own system. You should understand the paper and external readers' comments well enough that you can discuss them in the public discussions phase. Please mark the names of the external reviewers in the subreferee box on the review form. After the final decision, the chair(s) will send out a list of all the sub-reviewers so you can check that none of your sub-reviewers were forgotten.
Do not forward your account name and password to a subreferee. The review site contains very sensitive information. The program committee discussions should remain confidential within the program committee.
Writing a Report
Each reviewer assigns a grade (1-6) to each reviewed paper to reflect the recommendation on acceptance/rejection for the paper, as well as a weight (1-3) to reflect the confidence of the reviewer in the recommendation. Grades for other specific criteria may also be entered. Please use the scale below when completing the review form.
- Overall grades:
- 1: Very weak submission, would be an embarrassment to accept
- 2: Weak submission, should be rejected
- 3: Rather neutral, but I lean toward rejection
- 4: Rather neutral, but I lean toward acceptance
- 5: Solid submission, should be accepted
- 6: Strong submission, would be a shame to reject
- Confidence levels:
- 1: An educated guess
- 2: Quite confident (but I did not check many of the details)
- 3: Confident (I know the area and I studied the paper in sufficient detail)
- Technical:
- 1: Low
- 2: Adequate
- 3: Good
- Editorial:
- 1: Poorly written
- 2: Could use some improvement
- 3: Well written
- Suitability:
- 1: Does not belong in this conference
- 2: Marginal w.r.t. the scope of the conference
- 3: Fits the conference
Report contents
Reports will consist of grades, a weight, and comments. The comments should help the authors as well as other committee members understand your opinions. Please don't write one-line reviews and don't over-use committee-only comments. There is no need to copy the comments to authors in the committee-only field. The bulk of the report should be visible to the authors (including summary of contributions, general evaluation, and specific comments). Authors of rejected papers deserve to know why their papers were rejected, and we want accepted papers to be improved based on our comments.
There is no additional structure to the reports that you have to follow. Here are a few points that you may want to refer to when organizing and writing your report.
- Summary of the problem and paper contribution.
- What is best about the paper: new ideas, proofs, simplifications, formalizations, implementation, performance improvement, new insight, etc.
- What are the paper weaknesses: lack of originality, small increment over previous work, unsubstantiated claims, bad presentation, insufficient discussion of relation with prior work etc.
- Target audience: who will be interested in the results, who will be benefited from its publication in the proceedings, who will want to hear the talk at the conference.
- Summary of recommendation: Accept/Reject, main reason.
- Detailed technical comments.
Contacting authors
If you need to contact the authors of a submission, you can do this either through the chair(s) or directly. In the latter case, please let the chair(s) know that you contacted the authors.
The Discussion Phase
Please use the web site discussion boards for the discussions (and not the e-mail list); this will ensure that people who join in the discussion later can follow its history. (Conversation on the discussion boards are not accessible to program committee members who are authors of the paper in question or otherwise have a conflict of interest.)
Have fun.
The chair(s)