Web Submission and Review, Reviewer Documentation

Version 0.63 - June 2009

Discussion phase

Later access to the site

1. Initial access to the site

To access the review site you need to supply your email address and a password. The chair or administrator will send you an email message with the initial password that you should use for that purpose. Once you access the site, you are presented with your review page. At the top of the page there are a few "general purpose" links, looking more or less as follows:

[Guidelines] [Review Home] [List submissions] [Preferences] [Change Password] [Documentation]

(Note that the chair may customize some of these links, so they may not look exactly as the ones above.) These links will be displayed on all the review pages that you access. On the page you can also find a button that takes you to a list of all the submissions (except the ones that chair explicitly blocked you from seeing), and once the chair assigned submissions for you to review you will see these submissions listed directly on your review page.

1a. Changing your password

Clicking the "change password" link takes you to a standard form for changing password, where you need to provide your email address and old password and repeat the new password twice. Note that as soon as you submit this form your password changes immediately, and you will be prompted to insert the new password before you can continue to access the site.

1b. Specifying reviewing preferences

If the chair indicated that it want to see your reviewing preferences, you will see the "Preferences" link that lets you specify them. This link takes you to a list of all the submissions, and for each submission you can specify a rank from 0 to 5 (where larger numbers mean that you want to review these submissions more). The rank 0 is reserved for submissions for which you feel that you have a conflict of interests (e.g., ones for which you are a co-author). The chair can use the preferences that you indicate to decide which submissions will be assigned for you to review (or which submissions you should be blocked from seeing). Please be aware that specifying too many 0's and 1's would the chair's task of assigning submissions to everyone so much harder. The software will display a warning if you specify more than half the submissions in these two categories. At the top of the page you see a count of how many submissions you have at each level, and these counts are updated every time that you submit your preferences. You can always go back to the same page and change your preferences.

2. Individual-review phase

Typically, the reviewing process begins with an "individual review phase" in which the reviewers cannot see each other's reviews and cannot discuss the submissions. During this phase, all you can do is download the submission files and upload reports on individual submissions.

If the chair prepared an archive file containing all the submission files then you will see a link to "download all submissions in one file". You can also download individual submissions files using the "download" buttons in the submission list.

You can upload reports to the site using the "Review" buttons in the submission list. Once you upload a report on a submission, the "Review" button changes to a "Revise" button that you can use to revise your review. When submitting, you can check a box to have your review sent to you by email, and another box that says "remind me to go back to this review". The latter check-box only effects the color of the "Revise" button: it is orange if this was checked and red otherwise.

2a. Reviews with attachments

If the chair enables it, you have the option of uploading a file as an attachment to your review. This file is considered part of the comments to authors: it will be attached to the comments that are emailed to the authors at the end of the review process. This may be useful if you want to make comments directly on the submission file (e.g., as PDF comments), or if you want to add a drawing to your review. Below are some considerations that you need to be aware of when using this option:

It is less convenient for other reviewers to access the attached file than it is to see the text review: The text review is displayed on the discussion boards, whereas the attachment appears as a link that one needs to click on. Hence you should put as much as possible in the text review, and use the attachment mechanism only when there is a very compelling reason to do so. (The only exception is that small text attachments are added to the text review rather than stored as a separate attachment. Also, see the scorecard option if you want to prepare your review as a separate text file.)
Similarly, it could be less convenient for the authors to read your comments. For example, the authors may need to have a specific software to view an attachment in some proprietary formats. Moreover, sometime even old versions of some editors cannot view documents that were prepared with newer versions.
Keep in mind that some editors (such as MS-Word) may embed your name in the documents that you edit. By attaching a file that was prepared using such an editor, you are risking leaking your name to the authors. Take whatever steps necessary to prevent this leakage.
Currently at most one attachment can be stored with the review. To attach more than one file, you much pack all the files in one archive file (e.g., zip or tar) and attach that one archive file with your review.

2b. Working with scorecard files

Instead of uploading the reviews one at a time, you have the option of preparing a file with many reviews (called a scorecard file) and then upload all the reviews to the server at once. As a starting point, you can download your current scorecard file and save it into a text file. (This file contains all the reviews that you already uploaded, and also empty reviews for the submissions that were assigned to you but for which you did not yet upload a review.) Then you can fill that text file with more reviews and upload it back to the server.

Please note that you cannot mark a review as "work in progress" or attach any file to it when you upload it from a scorecard file.

3. Discussion phase

You move to the discussion phase once the chair decides to let you discuss submissions with other reviewers. Your review page changes to reflect that, adding a "Show reviews" button that lets you see reports from other reviewers. The "Show reviews" button takes you to a page that lists some review statistics for every submission (e.g., grade average etc.) as well as a list of the grades given to that submission by everyone who reviewed it. The "List submission" button is still available, with the corresponding page adding for each submission also its grade average and status. (The status of a submission could be either "Accept", "Maybe Accept", "Discuss", "Maybe Reject", "Reject", or "None".)

The submissions assigned to you are no longer listed on your review page, but you have the option of defining a watch-list of submissions that you want to be displayed on your review page (see below).

3a. Discussion boards

Next to each submission in the "List submissions" or "Show reviews" pages you can now find a "Discuss" button that takes you to the discussion board for tat submission. The presentation of that button may change, looking like this [Discuss*]

if anything on the discussion board changed since the last time you accessed it or like that [Discuss ]

if nothing had changed.

On the discussion board for each submission you can find all the reports on that submission as well as the discussion on it, and you can post new comments to the discussion. You can choose if you want to view the discussion as "threaded" (as in a newsgroup) or "UNthreaded" (i.e., just a linear list of all the posts). Note that the choice of threaded/UNthreaded view will affect your view of all the discussion boards, not only the one that you are currently viewing.

3b. Your watch-list

As mentioned above, you can define a watch-list of submissions for which you want to follow the discussion, and these submissions will be displayed on your review homepage. To add or remove a submission from your watch-list, just click the eye icon next to that submission in the submission list.

You can also add/remove submissions from your watch list by following the link "Work with watch list". On that page you can also indicate that you want to be notified by email on changes to the discussion boards of submissions on your watch list. Checking this option will result in an email message sent to you whenever anyone posts anything in one of these discussion boards, or when the status of any of these submission changes.

By default, the submissions on your watch list are displayed on your review homepage ordered by submission-ID. You can override this default from the watch-list page, and instead have them listed in the same order that you specify for the submission-list page.

3c. Program-committee votes

The chair may ask you to vote on various topics during the program-committee work (e.g., which papers to move from status "Discuss" to "Maybe accept", or who should be the invited speakers at the conference). When the chair sets up votes, you will see on your review page links that allow you to participate in these votes. Following these links gets you to a voting page where you can find further instructions and cast your vote. When the chair closes the vote, it can make the tally visible to PC members, in which case you will see a link on the review homepage to "See results of completed votes".

4. Later access to the site

The chair can keep the review site up also after all the decisions were made. At this phase, the site is operating in a read-only mode, where you still have access to all the reports and discussions that you could see during the discussion phase, but you can no longer revise them. You have the option of getting a single page with all the reviews and discussions (in html or ascii) to keep for your records.