Web Submission and Review, Reviewer Documentation
Version 0.63 - June 2009
Table of Contents
- Initial access to the site
- Changing your password
- Specifying reviewing preferences
- Individual-review phase
- Reviews with attachments
- Working with scorecard files
- Discussion phase
- Discussion boards
- Your watch-list
- Program-committee votes
- Later access to the site
1. Initial access to the site
To access the review site you need to supply your email address and a
password. The chair or administrator will send you an email message with
the initial password that you should use for that purpose. Once you
access the site, you are presented with your review page. At the top
of the page there are a few "general purpose" links, looking more or
less as follows:
(Note that the chair may customize some of these links, so they may not
look exactly as the ones above.) These links will be displayed on all
the review pages that you access. On the page you can also find a button
that takes you to a list of all the submissions (except the ones that
chair explicitly blocked you from seeing), and once the chair assigned
submissions for you to review you will see these submissions listed
directly on your review page.
1a. Changing your password
Clicking the "change password" link takes you to a standard form for
changing password, where you need to provide your email address and
old password and repeat the new password twice. Note that as soon as
you submit this form your password changes immediately, and you will
be prompted to insert the new password before you can continue to
access the site.
1b. Specifying reviewing preferences
If the chair indicated that it want to see your reviewing preferences,
you will see the "Preferences" link that lets you specify them. This
link takes you to a list of all the submissions, and for each
submission you can specify a rank from 0 to 5 (where larger numbers
mean that you want to review these submissions more). The rank 0 is
reserved for submissions for which you feel that you have a conflict
of interests (e.g., ones for which you are a co-author). The chair can
use the preferences that you indicate to decide which submissions will
be assigned for you to review (or which submissions you should be blocked
from seeing). Please be aware that specifying too many 0's and
1's would the chair's task of assigning submissions to everyone
so much harder. The software will display a warning if you specify more
than half the submissions in these two categories.
At the top of the page you see a count of how many submissions you
have at each level, and these counts are updated every time that you
submit your preferences. You can always go back to the same page and
change your preferences.
2. Individual-review phase
Typically, the reviewing process begins with an "individual review
phase" in which the reviewers cannot see each other's reviews and
cannot discuss the submissions. During this phase, all you can do
is download the submission files and upload reports on individual
If the chair prepared an archive file containing all the submission
files then you will see a link to "download all submissions in one file".
You can also download individual submissions files using the "download"
buttons in the submission list.
You can upload reports to the site using the "Review" buttons in the
submission list. Once you upload a report on a submission, the "Review"
button changes to a "Revise" button that you can use to revise your
review. When submitting, you can check a box to have your review sent
to you by email, and another box that says "remind me to go back to
this review". The latter check-box only effects the color of the
"Revise" button: it is orange if this was checked and red otherwise.
2a. Reviews with attachments
If the chair enables it, you have the option of uploading a file as
an attachment to your review. This file is considered part of the
comments to authors: it will be attached to the comments that are
emailed to the authors at the end of the review process. This may be
useful if you want to make comments directly on the submission file
(e.g., as PDF comments), or if you want to add a drawing to your
review. Below are some considerations that you need to be aware of
when using this option:
- It is less convenient for other reviewers to access the attached
file than it is to see the text review: The text review is displayed
on the discussion boards, whereas the attachment appears as a link
that one needs to click on. Hence you should put as much as possible
in the text review, and use the attachment mechanism only when there
is a very compelling reason to do so. (The only exception is that small
text attachments are added to the text review rather than stored as a
separate attachment. Also, see the scorecard
option if you want to prepare your review as a separate text file.)
- Similarly, it could be less convenient for the authors to read
your comments. For example, the authors may need to have a specific
software to view an attachment in some proprietary formats. Moreover,
sometime even old versions of some editors cannot view documents that
were prepared with newer versions.
- Keep in mind that some editors (such as MS-Word) may embed your
name in the documents that you edit. By attaching a file that was
prepared using such an editor, you are risking leaking your name to
the authors. Take whatever steps necessary to prevent this leakage.
- Currently at most one attachment can be stored with the review.
To attach more than one file, you much pack all the files in one
archive file (e.g., zip or tar) and attach that one archive file with
2b. Working with scorecard files
Instead of uploading the reviews one at a time, you have the option of
preparing a file with many reviews (called a scorecard file) and
then upload all the reviews to the server at once. As a starting point,
you can download your current scorecard file and save it into a text
file. (This file contains all the reviews that you already uploaded, and
also empty reviews for the submissions that were assigned to you but
for which you did not yet upload a review.) Then you can fill that text
file with more reviews and upload it back to the server.
Please note that you cannot mark a review as "work in progress" or
attach any file to it when you upload it from a scorecard file.
3. Discussion phase
You move to the discussion phase once the chair decides to let you
discuss submissions with other reviewers. Your review page changes
to reflect that, adding a "Show reviews" button that lets you see
reports from other reviewers. The "Show reviews" button takes you
to a page that lists some review statistics for every submission
(e.g., grade average etc.) as well as a list of the grades given to
that submission by everyone who reviewed it. The "List submission"
button is still available, with the corresponding page adding for
each submission also its grade average and status. (The status of
a submission could be either "Accept", "Maybe Accept", "Discuss",
"Maybe Reject", "Reject", or "None".)
The submissions assigned to you are no longer listed on your review page,
but you have the option of defining a watch-list of
submissions that you want to be displayed on your review page (see below).
3a. Discussion boards
Next to each submission in the "List submissions" or "Show reviews"
pages you can now find a "Discuss" button that takes you to the
discussion board for tat submission. The presentation of that
button may change, looking like this if anything on the discussion
board changed since the last time you accessed it or like that
if nothing had changed.
On the discussion board for each submission you can find all the
reports on that submission as well as the discussion on it, and
you can post new comments to the discussion. You can choose if you
want to view the discussion as "threaded" (as in a newsgroup) or
"UNthreaded" (i.e., just a linear list of all the posts). Note that
the choice of threaded/UNthreaded view will affect your view of all
the discussion boards, not only the one that you are currently viewing.
3b. Your watch-list
As mentioned above, you can define a watch-list of submissions for
which you want to follow the discussion, and these submissions will be
displayed on your review homepage. To add or remove a submission from your
watch-list, just click the eye icon next to that submission in the
You can also add/remove submissions from your watch list by following
the link "Work with watch list". On that page you can also indicate
that you want to be notified by email on changes to the discussion
boards of submissions on your watch list. Checking this option will
result in an email message sent to you whenever anyone posts anything
in one of these discussion boards, or when the status of any of these
By default, the submissions on your watch list are displayed on your
review homepage ordered by submission-ID. You can override this default
from the watch-list page, and instead have them listed in the same
order that you specify for the submission-list page.
3c. Program-committee votes
The chair may ask you to vote on various topics during the program-committee
work (e.g., which papers to move from status "Discuss" to "Maybe accept",
or who should be the invited speakers at the conference). When the chair
sets up votes, you will see on your review page links that allow you to
participate in these votes. Following these links gets you to a voting
page where you can find further instructions and cast your vote. When the
chair closes the vote, it can make the tally visible to PC members, in
which case you will see a link on the review homepage to "See results of
4. Later access to the site
The chair can keep the review site up also after all the decisions were
made. At this phase, the site is operating in a read-only mode, where
you still have access to all the reports and discussions that you could
see during the discussion phase, but you can no longer revise them.
You have the option of getting a single page with all the reviews
and discussions (in html or ascii) to keep for your records.