Web Submission and Review, Chair Documentation

Version 0.63 - June 2009

1. Installing the software

One of the goals of this software was to make administration as easy as possible, so you will not need to run scripts etc. In principle, to install this software you just unpack the zip file in some directory in your web tree (call it the BASE directory), point your browser at that directory and off you go. However, this simple method of installation requires the web-server to be able to write files in the web-tree, which in some cases is considered a security risk. I therefore provide also a non-web-based installation method, as explained below.

Prerequisites. This is a PHP/MySQL software, which means that to use it you must have a MySQL server and a PHP engine (and your web-server must supports PHP and MySQL). I tested the software with PHP 4.3.9 and MySQL 4.1.20 (and earlier versions of it were tested with PHP 4.3.2 and MySQL 3.23). It may work even with older versions but I cannot test it. Also, proper functioning of this software depends on "proper" functioning of you web-server. In particular, some issues that I came across are:

The web-server (and the PHP engine) must be willing to upload files. For example, in one system that I used to test it, the PHP engine was configured to only allow uploads of files upto 500KB (and I was wrecking my brains as to why I can only upload pdf and not postscript files..) In my case it turned out that I had to change the parameter LimitRequestBody in php.conf (which on that system was somewhere under /etc/httpd/).
The web-server must be able to send email. For example, if this is a Windows machine then the variable SMTP in php.ini should be set correctly.
If the server is set-up with PHP in safe mode, then everything under the BASE directory must have same owner. (Also everything under the UPLOAD directory -- see below.) If safe_mode_gid is set in php.ini then instead of having the same owner they must have the same group.

Some of the functionality of this package also depends on the availability of external programs. Currently the only parts of the functionality that need external programs are preparing an archive file with all the submissions (which needs either the Archive_Tar PEAR library or the tar or zip programs), and stamping the submissions with a header saying "do not ditribute" (which needs auxilary program to convert between PDF and Postscript). See some more details in the implementation notes.

Experimenting with the software. I wrote a small testing package for the purpose of experimenting with the software. To use this package, copy the contents of the "testOnly" subdirectory to the BASE directory where you installed the software itself before you customize the installation. This testing package adds 53 dummy submissions and 17 dummy PC members to the conference database.

Troubleshooting. Most of the parameters of the software are stored in the "parameters" table in the database. The details of the database itself are stored in the file init/confParams.php under the BASE directory. If you have problems with the software, looking at these parameters may provide some hints as to the source of the problem.

1a. Web-based installation

To use this method, simply copy the contents of the webtree directory to somewhere in your web tree (call it the BASE directory), making sure that the web-server has permission to write in the "init" sub-directory of the BASE directory. After copying the files to the BASE directory (and setting the permissions), point your browser to the page chair/initialize.php under the BASE directory. In this form you need to specify various details that are needed for initializing the new site:

Web Server. The DNS name or IP address of the web server, as accessed by the users of this software. This parameter is used when the software sends emails and needs to specify the address where some pages can be found. (Note that in some cases the administrator may use a different address. For example, the administrator uses "http://localhost/confName/" while everybody else uses "http://www.severName.org/confName/". In this case the chair needs to enter the latter address on the customization page.)

MySQL Server. This parameter tells the PHP scripts where to look for the MySQL server. In common installations the same host runs both the web-server and the MySQL server. If this is not the case (i.e., the web server communicates with a back-end database that is hosted on a different machine) then you need to check the radio-button for "The MySQL server runs on a different host" and provide the DNS name or IP address of the machine that hosts the MySQL server.

MySQL Administrator or MySQL Database and User. There are two ways to create a database for use with the submission and review software:

If you have administrative access to the MySQL server, you can simply enter the administrator username and password and the software will automatically create a new database. The "administrator" whose details are thus specified should have sufficient privileges to DROP and CREATE databases and to GRANT access privileges on these databases.
If you do not know the username/password of the MySQL administrator, you should create the database for this conference by some other means. (For example, some hosted sites have a "control panel" that lets users create new databases. In other sites one must ask the administrator to create new databases.)

Once you created the database, enter in the form the database name and the MySQL username/password that are needed to access it.

UPLOAD directory. This is the local directory where the uploaded submissions would be stored. If you do not specify anything in this field, the UPLOAD directory defaults to the subdirectory "subs" under the BASE directory. The UPLOAD directory need not be in the web-tree, but the web server must be able to read and write in that directory.

Program Chair email. In this field you can specify either just an email address or also the name (in the form Name <email>). If you do not specify the chair's name, it will default to something like "<conf-name> Chair". The email addresses that you specify here will be used as the program chair's username for the purpose of accessing the review site.

The chair's email will also be CC'ed on every email that is sent to authors of submitted papers, including submission/revision confirmations, the final decision notifications and the comments for authors. The address that you specify here can later be modified from the customization page, and also from the PC membership page.

Administrator email. The administrator's email is rarely used by the software: it's main purpose is to serve as a backup address for the chair's email. For example, if the need arises to reset the chair's password, then an email with the new password will be sent to both the chair and the administrator email addresses. Also, there are some rare error situations where the software would sends an email to the administrator. After it is set, the administrator email can only be changed by manually editing the file init/confParams.php under the BASE directory.

After Initialization

Once you hit the Submit button at the bottom of the initialization form, you are taken directly to the customization page in order to customize the site for the conference (see below). At this point the web server no longer needs to write to the "init" subdirectory, so you can change the permissions to make that directory read-only.

1b. Non-web-based installation

An alternative method of installation (in Unix systems) is to change directory to where you unzipped this software (above the "webtree" and "tools" directories) and to issue the following command from the command prompt:

  % ./tools/mkNewConf.sh confname

where confname is a short-name for the conference (e.g., xyz2007). It is recommended that you read the script "mkNewConf.sh" before running it, and you can optionally modify it to set some default values.

This shell script will ask you for the locations of the BASE and UPLOAD directories (and other parameters) and then it will install the software in the BASE directory, setting the permissions as needed. The script will then print out the username and password that are needed in order to access the customization page.

2. First-time customization

Before it can be used, the site still needs to be customized for the conference. This is done from the page chair/customize.php under the BASE directory. This form lets the program chair specify the conference name, deadlines, program-committee members and many other parameters. Links are provided from this form to this documentation page, where you can find some additional explanations about the role of different parameters. All the parameters that are specified in this form can be modified later from the administration page. Some fields in the customization form that require more explanation are described below:

Pre-registration. Some conferences require authors to register their submission before actually submitting it. Often the deadline for registration is 1-2 weeks before the submission deadline. If you leave the field for pre-registration deadline empty, then the software will assume that pre-registration is not required.
Supported Formats. The software is very permissive in terms of handling formats other than the ones that you specify here. If a submitter uploads a file in an unsupported format, the only thing that happens is that the extension unsupported is added to the file name. (For example, uploading a file xyz.txt, the file will be stored as xyz.txt.unsupported.
Program Committee. In this section you must enter the PC chair's email address, and you can optionally also specify the names and email addresses of the program committee members. For each PC member (or chair) you can specify either just an email address or also the name (in the form Name <email>). If you do not specify the chair's name, it will default to something like "<conf-name> Chair". For the other members, if you do not specify the name then it will remain blank.

The email addresses that you specify here will be used as the username for the purpose of accessing the review site. The names will be displayed on the reviews and discussion boards. All the names and addresses that you supply here can later be modified (or added or removed) from the administration page.
Reviewer Preferences. If you check this box, PC members will see a link on the review page asking them to specify their reviewing preferences. The chair will then be able to see all these preferences and can take them into account in assigning submissions to reviewers. Moreover, the chair will be given the option to let the software automatically compute an assignment or submissions to reviewers based on these preferences. UNchecking this box will turn off reviewer preferences, and also disable the automatic assignment option.
Reviewer Attachments. Check this box to allow the reviewers to upload a file as an attachment to their review. This file is considered part of the comments to authors, and will be attached to the comments that are emailed to the authors at the end of the review process. Click here for some considerations for and against allowing this option.

After Customization

Once you hit the Submit button at the bottom of the customization page, you are taken to a confirmation page where you can once again verify all the parameters. Then the software is customized with these parameters and you are presented with a receipt page. That page includes the chair's password and links to the submission page and the administration page. The submission site is now ready to receive submissions, and you can use the administration page to manage it.

3. The Administration Page

The administration page is what you use the manage the site and the entire submission and review process. The links on that page change according to the different phases of the process. You have one set of links during the submission phase, another during the review phase, and yet others during and after the final-version submission phase. You also have some "general purpose" links there for changing your password, looking at the error-log file, reading the documentation, etc.

You also have an "Undo/Redo" link that lets you either undo the last change that you did from the administration page, or redo the last change that you undid. For example, if you closed the submission site and the decided to reopen it, you can use the Undo link to do that.

4. Managing the submission site

The administration page for the submission site is fairly simple. The main things that you can do from that page are:

You can see a list of all the (non-withdrawn) submissions.
You can review and modify many of the submission-site parameters that you specified during customization, including the conference submission deadline.
You can tweak some parameters of the web-site (mainly email settings) as described below.
If the conference requires pre-registration of submissions, then you have a link to close the pre-registration. This prevents new submissions from being registered, but does not close the submission site: Authors who pre-registered earlier can still submit (and revise) their papers.
Finally, when the submission deadline passes you can close the submission site and activate the review site. Clicking this last link will take you to a confirmation page, where you can confirm that you really want to close the submissions. The same page also lets you modify some of the review parameters that you specified during customization.

You will also see on the administration phase links to some pages that deal with review-period activities, such as setting the review guidelines, managing PC membership and setting up PC votes. These are described in more details below.

4a. Tweaking Server Settings

This form is meant to provide some workarounds common bugs and configuration errors (mostly related to email). It currently consists of the following options:

Modify the server time. This provides a crude work-around in case the server′s internal clock is off. Here you can specify some "time shift" amount that will be added to the server's time.

Email header-line separation. RFC 2822 specifies that different lines in the email header should be separated using the two characters \r\n (CRLF). However, the documentation for the PHP mail function contains the following comment:

Note:If messages are not received, try using a LF (\n) only. Some poor quality Unix mail transfer agents replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with RFC 2822.

Accordingly, the default setting that is used in the software is CRLF, but you have the option to override that default.

X-Mailer Header Line. Sending this header line is considered standard net etiquette, but some spam-filters out there mark as spam anything that has this header line. UNcheck this box if this causes email loss.

Sender Header Line. If you know how the web-server is calling itself on email messages that it sends, you can specify it here. This is often something like userName@machineName where userName is the username of the web-server (e.g., www, apache, admin, etc.) and machineName is either the DNS name of the machine (e.g., www.mit.edu) or sometimes just localhost.

When the web-server sends email, the name of the web-server itself is set as the "envelope sender" of this message. It was observed that some ISPs filter as spam email messages where the From: header line does not match the "envelope sender". Hopefully specifying a matching "Sender:" line would prevent these emails from being discarded.

Specify "envelope sender". Sometimes you can explicitly specify the "envelope sender" to be used, by supplying the parameter "-f name-to-use" to the PHP mail function. This only works on (some) Unix systems, and does not work when PHP is in "safe mode". Also, using this option typically adds an "X-Warning:" header line to the message, which can again increase the odds of this message being discarded en-route. The PHP documentation has the following comment about this method:

The user that the web server runs as should be added as a trusted user to the sendmail configuration to prevent a 'X-Warning' header from being added to the message when the envelope sender (-f) is set using this method. For sendmail users, this file is /etc/mail/trusted-users.

5. Managing the review site

Once you close the submission site, you are taken back to the administration page, where you can now administer the review site. At the top-right corner you still have a "Submission-site" box, now consisting only of a few links that let you list the submissions is various orders (and also revise or withdraw submissions if need be). You can even submit new papers (just in case you want to accept for consideration a submission that arrived after the deadline.)

5a. Initial set-up

One of the first things that you would want to do after the deadline (if not before) is letting the PC members access the review site. This is done from the "Manage PC membership" page. On that page you can add or remove PC members, reset the password of a PC member and modify the email address or the name that is recorded in the database for each member. The PC members can login to the site using their email address and password. When you reset the password of a PC member, an email message is sent to that member with the new password. (Also, changing the email address of a PC member automatically resets his/her password.) Note that it is possible for several people to use the same email address as long as they use different passwords (but all the passwords will be sent to that one email address).

Other things that you can do from the administration page is to create an archive file with all the submissions (so that reviewers do not need to download each submission separately), and edit the review-guideline page. The review guidelines page consists of the grade semantics, important dates in the committee work (such as the individual review deadline), and other review policies.

5b. Paper assignments and blocking access to papers

The software also lets you assign some submissions to each reviewer or block reviewers from accessing specific submissions (e.g., blocking PC members from accessing the reviews of their own papers).

If you did not disable that option, then PC members are given the option to access a Reviewing-Preferences page where they can specify what submissions they want or do not want to review, and they can also specify if they feel that they have conflict of interest with some submissions. All these preferences are made available to the chair, should you want to use them in assigning or blocking access to submissions. Note that access to a submission is NOT blocked automatically when the reviewer specifies a conflict-of-interests. The chair must explicitly block access from the Edit conflicts page.

On the Edit conflicts page you are presented with a list of all the PC members, and for each member you are given a list of the submission-ID's that that member specified as having conflict-of-interests with. For each PC member you can then specify a list of submissions that this member should be barred from seeing. You are then asked again to confirm these choices, and once you do the submissions that you specified are removed from the view of that PC member. (You can access the Blocking-Access page at any time during the review process, should you need to modify these decisions.)
For the task of assigning submissions to reviewers, you are given the choice to either assign everything manually, or to let the software compute an assignment (using the reviewer's preferences) that you can later adjust.

Manually Assigning Submissions to Reviewers

There are two interfaces that you can use to manually specify the reviewing assignments. You can use either of these interfaces as you see fit, and once you "save the assignments" with one interface, the other interface is automatically updated as well.

The matrix interface: This interface consists of a matrix of check-boxes, with one column for each PC member and one row for each submission. Throughout each column there are running headers that lets you see the number of submissions that are assigned to that PC member as well as a heuristic indication as to "how happy" this PC member will be with this assignment. Also, at the right end of each row you can see how many reviewers are assigned to that submission. If you have Javascript enabled in your browser, then these sums and indicater are updated every time that you check or clear a check-box. Note, however, that you must use the "save assignments" button to have these changes recorded by the server.

The heuristic "indication of happiness" is computed as follows: The software computes the fraction of requested papers from all the assigned ones, and the faction of assigned papers from all the requested ones, and the "happiness level" is taken as the largest of these two fractions. A level below 40% is "Angry" , 40-59% is "Sad" , 60-79% is "Satisfied" , and 80% and up is "Happy" .

Every entry in the matrix also displays the preferences that the PC member specified for that submission, encoded as a number between 0 and 5 and colored from red to green (if nothing appears in the entry, it means that the PC member specified the default number 3). Also, if you specified some preferences of your own from the chair-preferences page, then the checkbox itself will be colored either red (should not review) or green (should review).
The submission-list interface: This interface consists of a list of submissions, where for each submission you can specify a list of PC members that are assigned to that submission.

Automatically Computing Paper Assignments

If you did not disable that option, you will find a link off the main assignment page that points to the auto-assignment page, where you can have the software compute an assignment of submissions to reviewers. The software uses an algorithm based on maximum-flow to generate this assignment, as described next.

In addition to the reviewers specifying their preferences, the chair can specify preferences as to who should (or shouldn't) review what submission. Note that specifying that "I want Joe to review submission 53" is different than manually assigning submission 53 to Joe. Specifically, this is just a hint to the software and is not binding (which means that you can specify as many preferences as you want without having to worry about load balancing and such like). The chair preferences are used to modify the reviewer preferences during the auto-assignment procedure as follows:

When the chair indicates that a certain PC member SHOULD NOT review a certain submission, the auto-assignment procedure will decrease that preference. Denoting the preference-level that the reviewr assigned to tha tsubmission by P (between 0 and 5), then the auto-assignment procedure will treat it as max(0,P-2).
When the chair indicates that a certain PC member SHOULD review a certain submission, the auto-assignment procedure will increase that preference. Denoting the preference-level that the reviewr assigned to tha tsubmission by P (between 0 and 5), then the auto-assignment procedure will increase it to min(6,P+1) if P was at least 3, and will leave P unchanged otherwise.

Note that after the adjustments due to the chair preferences, the auto-assignment algorithm treats the reviewer preferences as taken from the range 0-6. Another adjustment that is made is that for reviewers that do not have any submission at preference levels 5/6, all their preference-4 submissions are elevated to preference-5. The auto-assignment algorithm will not assign to a submission to a reviewer if the (adujsted) preference level for it is less than 3.

After adjusting the preferences, the software prepares a flow network graph where each unit of flow from the source to the sink corresponds to an assignment of a submissions to a PC member. The graph has a source node, a sink node, and also four nodes for each reviewer (correpsonding to pref6 → pref5 → pref4 → pref3) and one node for each submission.

There is a capacity-1 edge from prefP to every submission for which the reviewer has (adujsted) preference level P.
The capacity of the edges source → pref6 and pref6 → pref5 is the maximum load that can be added to this reviewer. That is, if each PC member can be assigned upto 15 submissions, and you already assigned two submissions to that PC member, then the capacity on these edges will be set to 13.
The capacity of the edge pref5 → pref4 is the load minus something, to force at least a few assignments with pref-5/6 for each reviewer. The "something" is set to half the smallest of the number of pref-5/6 submissions of this reviewer and the global load, rounded down.
The capacity of the edge pref4 → pref3 is set to at least 20% of the load. However, it could be increased to make sure that the total of pref4 → pref3 capacity plus the number of pref-4/5/6 submissions is at least 120% of the total load for that reviewer.
From each submission there is an edge to the sink with capacity equal the number of reviewers that should be assigned to that submission, minus the number of reviewers that are already assigned to it.

After setting this network, the assignment is computed in several steps. First, a maximum flow is computed using only the pref-5/6 nodes, then the pref-4 nodes are added and the flow is augmented, and finally the pref-3 nodes are added and the flow is augmented again. The maximum flow in each step is computed using the Edmonds-Karp/Dinic algorithm [1], which prefers short flow-paths over long ones. It is expected that this process will assign submissions at higher preference levels when possible, and "resort" to preference-level 3 only when there is no other option.

Parameters of the algorithm. Once you got the preferences from all the PC members and also specified your own preferences, you are ready to let the software compute the initial assignments for you. Before the algorithm is run, however, you have to specify some parameters. In particular, you can specify a list of excluded reviewers that will be hidden from the view of the algorithm. This means that the algorithm will not assign any submissions to these reviewers. However, submissions that you assigned to the manually before running the algorithm will be kept, and will be counted against the number of required reviews per submission. You can use this option, for example, if one of the "PC members" on your list is the PC chair (to whom you do not want to assign any submissions).

Other parameters that you need to specify are how many reviewers are assigned for each submission and whether or not you want to keep existing assignments (if any) or start from scratch. Once you specify all the parameters, the software automatically computes an assignment of submissions to reviewers, and you are taken to the manual-assignment page where you can review and modify this assignment.

Sketch Assignments vs. Visible Assignments

The assignments that are automatically computed by the software or set from the manual assignment page are stored as a "sketch copy" that is only visible to the chair but not to the reviewers. This allows the chair to play around with the assignments before making them publicly available.

To allow the PC members to see their assigned submissions, the chair must check the box to "Make these assignments visible to the reviewers" next to the "Save Assignments" button on either the matrix or the list interface. Hitting the "Save Assignments" button with this box checked will copy the sketch assignments to the visible assignments, thus letting the reviewers see their assigned papers.

Note: As opposed to paper assignments, setting or clearing conflict-of-interests status for papers will immediately be visible to the reviewers.

5c. Tracking progress

After you assigned the reviewing duties, you can track the progress of the reviewing process off the administration page. The Overview page shows a matrix of all PC members and submissions, and each entry in that matrix shows whether a PC member is assigned to a submissions and whether it entered a review of it. If the PC member entered a review for the submission, you can click on the entry to see that review. At the bottom of the page you can manage the "discuss flags" that let PC members see each other's comments and participate in the discussions.

At any time you can go to the Set Submission Status page where you can modify the status of any submission. The available status values are the usual ones, namely "No status", "Reject", "Maybe Reject", "Discuss", "Maybe accept", and "Accept". At the top of the Overview and Set-Status pages you can see how many submissions are currently assigned any of the status values.

Also, at any time you can also go from the Administration page to the Review page (via the links at the top and bottom of the page), where you can participate in the review and discussions just like any other PC member.

5d. Program-committee votes

The software includes support for setting up and managing program-committee votes. You can have the committee vote on things such as "what papers to move from 'Discuss' to 'Maybe Accept'", or "which of the accepted papers should receive the best paper award", or even "who will give the invited talk at the conference". Managing the votes is done from the Voting page. On that page you can set the parameters for a new ballot, edit the parameters of the ballots that are currently in progress, and see a list of all the ballots (in progress and otherwise) with a link to the result page for each one.

You can set up a ballot to vote either on submissions or on "other things" (that you need to specify explicitly). Note that PC members can only vote on submissions when they are in the public-discussion phase of the review. Before that they can only vote on "other things" (e.g., invited speakers). Also, PC members will not see entries for submissions that are blocked for them, even if these submissions are included in the current vote.

When voting on submissions, you can either include all the submissions in the vote, include only submissions from specific categories (e.g., only submissions in the "Discuss" category), or explicitly list the submissions that you want to include in the vote. Note that if you choose to include in the vote, say, all the submissions in the "Discuss" category, then the submissions that the PC members see when they vote are the ones in the "Discuss" category at the time that they are voting. Put in other words: adding or removing submissions to the "Discuss" category in mid-vote will add or remove these submissions from the ballot.

When you set up ballots, the PC members will see a link from their review page that lets them participate in the current ballots, and you will be able to track the voting results off the Voting page. You can also modify the parameters of a vote, even when this vote is already in progress. Note that if you modify the list of submission-IDs or the list of "other things to vote on" in mid-vote, you must make sure that you do not modify the order of items, since the software identifies vote-items with their position in the list. For example, swapping the order of two items will result in each of them being assigned the tally of the other.

When you close a vote, the tally becomes visible to the PC members via a link off their review homepage. Note that only the tally is visible to PC members, not the individual votes. Also, reviewers cannot see the tally for submissions for which they have a conflict. You can prevent even the tally from becoming visible by checking the box to "Hide tally from PC" (next to the "Close vote" button) when you close the vote.

5e. Wrapping up the PC work

After all the decisions are made, the software lets you send to the authors the notification of decisions and the comments-to-authors that were entered during the review period. Just follow the appropriate link off the administration page, and you will be given the opportunity to edit these emails before they are sent. In addition to the contact authors, the chair is also CCed on all these emails.

Note that the Notification and Comments pages only send emails to contact authors of submissions whose status is either "Accept" or "Reject". These pages give you the choice of either sending emails to the contact authors of all the accepted and rejected submissions, or only to specific submissions that you explicitly list. This can be used to hold off on sending emails to some submissions (by setting their status to something other than "Accept" or "Reject") and later assigning the correct status and specifying these submission explicitly.

If you want to keep all the reviews and discussions for future reference, you can find a link off the administration page that packs all of them in one text ascii page that you can then save for your files.

You also have a link to a page that lets you edit the final-version preparation instructions (for authors of accepted papers), and a link to close the review site and activate the site for final-version submission.

Once the review site is closed, PC members can still access the site and read the comments and discussions, but they can no longer modify them.

6. Managing the final-version submission site

After you activated the final-version submission site, you still see a "Review site" box at the top right corner of the administration page, that lets you send emails to authors or change the status of submissions if need be. Other than that, you can see a list of accepted papers (with links to the final version when it is uploaded), create (and then download) an archive file containing the final versions of all the accepted papers, or send email to the authors of all the accepted papers (e.g., to remind everyone that the deadline is in three days).

When the camera-ready deadline arrives, you can shutdown the final-version submission site. After you do that, you are given yet another link that automates the creation of the various indexes for the proceedings. Specifically, you are given the option of specifying the order in which the papers will appear in the proceedings and the length of each paper, and then the software generates for you a LaTeX file (using Springer llncs style) that includes the external-reviewer list, the list of program committee members, the table-of-contents and the author index.

6.a Adding invited talks to the program

During the final-version submission phase you can add to the program invited talks. Doing so will allow the invited speakers to upload their talk to the server, to be included in the conference proceedings just like any other accepted paper.

7. Feedback

I will very much appreciate any comments that you can give me regarding this software. Please contact me at shaih@alum.mit.edu.