Firelily Designs - Sites - UPA Review System

The UPA review system auomated the process of submitting, reviewing, ranking, and selecting papers for the UPA conference, including feedback to submitters.

This is an example of a completely data-driven site; all of the content either comes from a database or gets stored into one. More than 200 individuals interact with the system in a variety of roles. See the task analysis for a more complete view of the scope of roles and responsibilities supported by the review system.

The review system was developed in PHP using MySQL as the database. The submission system, developed as a separate project, was written in Perl and access the same MySQL database.

	The process starts when people who are intrested in participating in the conference submit their proposals and papers. Proposals are sent to committee chairs, but key information about submissions are retained in a database, in preparation for reviews.
	As submissions come in, submission data is available to the conference committee, organized by submission type, and title. From this page, committee members can upload the actual submission documents for review (after a first review by the content chairs), and assign reviewers for the submission.
	The conference committee assigns reviewers to a submission from a pool of reviewers pre-selected for each category of submission. Available reviewers are filtered to prevent double assignments to the same submission, and are ranked by workload already assigned to them. The page also lists professional and employment information for speakers and reviewers, so that committee members have a check against conflict of interest in assigning reviewers.
	When a reviewer logs in to review their assigned papers, they have access to the papers they will review, as well as to the correct review forms for the type of submission they are reviewing.
	Each review form includes a series of questions with both ranking scores and comment text. Reviewers can save a partially completed review and return to finish it later. All review data is stored in a database on the UPA web server.
	As reviews are completed, the conference committee can see an aggregation of all reviews in a submission category, ranked by number of completed reviews, overall score, or averaged numeric scores.
	To evaluate a submission, the conference committee must first do a quality check of each review. This is done using the same review form that reviewers use to create their evaluations. The conference committee can edit comments if necessary to be sure that comments are appropriate for feedback to the submitters.
	When all reviews have been completed, the conference committee can look at an aggregate result from all the reviews for a single submission. This includes information about the submitter and the submission, averages and scales for all numerical scores, and all comments from the review form, grouped by question. Reviewers are anonymous at this stage.
	Submitters receive a subset of review data as feedback. They receive all comments (except for questions which are specifically identified to reviewers as "for committee use only"), but no numerical scores.