Double blind reviewing

This year, we will use double blind reviewing. The authors do not know the identity of the reviewers; this also holds for authors who are on the program committee. In addition, the reviewers do not know the identity of the authors.

You should not actively attempt to discover the identities of the authors. Based on the experience of other double-blind conferences, we caution reviewers that the assumed authors may not be the actual authors; multiple independent invention is common and different groups build on each other’s work.

If you believe that you have discovered the identity of the author, please let us know in the “Confidential remarks for the program committee” in your review (see below).

We are also implementing the paper bidding feature to ensure that reviewers get affected papers fitting their expertise. To get more details about the bidding instruction, go to the the Bidding Instruction page.

Review Criteria & Numeric Scoring Guide

Your review should be constructive, thorough, and polite. Please take the time to fully assess the papers assigned to you.

We use six criteria to judge a paper: Relevance, Significance, Novelty, Clarity, Soundness and Evaluation. You should also provide detailed comments justifying your evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their author response.

Please find below some guidance on the meaning of the criteria and their numerical scores. 


Summary. In your own words, please summarize the main claims/contributions of the paper, relating the ideas to previous published work.

This part of the review may not provide much new information to authors, it is invaluable to members of the program committee, and it demonstrates to the authors that you understand their paper.


Relevance.* What is the relevance of this paper to the conference?

 4: Likely to be of interest to a large proportion of researchers

 3: Relevant to researchers in subarea only

 2: Of limited interest to the conference

 1: Not relevant


Significance.* Are the results significant?

 4: Highly significant

 3: Significant

 2: Moderately significant

 1: Not significant

Are the results important? Are other researchers likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique approach?


Novelty.* Are the problems or approaches novel?

 4: Very novel

 3: Novel

 2: Moderately novel

 1: Not novel

Are the problems or approaches novel? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?


Clarity.* Is the paper clearly written? Is it well-organized? Does it adequately inform the reader?

 4: Excellent

 3: Good

 2: Satisfactory

 1: Poor

Is the paper clearly written? Is it well-organized? (Feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A superbly written paper provides enough information for the expert reader to reproduce its results.)


Soundness.* Is the paper technically sound? 

 3: Technically sound

 2: Has minor errors

 1: Has major errors

Are the concepts correct and accurate? Do you think that the paper has minor/major errors?


Evaluation.* Are claims well-supported by theoretical analysis or experimental results?

 4: Very convincing

 3: Sufficient

 2: Somewhat weak

 1: Not convincing

How convincing is the evidence in support of the conclusions? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work?


Overall evaluation.* Please provide an overall score for the submission.

 3: Strong accept

 2: Accept

 1: Weak accept

 0: Borderline paper

 -1: Weak reject

 -2: Reject

 -3: Strong reject

To decide on an overall evaluation grade, you should consider how you graded the paper based on the Relevance, Significance, Novelty, Clarity, Soundness and Evaluation. If the submission is not a full paper, please consider the nature of the submission (extended abstract, poster) to give your score.


Detailed comments.* Please elaborate on your assessments and provide constructive feedback.

Please use this area to provide rationale for the above criteria ratings. Also use this area for other comments, questions, and suggestions for how to improve the paper. Consider trying to articulate the strengths and weaknesses of the paper in advance of forming your overall assessment.


Questions for authors.* Please provide questions for authors to address during the author feedback period.

Please provide questions for the authors to address during the author feedback period. If there are things you think are incorrect or unclear, this is your opportunity to have them addressed.


Reviewer’s confidence.* Please enter your confidence about your review.

 5: (expert)

 4: (high)

 3: (medium)

 2: (low)

 1: (none)

This has to do with how knowledgeable the reviewer is on the paper’s topic. Please give an objective assessment of your confidence. This is not visible to the authors.


Confidential remarks for the program committee. If you wish to add any remarks intended only for PC members please write them below.

This field is optional and only visible to the program committee. It is not visible to the authors. Please enter any of your confidential remarks and concerns about the submission.