Questions when reading a paper

Here are some questions to ask yourself when reading a paper. These questions are focused on discussions and presentations, but are equally important when reviewing (or writing) papers.

(Exception: do not try to connect your work to that of a paper you are reviewing. A review should be impartial, and a reviewer should not try to build on the work of a paper until it is published.)

Some of these questions have been influenced by the Heilmeier Catechism.

  1. What is the problem the paper is trying to tackle?
    • As the authors explain the problem domain?
    • How do the authors frame the introduction to make the problem compelling?
    • ... Can you think of a broader domain where it's applicable?
  2. What is hard about the problem?
  3. What new tool, technique, or state of affairs makes it possible to solve the problem?
    Is this paper primarily about a "hammer" (new technique), "nail" (new problem for old technique), new theoretical topic, or experimental study?
  4. What is a brief overview of the main idea, algorithm, or architecture?
  5. What questions need to be asked to evaluate the success of the contribution?
  6. What experiments are run?
    • Do the experiments answer the key questions from above?
    • How believable are the experimental settings, e.g., workload, implementation, etc.?
    • Did the authors systematically investigate the different parameters affecting their solution?
  7. How well was the paper written?
    Was it clear?
  8. Where could one build upon the paper's contributions?
    [This one doesn't apply to reviewing papers:] Is there a way the work could be leveraged in your own project?