Here are some questions to ask yourself when reading a paper. These questions are focused on discussions and presentations, but are equally important when reviewing (or writing) papers.
(Exception: do not try to connect your work to that of a paper you are reviewing. A review should be impartial, and a reviewer should not try to build on the work of a paper until it is published.)
Some of these questions have been influenced by the Heilmeier Catechism.
- What is the problem the paper is trying to tackle?
- As the authors explain the problem domain?
- How do the authors frame the introduction to make the problem compelling?
- ... Can you think of a broader domain where it's applicable?
- What is hard about the problem?
- What new tool, technique, or state of affairs makes it possible to solve the problem?
Is this paper primarily about a "hammer" (new technique), "nail" (new problem for old technique), new theoretical topic, or experimental study?
- What is a brief overview of the main idea, algorithm, or architecture?
- What questions need to be asked to evaluate the success of the contribution?
- What experiments are run?
- Do the experiments answer the key questions from above?
- How believable are the experimental settings, e.g., workload, implementation, etc.?
- Did the authors systematically investigate the different parameters affecting their solution?
- How well was the paper written?
Was it clear?
- Where could one build upon the paper's contributions?
[This one doesn't apply to reviewing papers:] Is there a way the work could be leveraged in your own project?