Paper critique

Your grade in this course will be determined on a semester-long paper critique project. You need to pick a paper at the start of the semester, and during the semester use the material we cover to analyze and evaluate statistical choices made by the authors of that paper.

At the end of the semester, you will have produced an essay that will contain

  1. A description of the objective of the paper
  2. A comprehensive description of the statistical methods chosen by the authors
  3. An evaluation of whether the methods chosen were appropriate.
    If you find that the authors made mistakes in choosing their analysis methods, you should suggest better choices.
  4. A reproduction of the statistical analyses made by the authors: compute the estimators they computed, and compare with the values the authors reported.
    If a reproduction is not possible, you should instead write a comprehensive description of what the authors should have provided to enable your reproduction of their analysis: is the problem missing data? missing descriptions of data cleaning, or post-processing, or of exact methods chosen?

You may optionally include, in order to raise your grade

  1. A reproduction of the statistical analyses using the methods you deem should have been chosen.
  2. Compare and contrast different methods that could have been applicable, and state their respective merits and drawbacks.

Evaluation criteria

Each of the four paper tasks tasks is graded on a scale of A/C/D:

D is awarded for a completed report, with no more than 4 minor flaws. Any submission that misses the course deadline will be awarded this grade.

C is awarded for a completed report without any flaws.

A is awarded for a completed report without any flaws that also contains a suggestion and evaluation of alternative choices the paper authors could have made.

Final grade is the arithmetic mean of these scores, on a scale of A=4.0, B=3.0, C=2.0, D=1.0.

Each of the additional two tasks provides 0.25 to the total.

Minor flaws

As minor flaws count:

Major flaws that award a failing grade to the paper and thereby the course include:

Rubric

Task 1: Objective of the paper

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Describes in your own words what the paper is trying to achieve.

A

Describes in your own words what data the authors are analyzing, how it was acquired, and evaluate whether their experiment designs or sampling designs are appropriate for their task.

Task 2: Statistical method description

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Describe their entire data cleaning and data analysis process in as much detail as the paper admits.

A

Qualified and motivated guesses for the steps the authors leave out of their descriptions. For example: a paper that only presents a ± error could have reached it with or without a particular number of standard deviations, or particular confidence level, with a normal or a t-distribution as basis. If the authors do not state enough details, fill in the blanks and give an argument why your suggestions are plausible.

 Task 3: Evaluation of whether methods were appropriate

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Identify what aspects of the method choices depend on features in the data: is there a requirement that data be normally distributed? have sufficiently many observations?

Report the extent to which and the details of how the author has handled any method prerequisites.

A

Qualified and motivated guesses for what the results of correctly testing for applicability of their chosen methods.

Qualified and motivated suggestions for better methods if any of their chosen methods are lacking in prerequisites.

 Task 4: Analysis reproduction

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

If the authors provide data

Recompute any estimators and test statistics reported by the authors.

Recompute any p-values or confidence intervals reported by the authors.

Compare your results with the results reported by the authors. Comment on the comparison.

If the authors do not provide data

Identify what data was available, and what data was not available: did they give a graph but no concrete data files?

Identify how far the given data could take you: if you were to carefully read the graphs in the paper, measuring out positions, and use whatever summary statistics thee authors provide, how close to a test statistic, a confidence interval, a p-value or some other desirable component of statistical analysis can you get?

Identify what the authors should have provided. Raw data? Processed data? Scripts for their own data cleaning process? Method details?

A

If the authors provide data

Backtrack any differences: if you and they find different p-values, explore whether you can find the source of the discrepancy. For example, did the authors use a normal distribution when they should have used a t-distribution?

If the authors do not provide data

Calculate as much as you can with what you have available. Make as many educated guesses as you can on the remaining data: what would the test statistic be if you have a reported estimator value, p-value and standard error?

Optional extras

Extra # 1; for A

Calculate all relevant values with your preferred technique: test statistics, confidence intervals, p-values, and compare with the techniques in the paper: do they agree in conclusion?

 Extra # 2 ; for A

List all statistical analysis methods you know that could have filled the need for each task the paper performed, and compare their prerequisites to what the data actually displays.

Combination for grade

The four highest of your results are averaged together to form your grade. This means the two optional tasks can compensate for missing parts of the four main tasks.

Paper selection

Your paper should be chosen and approved by September 14. You can pick a paper on a topic of your own interest, but you need to get professorial approval before the selection deadline.

You are not allowed to critique the same paper as a class-mate. All reports will go through Turnitin to check for plagiarism.

Some suggestions include: