# Paper critique

Your grade in this course will be determined on a semester-long paper critique project. You need to pick a paper at the start of the semester, and during the semester use the material we cover to analyze and evaluate statistical choices made by the authors of that paper.

At the end of the semester, you will have produced an essay that will contain

- A description of the objective of the paper
- A comprehensive description of the statistical methods chosen by the authors
- An evaluation of whether the methods chosen were appropriate.

If you find that the authors made mistakes in choosing their analysis methods, you should suggest better choices. - A reproduction of the statistical analyses made by the authors: compute the estimators they computed, and compare with the values the authors reported.

If a reproduction is not possible, you should instead write a comprehensive description of what the authors should have provided to enable your reproduction of their analysis: is the problem missing data? missing descriptions of data cleaning, or post-processing, or of exact methods chosen?

You may optionally include, in order to raise your grade

- A reproduction of the statistical analyses using the methods you deem should have been chosen.
- Compare and contrast different methods that could have been applicable, and state their respective merits and drawbacks.

# Evaluation criteria

Each of the four paper tasks tasks is scored:

**Acceptable** is awarded for a completed report, with no more than 4 minor flaws. Any submission that misses the course deadline will be awarded this grade.

**Good** is awarded for a completed report without any flaws.

**Excellent** is awarded for a completed report without any flaws that also contains a suggestion and evaluation of alternative choices the paper authors could have made.

## Minor flaws

As minor flaws count:

- Significant language or readability issues, making it hard to understand the text
- Arithmetic errors in an appropriately chosen statistical method

Major flaws that award a failing grade to the paper and thereby the course include:

- Missing a deadline
- Not covering some of the four requested topics at all
- Readability issues to the point of our not being able to understand the text
- Evaluation without motivation for task 3.

# Rubric

### Task 1: Objective of the paper

#### Good

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Describes in your own words what the paper is trying to achieve.

#### Excellent

Describes in your own words what data the authors are analyzing, how it was acquired, and evaluate whether their experiment designs or sampling designs are appropriate for their task.

### Task 2: Statistical method description

#### Good

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Describe their entire data cleaning and data analysis process in as much detail as the paper admits.

#### Excellent

Qualified and motivated guesses for the steps the authors leave out of their descriptions. For example: a paper that only presents a ± error could have reached it with or without a particular number of standard deviations, or particular confidence level, with a normal or a *t*-distribution as basis. If the authors do not state enough details, fill in the blanks and give an argument why your suggestions are plausible.

### Task 3: Evaluation of whether methods were appropriate

#### Good

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Identify what aspects of the method choices depend on features in the data: is there a requirement that data be normally distributed? have sufficiently many observations?

Report the extent to which and the details of how the author has handled any method prerequisites.

#### Excellent

Qualified and motivated guesses for what the results of correctly testing for applicability of their chosen methods.

Qualified and motivated suggestions for better methods if any of their chosen methods are lacking in prerequisites.

### Task 4: Analysis reproduction

#### Good

**If the authors provide data**

Recompute any estimators and test statistics reported by the authors.

Recompute any p-values or confidence intervals reported by the authors.

Compare your results with the results reported by the authors. Comment on the comparison.

**If the authors do not provide data**

Identify what data was available, and what data was not available: did they give a graph but no concrete data files?

Identify how far the given data could take you: if you were to carefully read the graphs in the paper, measuring out positions, and use whatever summary statistics thee authors provide, how close to a test statistic, a confidence interval, a p-value or some other desirable component of statistical analysis can you get?

Identify what the authors should have provided. Raw data? Processed data? Scripts for their own data cleaning process? Method details?

#### Excellent

**If the authors provide data**

Backtrack any differences: if you and they find different p-values, explore whether you can find the source of the discrepancy. For example, did the authors use a normal distribution when they should have used a *t*-distribution?

**If the authors do not provide data**

Calculate as much as you can with what you have available. Make as many educated guesses as you can on the remaining data: what would the test statistic be if you have a reported estimator value, p-value and standard error?

## Optional extras

The extra tasks here could cover for a main task with errors, improving the overall grade.

#### Extra # 1; for full mark

Calculate all relevant values with your preferred technique: test statistics, confidence intervals, p-values, and compare with the techniques in the paper: do they agree in conclusion?

#### Extra # 2 ; for full mark

List all statistical analysis methods you know that could have filled the need for each task the paper performed, and compare their prerequisites to what the data actually displays.

# Paper selection

Your paper should be chosen and approved by September 14. You can pick a paper on a topic of your own interest, but you need to get professorial approval before the selection deadline.

You are not allowed to critique the same paper as a class-mate. All reports will go through Turnitin to check for plagiarism.

Some suggestions include:

- Cooper, Delmonico, Burg (2000) Cybersex users, abusers, and compulsives: New findings and implications
- Achab, Nicolier, Mauny, Monnin, Trojak, Vandel, Sechter, Gorwood, Haffen (2011) Massively multiplayer online role-playing games: comparing characteristics of addict vs non-addict online recruited gamers in a French adult population
- Beaulieu, Yatawara, Wang (2005) Who supports free trade in Latin america?
- Boyle, Desvousges, Johnson, Dunford, Hudson (1994) An investigation of part-whole biases in contingent-valuation studies
- Norman, Marlow, Messow, Shennan, Bennett, Thornton, Robson, McConnachie, Petrou, Sebire, Lavender, Whyte, Norrie (2016) Vaginal progesterone prophylaxis for preterm birth (the OPPTIMUM study: a multicentre, randomised, double-blind trial)
- Xu, Schwarz, Wyer (2015) Hunger promotes acquisition of nonfood objects
- Tol (2009) The economic effects of climate change
- Cuddy, Norton, Fiske (2005) This old stereotype: the pervasiveness and persistence of the elderly stereotype
- DeCelles, Norton (2016) Physical and situational inequality on airplanes predicts air rage
- Jung, Shavitt, Viswanathan, Hilbe (2014) Female hurricanes are deadlier than male hurricanes
- Staples, Redelmeier (2018) The April 20 Cannabis Celebration and Fatal Traffic Crashes in the United States
- Al-Mosaiwi, Johnstone (2018) In an Absolute State: Elevated Use of Absolutist Words Is a Marker Specific to Anxiety, Depression, and Suicidal Ideation