Method Paper semester project

Method paper

Your grade in this course will be determined on a paper and presentation describing an additional statistical topic. You need to clear your planned project with the professor early in the semester, and you are encouraged to seek help and show early drafts.

Some suggested topics to cover include:

  1. Bayesian Estimation (TS 7.1-3)
  2. Empirical Bayes (11.1-3)
  3. Bayesian Inference and MCMC (15.1-4, skim 15.5-6)
  4. Non-parametric regression (18.1-2 / 18.3-4 -- pick one, or two projects)

At the end of the semester, you will have produced an essay that will contain

  1. A description of the method
  2. Any theorems and proofs required to motivate or enable the method
  3. Conditions for use, applicability and rules of thumb
  4. An example of the method in use

Evaluation criteria

Each of the four paper tasks tasks is scored:

Acceptable is awarded for a completed report, with no more than 4 minor flaws. Any submission that misses the course deadline will be awarded this grade.

Good is awarded for a completed report without any flaws.

Excellent is awarded for a completed report without any flaws that also contains a description and evaluation of underlying philosophical differences to the topics covered.
[eg, if Bayesian, describe and evaluate difference to Frequentism; if non-parametric, describe and evaluate difference to parametric]

Minor flaws

As minor flaws count:

  • Significant language or readability issues, making it hard to understand the text
  • Arithmetic errors in an appropriately chosen statistical method

Major flaws that award a failing grade to the paper and thereby the course include:

  • Missing a deadline
  • Not covering some of the four requested topics at all
  • Readability issues to the point of our not being able to understand the text

Rubric

Task 1: Objective of the paper

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Describes in your own words what the paper is trying to achieve.

A

Describes in your own words what data the authors are analyzing, how it was acquired, and evaluate whether their experiment designs or sampling designs are appropriate for their task.

Task 2: Statistical method description

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Describe their entire data cleaning and data analysis process in as much detail as the paper admits.

A

Qualified and motivated guesses for the steps the authors leave out of their descriptions. For example: a paper that only presents a ± error could have reached it with or without a particular number of standard deviations, or particular confidence level, with a normal or a t-distribution as basis. If the authors do not state enough details, fill in the blanks and give an argument why your suggestions are plausible.

Task 3: Evaluation of whether methods were appropriate

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

Identify what aspects of the method choices depend on features in the data: is there a requirement that data be normally distributed? have sufficiently many observations?

Report the extent to which and the details of how the author has handled any method prerequisites.

A

Qualified and motivated guesses for what the results of correctly testing for applicability of their chosen methods.

Qualified and motivated suggestions for better methods if any of their chosen methods are lacking in prerequisites.

Task 4: Analysis reproduction

C

Written in comfortably readable English, with proper academic citation styles for any relevant references.

If the authors provide data

Recompute any estimators and test statistics reported by the authors.

Recompute any p-values or confidence intervals reported by the authors.

Compare your results with the results reported by the authors. Comment on the comparison.

If the authors do not provide data

Identify what data was available, and what data was not available: did they give a graph but no concrete data files?

Identify how far the given data could take you: if you were to carefully read the graphs in the paper, measuring out positions, and use whatever summary statistics thee authors provide, how close to a test statistic, a confidence interval, a p-value or some other desirable component of statistical analysis can you get?

Identify what the authors should have provided. Raw data? Processed data? Scripts for their own data cleaning process? Method details?

A

If the authors provide data

Backtrack any differences: if you and they find different p-values, explore whether you can find the source of the discrepancy. For example, did the authors use a normal distribution when they should have used a t-distribution?

If the authors do not provide data

Calculate as much as you can with what you have available. Make as many educated guesses as you can on the remaining data: what would the test statistic be if you have a reported estimator value, p-value and standard error?

Optional extras

Extra # 1; for A

Calculate all relevant values with your preferred technique: test statistics, confidence intervals, p-values, and compare with the techniques in the paper: do they agree in conclusion?

Extra # 2 ; for A

List all statistical analysis methods you know that could have filled the need for each task the paper performed, and compare their prerequisites to what the data actually displays.

Combination for grade

The four highest of your results are averaged together to form your grade. This means the two optional tasks can compensate for missing parts of the four main tasks.

Paper selection

Your paper should be chosen and approved by September 14. You can pick a paper on a topic of your own interest, but you need to get professorial approval before the selection deadline.

You are not allowed to critique the same paper as a class-mate. All reports will go through Turnitin to check for plagiarism.

Some suggestions include: