How to audit one published paper end to end and write the result as a structured, evidence-backed report.
SKILL.MD · v1
Paper audit
Audit the bound paper (paper) and, where present, its replication package or data (replication). Produce one JSON report that satisfies document_schema.
Passes, cheapest first
Triage. Read the paper. List every claim that carries weight: headline results, effect sizes, p-values, confidence intervals, derivations, tables and figures. Note anything that looks off on its face (impossible statistics, mismatched N, text that contradicts tables).
Rederive. For each analytic or numeric claim, rederive it from the stated inputs. For empirical claims with a replication package, rerun it. Record what you ran and what came out, exactly.
Statistical inference. For each test, check the test matches the design, the degrees of freedom and N are consistent, the reported statistic implies the reported p-value, and multiple-comparison handling is stated. Flag signs of selective reporting.
Citations, both directions. For the paper's key cited claims, check the cited source actually says what it is cited for. For the most-cited uses of this paper, check later papers cite it for what it shows.
Report. One finding per discrepancy, with severity, the exact location in the paper, what you expected, what you found, and the evidence (commands, outputs, page/table references). Say plainly what you could not check and why.
Rules
Be concrete. "Table 3, column 2: reported t=2.41 with N=48 implies p≈0.020, paper reports p<0.01" beats "statistics look wrong".
Separate error from fraud. Report what the evidence shows; do not infer intent.
Never fabricate a rerun. If you could not execute the replication package, say so and mark those claims not_checked.
Put working Notes on the Attempt as you go: approach, dead ends, what each pass found. The next solver reads them.