TfP Docs

Reproducing Computational Papers

A worked example of one Campaign's design, kept for its detail. It predates the local-file authoring convention; the general way to author a Problem today is Authoring with Your Agent, and the story this page instantiates is Mass Computational Reproduction.

This page is a worked example for Problem authors. It shows how TfP's verification model — a VerifierContract with reference input slots, a Problem Revision that binds those slots to exact bytes, and an ordered list of stages — expresses the reproduction of a computational paper. Nothing here introduces a new object. A reproduction is a Problem like any other; what is specific is how the paper, its data, and the claims being tested are attached, what a solver submits, and what each stage establishes.

The overall path: an author reads a paper and extracts its computational claims; each claim, or a tightly coupled set of claims, becomes one Problem; a Problem Revision binds the paper, the data, and the target claims as reference inputs; a solver submits a reproduction package; a sandbox stage re-executes or checks the package against the target claims; a review stage judges whether the number was recovered the way the paper says it was.

flowchart LR
    P[Paper] --> E[Extract computational claims]
    E --> Q["One Problem per claim<br/>(or coupled claim set)"]
    Q --> R["Revision binds<br/>paper / data / target claims"]
    R --> S[Solver submits a<br/>reproduction package]
    S --> X[Sandbox re-executes<br/>and compares]
    X --> V[Review checks<br/>methodological fidelity]

Concepts: Verifying Results. Exact semantics: Verification reference.

Four questions, four Problem kinds

"Reproduced" is not one thing. A paper's result can survive one of the questions below and fail another, and each answer is useful on its own. Do not fold them into a single badge; give each its own Problem, with its own Contract and its own target claims where those differ.

Question Problem kind What a pass establishes
Can the authors' original code, on the stated data, regenerate the reported result? Re-execution The artifact the authors published does what the paper says it does
Can a fresh implementation of the method as described regenerate the result? Independent reproduction The paper's description of the method is sufficient and the result is not an accident of one codebase
Does the result survive reasonable perturbations — seeds, splits, minor hyperparameter changes, alternative preprocessing? Robustness The result is not fragile to choices the paper does not defend
Does the claim hold on new data drawn from the same population or setting? Replication The finding generalizes beyond the specific dataset used

The four differ in what the solver is allowed to reuse (the authors' code or not), what counts as the same result (a point value within tolerance, or a direction and magnitude), and what the reviewers are asked to check. A Problem statement should say which question it is asking. A Campaign can ask all four about the same paper.

A Campaign layout

A paper rarely makes one claim. A Campaign gives the per-claim Problems a shared home — the paper, the dataset, and any shared Skills for running the authors' environment — while each Problem stays independently attemptable.

Campaign: Reproduce Paper X
├── Problem: Main numerical result (Table 1, row "Method X")
├── Problem: Figure 3 — scaling curve
├── Problem: Key ablation (Table 4, without component C)
└── Problem: Robustness analysis — seeds and splits

Each child Problem has its own revision, its own target claims, and its own verification runs. The Campaign view then shows which claims reproduced, which were refuted, and which could not be determined, rather than a single yes or no for the paper. This is more useful to readers of the paper: the headline number may reproduce while the ablation that motivates the design does not, and that is exactly the information a per-claim layout preserves.

Some claims are tightly coupled — a figure and the table it is drawn from, or two numbers produced by the same run. Those can share a Problem, with several entries in one target claims file. The test is whether a solver could sensibly attempt one without the other.

A reusable Contract

The Contract does not contain the paper. It declares slots, and each Problem Revision binds them. One reproduction Contract therefore serves many papers, in the same way one SAT Contract serves every CNF instance.

Three slots cover the common case.

reference_inputs:
  - key: paper
    required: true
    media_types: ["application/pdf"]
    max_bytes: 52428800
    description: >-
      The paper whose claims are being reproduced, as published or as
      the accepted preprint. The sandbox does not read it; reviewers do.
  - key: dataset
    required: true
    media_types: []
    max_bytes: 10737418240
    description: >-
      The data the paper's result is computed on, packaged as the
      revision specifies (an archive, or a manifest of pinned sources).
  - key: target_claims
    required: true
    media_types: ["application/json"]
    max_bytes: 1048576
    description: >-
      The claims under test: metric, reported value, tolerance, and
      evaluation split for each. The sandbox compares results against it.

A Problem Revision binds each slot to exact content-addressed bytes in verifier_input_bindings. At publication TfP checks that every required slot is bound once, that the media type and size satisfy the slot's constraints, and that the hash matches. Bindings are immutable with the revision. If the author later corrects a tolerance or swaps in a cleaned dataset, that is a new revision; earlier Attempts keep the bindings they were made against.

Slots are optional where a Problem kind does not need them. A replication Problem might add a replication_dataset slot and leave dataset as the original for reference; a re-execution Problem might add original_code, so that the authors' code is a pinned input rather than something the solver locates.

The target claims file

target_claims is the file the sandbox compares against. It is Contract material: the Contract's description for the slot and its statement_md say what the file must contain, and the VerifierImplementation parses it. There is no TfP schema for it, and the shape below is conceptual — a Contract author is free to define a different one.

{
  "claims": [
    {
      "id": "table1_method_x",
      "metric": "accuracy",
      "reported_value": 0.874,
      "tolerance": 0.005,
      "evaluation_split": "test"
    },
    {
      "id": "table1_baseline_z",
      "metric": "accuracy",
      "reported_value": 0.849,
      "tolerance": 0.005,
      "evaluation_split": "test"
    }
  ]
}

Tolerances are the author's judgment, recorded up front. They should reflect the paper's own reporting (a standard deviation across seeds, if given), the precision of the reported number, and the Problem kind: a re-execution can be tight, an independent reproduction should allow for implementation variance, and a robustness Problem may express the claim as a range or a direction rather than a point.

What the solver submits

submission_requirements names the roles a reproduction package must fill. Roles map to the Submission's Artifacts by order, so the Contract never dictates filenames.

submission_requirements:
  accepts_text_only: false
  answer_format_md: null
  required_artifacts:
    - role: reproduction_bundle
      description: >-
        The code that produces the results, as an archive. For a
        re-execution Problem this is the authors' code plus any patches;
        for an independent reproduction it must not include it.
      media_types: ["application/zip", "application/gzip"]
      max_bytes: 524288000
    - role: environment
      description: >-
        A pinned environment specification the bundle runs under
        (a lockfile, a container definition, or both).
      media_types: []
      max_bytes: 1048576
    - role: results
      description: >-
        results.json: one entry per target claim id with the value the
        solver obtained, and the seeds and configuration used.
      media_types: ["application/json"]
      max_bytes: 1048576
    - role: figures_tables
      description: Generated figures and tables corresponding to the claims.
      media_types: []
      max_bytes: 104857600
    - role: report
      description: What was run, what deviated from the paper and why, and what was observed.
      media_types: ["text/markdown", "application/pdf"]
      max_bytes: 10485760

A Submission that omits a role or exceeds a limit is not refuted; its run finishes inconclusive with the unmet requirement recorded. A Skill pinned on the revision can tell the solver how to run the same comparison locally before submitting; that preflight is not authoritative.

Stages

Two stages: a sandbox stage that establishes the numbers, then a review stage that establishes the method. Each stage claims only what it checks, and a run that fails the sandbox never consumes reviewer time.

Sandbox: execute or compare

The sandbox stage sees the Submission's Artifacts at /input/artifacts/{role}/ and the revision's bound inputs at /input/reference/{key}/, and writes /output/result.json with verified, refuted, or inconclusive.

There are two designs, and the Contract must say which it uses:

Either way, the comparison is per claim: every claim within tolerance is verified; any claim outside tolerance is refuted; a results file that cannot be parsed, or a bundle that fails to run, is inconclusive. Per-claim findings go in the result's findings and in /output/evidence/, so a reader can see which claim failed.

Review: methodological fidelity

Numbers within tolerance do not establish that the paper's method was followed. The review stage pins a ReviewProtocol whose reviewers read the paper, the bundle, and the report, and answer questions such as:

The last question is the one a sandbox cannot answer and the reason the stage exists. How to write the protocol — reviewer kinds, rubric, reducers — is covered in Authoring Review Protocols.

The stages block

stages:
  - key: claim_comparison
    kind: sandbox
    title: Target claim comparison
    description_md: >-
      Reads the solver's results against the bound target_claims and
      reports each claim as within or outside tolerance. Establishes
      numerical agreement only; says nothing about how the numbers were
      produced.
  - key: fidelity_review
    kind: review
    title: Methodological fidelity review
    description_md: >-
      Reviewers judge whether the reproduction follows the method the
      paper describes: preprocessing, exclusions, hyperparameters,
      evaluation split, and whether the result was obtained by a
      materially different method. Establishes reviewer judgment only.
    review_protocol_id: rpt_0000000000

A sandbox stage may add an evidence_schema for /output/evidence/summary.json — for example, the number of claims checked and the number within tolerance — so evidence is comparable across implementations of the Contract. The protocol id above is a placeholder for the immutable ReviewProtocol the author publishes.

Example: a machine-learning result

The paper reports: "Method X gets 87.4% accuracy on Dataset Y, against 84.9% for baseline Z." The author creates an independent-reproduction Problem for the main result and binds:

The solver implements Method X and baseline Z from the paper's description, trains both, and submits the bundle, an environment lockfile, results.json with the two obtained accuracies and seeds, the generated table, and a report noting any places where the paper was ambiguous and what they chose.

The sandbox stage reads results.json and compares each entry with the target claims. Suppose the solver obtained 87.1% and 85.2%: both within tolerance, so verified, with per-claim findings recorded. Had Method X come in at 85.0%, the run would be refuted at this stage and review would not run.

The review stage then asks whether the reproduction is faithful: the same augmentation pipeline, the same optimizer and schedule, no test set leakage into model selection, and — the question that matters most — whether Method X as implemented is the method the paper describes or a stronger variant that happens to reach the number. A verified run means both stages passed; the stage results say which questions each one answered.

Example: an economic simulation

The paper presents a calibrated general-equilibrium model and reports a baseline equilibrium and a policy counterfactual. The author creates a re-execution Problem and binds:

Curve claims need a representation the sandbox can compare — a list of (x, value, tolerance) points rather than an image. The conceptual claim shape extends naturally: the reported_value becomes an array and the implementation compares point-wise. The Contract's slot description says so.

The solver submits the model code, an environment specification, results.json with the four obtained values (the curve as points), the regenerated figure, and a report. Because a solve of this kind usually fits in a sandbox run, the Contract uses the execute design: the implementation runs the model under the pinned parameters and calibration data and compares its own outputs with the target claims, rather than trusting the solver's results file.

The review stage checks that the model is the one in the paper: the same market structure and timing, the same solution method or one the paper permits, and the same calibration targets. It also records undocumented assumptions the solver had to make — an initial condition, a grid resolution, a convergence criterion — since those are the usual reason a re-execution disagrees with a published table and are worth surfacing whether or not the numbers matched.

Reducing authoring effort

Most of the work in setting up a reproduction Problem is reading the paper carefully: listing its claims, finding the reported numbers and their precision, identifying the figures and tables, noting the dependencies and whether the code and data are available, and choosing tolerances. A future authoring agent can do a first pass of that extraction and present candidate claims, numbers, figures, dependencies, availability notes, and suggested tolerances for the author to review and correct. The author remains responsible for what is published: which claims become Problems, which Problem kind each is, and what the tolerances are.

There is no paper-specific schema on TfP and none is planned in this pass. Everything the reproduction needs is carried by the Contract's reference input slots, the bytes a Problem Revision binds to them, the roles the Contract requires of a Submission, and the ReviewProtocol the review stage pins. An authoring tool produces those; it does not add objects.

Where to go next