TfP Docs

Mass Computational Reproduction

Imagine an organization such as I4R running a large computational-reproducibility program.

It partners with a scholarly-data provider such as OpenAlex to identify papers, publishes a standard for describing reproducibility work, and lets agents and human contributors turn papers into concrete reproduction tasks.

The Campaign does not need TfP to know what a “reproduction manifest” is.

The organization defines that standard itself.

The Campaign

Campaign:
Mass Computational Reproduction

The goal:

Continuously identify computational papers, describe exactly what would need to be reproduced, carry out the reproduction work, and maintain a structured corpus of what succeeded, failed, or could not be evaluated.

At a high level:

flowchart TD
    OA[Scholarly paper index] --> D[Standing DISCOVERY Problem]
    D --> PP[Candidate paper]
    PP --> SP[Specify reproduction of Paper X]
    SP --> M[Structured reproduction-plan Artifact]
    M --> V[Verification + review]
    V --> NP[Follow-on Problems]
    NP --> R[Reproduction Attempts]
    R --> RR[Verified reproduction results]
    RR --> A[Campaign-wide analysis]

1. The organization publishes its standard

The Campaign owner publishes a Skill such as:

@i4r/computational-reproduction

The package could contain:

SKILL.md
schemas/reproduction-plan.schema.json
schemas/reproduction-result.schema.json
examples/
paper-analysis scripts/
repository-inspection scripts/
preflight/

The Skill explains:

The JSON Schemas define the structured Artifacts the community wants.

They are domain standards, not TfP database entities.

2. A standing discovery Problem finds papers

The Campaign keeps a DISCOVERY Problem open:

Find newly indexed computational papers that should enter this reproduction program.

Its Skill might explain how to query OpenAlex or another scholarly index, apply the Campaign's inclusion criteria, deduplicate against existing work, and submit candidate papers.

Different Attempts might be run by:

A result might say:

{
  "paper_id": "...",
  "title": "...",
  "paper_url": "...",
  "code_repository": "...",
  "field": "machine learning",
  "computational": true,
  "why_candidate": "..."
}

A lightweight review or deterministic check can reject duplicates and obvious non-candidates.

Accepted candidates can lead to a Problem:

Specify the computational reproduction of Paper X.

The future ProblemProposal mechanism is a natural way to mediate this creation.

3. Specify the paper before spending large compute

The specification Problem asks a contributor to produce a structured reproduction plan.

The Submission might contain:

reproduction-plan.json
paper-notes.md
repository-inventory.json

The plan could identify:

paper metadata

current source repository
exact repository revision inspected

datasets and access requirements

software / environment assumptions

compute requirements

important computational claims

for each claim:
  paper location
  experiment required
  reported output
  proposed reproduction criterion
  likely cost
  dependencies

known ambiguities

suggested follow-on Problems

For example:

claims:
  - id: headline_accuracy
    paper_locator: Table 1, row Method X
    kind: scalar
    reported_value: 0.874
    suggested_check:
      type: tolerance
      absolute: 0.005

  - id: scaling_curve
    paper_locator: Figure 3
    kind: curve
    suggested_check:
      type: structured_curve_comparison

resources:
  accelerator: gpu
  estimated_accelerator_hours: 120

proposed_work:
  - make_original_repository_runnable
  - reexecute_headline_result
  - independently_reproduce_headline_result
  - reproduce_scaling_curve
  - run_seed_robustness

The exact schema belongs to the Campaign Skill.

4. Verify the reproduction plan

The plan should not automatically create expensive work merely because an agent produced JSON.

A VerifierContract can first run cheap structural validation:

sandbox
→ does reproduction-plan.json satisfy the I4R schema?
→ do claimed URLs / identifiers have valid shapes?
→ are required fields present?

Then a ReviewProtocol asks questions requiring judgment:

Did the plan identify the important computational claims?

Does the cited repository appear to be the correct codebase?

Are material figures, tables, or ablations missing?

Are the proposed criteria plausible measures of reproduction?

Are the resource estimates reasonable enough for routing?

Does the decomposition produce Problems that can actually be attempted independently?

The ReviewProtocol may begin with several agent reviewers and escalate disagreement to a domain expert.

The accepted reproduction plan is now a trusted Artifact.

5. The plan generates more work

The plan's proposed work can become ProblemProposals.

For one paper:

Paper X reproduction
│
├── Make the authors' repository reproducibly runnable
├── Map data-access requirements
├── Map compute requirements
├── Re-execute the headline result
├── Independently reproduce the headline result
├── Reproduce Figure 3
├── Reproduce the main ablation
├── Run seed robustness
└── Whole-paper synthesis

These are separate Problems because they have different:

A person should be able to take only:

Reproduce Figure 3.

They should not need to understand the Campaign's entire coordination system.

6. Making an unusable repository runnable

One common Problem is:

Turn the authors' published repository into a reproducibly executable package without materially changing the scientific method.

A solver may need to:

The Submission might contain:

reproduction_bundle.tar.gz
environment.lock
Dockerfile
patches/
experiment-manifest.yaml
REPRODUCTION.md

Verification can combine:

sandbox
→ package builds
→ smoke tests run
→ expected experiment entrypoints exist

review
→ patches do not materially change the scientific method
→ experiment mapping is faithful to the paper

Passing this Problem does not mean the paper reproduced.

It means:

The published computational procedure has been converted into a trustworthy executable representation suitable for reproduction work.

7. Reproduce individual claims

A claim Problem can bind:

paper
dataset
target_claims
original code, where appropriate

The solver submits:

code / bundle
environment
results.json
generated tables / figures
report

A sandbox may execute the reproduction directly when the compute fits.

For expensive training, the actual scientific computation may happen on compatible external compute, while verification evaluates the resulting evidence according to the Contract.

Review then checks methodological fidelity.

Different Problems should distinguish:

Problem Question
Re-execution Does the authors' code regenerate the result?
Independent reproduction Does a fresh implementation of the described method recover it?
Robustness Does the result survive reasonable perturbations?
Replication Does the scientific claim hold on new data?

Do not collapse all four into a single “reproduced” badge.

8. Whole-paper reproduction

A whole-paper Problem can exist in addition to per-claim Problems.

It asks:

Produce a complete, auditable reproduction package and account for the major computational claims in Paper X.

Its synthesis Attempt can build on the verified outputs of the individual claim Problems.

Conceptually:

flowchart TD
    A[Repository runnable] --> S[Whole-paper synthesis]
    B[Headline result] --> S
    C[Figure 3] --> S
    D[Ablation] --> S
    E[Robustness] --> S
    S --> W[Whole-paper reproduction report]

The result should not merely say yes or no.

A useful report might say:

Executable experiments:        24 / 26
Claims reproduced:             19
Material numerical deviations: 3
Claims not reproduced:          2
Could not evaluate:             2

Original repository patches:    4
Methodological fidelity review: accepted

Each row should link to underlying verification evidence.

9. Results improve over time

The reproduction specification itself can be improved.

Suppose another expert notices:

The original reproduction plan completely missed the experiment in Appendix C.

They can start another Attempt against the same standing specification Problem and submit a better plan.

No new Problem Revision is necessary unless the instructions or review standard changed.

Likewise, somebody may independently produce a better whole-paper reproduction.

same Problem Revision
    ↓
Submission A
Submission B
Submission C

The interface may show the best/latest trusted result while preserving every historical Submission.

Future result-lineage should let B say:

based on A
adds Appendix C experiment
corrects dataset version

10. Campaign-wide research

Once thousands of papers have structured, verified results, the Campaign itself becomes a metascience dataset.

The organization might ask:

What fraction of papers with public repositories are executable without modification?

What are the most common causes of reproduction failure?

Which fields have the largest gap between available code and runnable code?

How often do headline results reproduce while ablations fail?

What kinds of undocumented assumptions most often require expert reconstruction?

TfP should not answer these by reparsing millions of R2 blobs on each query.

Verified structured Artifacts can be asynchronously projected into analytical storage.

flowchart LR
    A[Verified structured Artifacts] --> P[Projection/index]
    P --> Q[Campaign queries]
    P --> E[Parquet / NDJSON exports]

Raw Artifacts remain the canonical evidence.

The analytical index is derived infrastructure.

11. What this Campaign requires from TfP

Existing primitives

Stage-8 candidate features

Infrastructure rather than new ontology

Explicitly not required

How such a Campaign is built, stage by stage, is worked through in Reproducing Computational Papers.