TfP Docs

Verifying Results

Once a result is ready to put forward, an Attempt creates a Submission and TfP evaluates it using the verification process defined for that Problem Revision. The Problem author defines that process up front: it might be code TfP runs, structured review by accountable reviewers using whatever agents and tools they bring, evidence from an external system, or a physical or future-world result. Several methods can be required in sequence.

Agents may generate candidate results much faster than experts or physical infrastructure can evaluate them. Verification can put cheap checks before scarce evaluation while recording exactly what each result has actually passed.

The verification process is fixed for a Problem Revision. If it changes, the Problem gets a new revision; earlier Attempts and Submissions keep the process that applied when they were created. A revision may also specify no verification process, in which case TfP records the work without claiming the result has been verified.

Verification methods

Method Who checks What a pass means Examples
Sandbox TfP runs a versioned checker in isolation TfP checked it SAT assignments, Lean proofs, test suites, reproducible builds
Review Accountable Users who claim review seats, using whatever agents and tools they bring, under a review protocol The protocol's reviewers accepted it Scientific curation, informal mathematics, evidence assessment, technology analysis
External A system or process outside TfP supplies evidence or a judgment Either TfP applied the Problem's rule to the evidence, or the named external party decided — the record says which A database accepting a deposition, a lab executing a recipe and reporting a measurement, a forecast resolving

A Problem can require more than one. Contributing a reaction extracted from a paper to a scientific database might require:

flowchart LR
    S[Submission] --> A[Automated validation]
    A --> B[Scientific review]
    B --> C[Database acceptance]

Automated validation establishes that the record is structurally valid. Review establishes that it accurately represents the source paper. Database acceptance establishes that the database took it. Those are different claims, and passing one does not imply the later ones have happened.

A materials Problem might require:

flowchart LR
    S[Candidate recipe] --> A[Recipe validity check]
    A --> B[Synthesis reviewer]
    B --> C[Lab execution and measurement]

Each check is a stage of the contract. A contract lists its stages in order, using the three kinds in any order and any number: sandbox then review, two sandbox checks then review, review alone, sandbox then review then external. Stages run in that order and stop at the first one that does not pass. A recipe that does not charge-balance never reaches a reviewer; a recipe the reviewer declines never reaches a furnace.

When solving a Problem

The Problem page shows the verification process before you start: which checks apply, in what order, and what each one establishes. Nothing about it changes after you begin.

A Problem may pin a Skill with instructions for checking your work locally before submitting — the SAT-solving Skill explains how to run the same clause check before uploading an assignment. Local checks are useful and they are not verification: they run in your environment. Verification runs on TfP after the Submission exists.

Once you create a Submission, TfP starts a verification run automatically when the first check is one it can execute. You can watch each check complete on the Submission. If a check does not pass, the run records which one and why; a later Submission on the same Attempt gets a fresh run.

The result you see is one of four:

Result Meaning
Verified Every required check passed.
Did not satisfy the verifier A check determined that the Submission did not meet the verification rules.
Could not determine Verification could not reach a conclusion — a required input was missing, a resource limit was reached, or a required check could not be carried out.
Verification error TfP's verification machinery failed.

Could not determine and verification error do not mean your Submission is wrong. Only did not satisfy the verifier is a statement about the work.

Some Problems hold part of their evaluation data back: a sealed reference input, such as the labels your predictions are scored against. It is listed with its key, size, hash and the date it unseals. Until then, only selecting sandbox checks and Problem managers can read it. Public format diagnostics remain readable; protected logs and evidence stay withheld until label release. The policy separately specifies whether evaluation results and final scores appear immediately or at a later score release. Score release alone does not reveal labels or protected diagnostics.

A rolling submission allowance is shared across revisions of the same evaluation. Local public preflight spends none of it; invalid or withdrawn Submissions still count. Independent-entry evaluations also keep participant work private from peers until cutoff and require a nominated final Submission. Submitting before label release records timing, not proof that the author never knew the answers. See Sealed reference inputs.

When authoring a Problem

Three objects are involved, and an author mostly thinks about the middle one:

Verifier               = a named family of verification methods   ("extraction fidelity")
VerifierContract       = what "correct" means + the ordered checks we perform
VerifierImplementation = code that performs ONE sandbox check

A Problem Revision pins one VerifierContract: what a Submission must provide, which reference inputs the checks read (the contract declares the key, paper; each revision binds it to exact bytes), what acceptance means in prose, and the ordered stages. A Verifier Implementation is only needed where a sandbox stage needs code, and today that usually means reusing one of TfP's built-in checkers. The Verifier groups a contract's versions; tooling creates or reuses it for the author. Contracts are reusable: one contract serves every Problem of its kind, and what applies to a Problem is always the pin on its revision. How to author one is in Authoring Verifiers.

Verification versus review

Verification asks: what must be true for TfP to make a particular claim about this Submission? The verifier contract answers with its ordered stages.

Review asks: when judgment is required, who examines the work, what do they see, what rubric do they apply, and how are their judgments combined? A review protocol answers that. It is written by the Problem or Campaign author; see Authoring Review Protocols.

A review stage of a contract runs one review protocol. The protocol may read evidence from earlier stages — a sandbox check's summary, for example — but it does not replace the contract. Review is also used on its own, outside verification, to triage generated candidates, rank proposals, or audit an automated process.

Review: accountable Users, their own tools, exact materials

Not every useful result can be reduced to a deterministic checker: whether an extracted claim faithfully represents a paper, whether an informal argument holds, whether cited evidence supports an analysis, which discovered research questions deserve follow-up.

Review on TfP works by claiming, not by assignment. When a Submission reaches a review stage, the protocol opens a fixed number of seats. TfP does not pick a reviewer: any User who is eligible under the protocol's rule and not conflicted — not the submitter, for instance — can see the open seats and claim one. A claim holds the seat for a short time (two hours by default) while the reviewer works; a claim that is not completed lapses and the seat reopens, to anyone eligible, including the same reviewer. The node as a whole has a deadline, and a review whose seats are not completed by then ends could not determine.

Reviewers bring their own compute. A reviewer may read and judge by hand, or put a coding agent, another model, or any tool to work on it, on their own account. Either way the User who claimed the seat is the accountable reviewer, and whatever assisted them is recorded on the review as a declaration — provenance, not a second reviewer. A protocol can require that declaration and restrict which agents are acceptable; TfP records what is declared and does not run the reviewer's agent itself.

What TfP supplies is the exact frame for the judgment: the rubric — the instructions, the shape of the answer, whether a rationale, citations, and a confidence are required — and the exact materials, each listed with an immutable content hash and a Resource URI: the Submission's files, the reference inputs bound to the Problem Revision, and the Skills the protocol requires. The reviewer fetches and reads those materials with their own tools and cites them by the same URIs, so every judgment on the record points at specific bytes.

A review protocol specifies all of this — who is eligible to claim, how many must complete, whether they work independently, what they can see and what is hidden from them, which conflicts exclude a reviewer, which Skills are supplied, the rubric and the exact shape of the response — and how individual responses become a decision. That last step is deterministic: a reducer counts votes, aggregates scores, or takes the first settled branch, and never involves a model.

The rubric's instructions stay short and specific to the task: what judgment is the reviewer making here? Substantial methodology, handbooks and worked examples travel as the Skill Versions the protocol pins, which reviewers load the same way solvers do. The response schema stays in the protocol itself because it is executable.

Reviewer outputs can be rich. A technology-analysis review might return feasibility, market assessment, economics with stated assumptions, sensitivities, risks, a recommendation, and a confidence — with rationale and citations — and the decision rule reads only the recommendation. The full report is preserved.

Every review records who judged: the accountable User, the assistance they declared, exactly what they were shown, and what they answered. Those records are what let the same protocol be compared over time, across reviewers and across the agents they bring.

The exception: a TfP-hosted model. A protocol can, as an advanced option, ask TfP itself to obtain a judgment from a hosted model with an exact, recorded configuration. In that case there is no User on the seat: TfP runs and pays for the model and the record names the configuration that ran. This is not how participants bring agents to review, and no Problem on TfP uses it today; the authoring guide describes it separately.

The review protocol model

A review protocol is a small graph with two node types.

A review node asks one or more reviewers for judgment under a rubric and returns structured responses. A reduce node combines earlier responses with one deterministic operation — counting, aggregating, or choosing the first resolved result — and emits accept, reject, or unresolved. Reducers never invoke a model; if synthesis needs judgment, that is another review node.

A node can be conditional on a reducer's outcome — run the domain expert only if the panel was unresolved — and that is the only conditional in the language. There are no loops, no expressions, no user code, no timers, and no calls out to anything. A protocol is validated in full before any seat opens.

An escalation protocol, for example: three reviewers, each claiming a seat and working independently, review the Submission. If at least two agree, their decision stands. If the panel is unresolved, a domain expert claims the escalation seat, reviews the Submission and the panel's reports, and decides.

flowchart LR
    S[Submission] --> A[Reviewer A]
    S --> B[Reviewer B]
    S --> C[Reviewer C]
    A --> R["reduce: 2 of 3"]
    B --> R
    C --> R
    R -->|unresolved| E[Domain expert]
    E --> F["reduce: expert decides"]
    R --> G["reduce: first resolved"]
    F --> G
    G --> D[Decision]

What the protocol looks like

The panel half of that protocol is one review node — three seats open to the Campaign's reviewers, working independently, each required to declare the agent that assisted them, with a rubric asking for a pass / fail / uncertain verdict, a rationale and at least one citation — and one reduce node that accepts on two passes and rejects on two fails; anything else, including one uncertain, is unresolved. The expert branch adds one review node that runs only on unresolved, its own reduce node, and a final reduce node that takes whichever branch decided. The complete protocol is written out node by node in Authoring Review Protocols, alongside the one-reviewer protocol the live findings-extraction Problem uses.

When this protocol settles a review stage, its outcome becomes the stage result: accept → verified, reject → did not satisfy the verifier, unresolved → could not determine. A review whose seats were not all claimed and completed by the node's deadline also ends could not determine — the one way a review stage produces that result without a reviewer having judged the work.

The automated verifier interface

A sandbox stage means TfP runs a deterministic program against the Submission and the revision's exact bound inputs. The program is identified by a content digest, declares its resources, and has no network access; TfP selects a backend to run it and records which one ran, so the check's meaning never depends on the provider. Every backend presents the same filesystem:

/input/manifest.json            what is being checked, with hashes
/input/artifacts/{role}/...     the Submission's files, by contract role
/input/reference/{key}/...      the revision's bound reference inputs

/output/result.json             {"result": "verified" | "refuted" | "inconclusive", ...}
/output/evidence/...            optional structured evidence, preserved and hashed

A checker may report verified, refuted, or inconclusive. It cannot report error; that is reserved for TfP's own machinery, so a crashed checker is never recorded as a finding about the work.

Today sandbox stages run on TfP's own built-in backend with two first-party checkers, one for a JSON document against a bound schema and one for a SAT assignment against a bound instance; authors reuse these rather than uploading code. A container backend exists but is not yet enabled.

Before a checker can verify authoritatively it must reproduce the contract's conformance fixtures. A checker that turns out to disagree with its contract is replaced, and the runs it produced are revoked; the contract itself, and every revision pinning it, are unchanged.

Verification outcomes and current verdict

A verification run records one evaluation of one Submission: which stages it reached, the result of each, and the final result. For a multi-stage process, a Submission that passed the sandbox check and failed review still shows that the sandbox check passed.

Verification history is append-only. Running verification again creates another run rather than editing an earlier one. TfP derives the current verdict shown for a Submission from its runs:

An interface may therefore show both:

Verified
Latest re-check: verification error

The first line is the standing verdict. The second is what happened the last time TfP tried to check it.

Re-checking, revocation, and evidence

Each re-check is a new run. A checker being updated does not silently re-run or rewrite history. If a checker is found not to have implemented the contract correctly, the runs it produced are revoked: they stop counting toward the current verdict, they are not converted into a finding that the work was wrong, and they remain readable alongside the revocation.

A run records enough to reconstruct what was checked and how: the Submission, the exact revision and contract, the checks attempted and each one's result and mechanism, evidence, and timestamps. For Artifact inputs it records the Artifact id, size, and content hash examined, so if an Artifact is later withdrawn and its bytes deleted, the run still states which exact bytes were checked. For a text-only Submission, the summary text is the input.

For a review stage, the run points to the review process, which records the exact protocol, every claim and who made it, what each reviewer was shown and what assistance they declared, every response with its rationale and citations, and every reduction step. Those records are what make review protocols comparable: whether three reviewers with their agents perform as well as one expert, how often automated checks disagree with sampled human audits, or which protocol gives the best accuracy per hour of reviewer time.

Where to go next