TfP Docs

Authoring Verifiers

A verifier defines what Tokens for Progress is willing to say about a submitted result. Three objects are involved; most of the time you only think about the middle one.

Verifier               = a named family of verification methods   ("extraction fidelity")
VerifierContract       = what "correct" means + the ordered checks we perform
VerifierImplementation = code that performs ONE sandbox check

For example, a verifier for extracting structured findings from a paper (the live Problem uses the same shape with one sandbox stage and one review stage):

Verifier "Structured findings extraction"
  Contract v1
    1. sandbox: JSON has required fields
    2. sandbox: citations point to real locations
    3. review:  reviewer checks fidelity to source
  Implementation A — code for sandbox stage 1
  Implementation B — code for sandbox stage 2

An author's questions are, in order:

  1. What needs to be checked, and in what order? That is the VerifierContract: one immutable document holding the acceptance statement, the ordered stages, what the solver must submit, and which reference inputs the checks need.
  2. Does a deterministic check need code? Each sandbox stage is performed by a VerifierImplementation. Today you normally reuse one of TfP's built-in checkers rather than writing one (see Sandbox stages).

The Verifier itself is grouping, identity and lineage: it is what "Contract v1, v2, v3" are versions of. Authoring tooling creates one or reuses one for you; you rarely name it.

A Problem Revision pins one Contract (or none), and binds the reference inputs the Contract declares. Participant-facing explanation: Verifying Results. Exact semantics: the Verification reference.

Stages

A Contract's stages is an ordered list. Each stage has one of three kinds, and a Contract may use any kinds in any order and any number of times: sandbox → sandbox → review, sandbox → review → external, review only, and so on.

Kind Who settles it A pass means
sandbox TfP runs a deterministic program TfP checked it
review Users who claim seats under a ReviewProtocol, bringing their own agents The protocol's reviewers accepted it
external A system or process outside TfP See rule_owner below

Stages run in order and stop at the first one that does not pass; a run is verified only when every stage passed. Put cheap checks before scarce human or physical evaluation. There is no branching or conditional logic in a Contract; anything conditional (escalate on disagreement, expert only if the panel is unresolved) lives inside a ReviewProtocol. Each stage's description_md says what that stage alone establishes.

Sandbox stages

A sandbox stage means: TfP runs a deterministic program against the Submission and the Problem's exact bound inputs. The Contract says "run a sandbox check here"; the Implementation says "this is the program that performs it".

The program sees the same filesystem on every backend:

/input/manifest.json            what is being checked, with hashes
/input/artifacts/{role}/...     the Submission's files, by Contract role
/input/reference/{key}/...      the revision's bound reference inputs, by key

/output/result.json             {"result": "verified" | "refuted" | "inconclusive", ...}
/output/evidence/...            optional structured evidence, preserved

The program may answer verified, refuted or inconclusive. It may not answer error: that word is reserved for TfP's own machinery (the sandbox failed to start, a limit was exhausted), so an infrastructure failure is never recorded as a finding about the work.

What runs today. Sandbox stages run on TfP's in-Worker tfp-builtin backend, which executes first-party checkers written and reviewed by TfP. Two exist: json-schema-check (a submitted document against a bound JSON Schema) and sat-assignment-check (an assignment against a bound CNF instance). Each is identified by a content digest derived from its image reference (for example tfp-builtin/json-schema-check:1), and every run records which backend and digest ran. When no builtin fits, an author writes one Python file beside the Contract and names it on the stage ("implementation": "./checker.py"): TfP stores the exact bytes and runs them on its own Python runtime, tfp-runtime/python:1, with nothing to build (Script checkers). Anything a single standard-library file cannot do is a digest-pinned OCI image of the author's (Custom checkers); both run on TfP's runner data plane (tfp-runner). A Contract whose sandbox stage has no implementation can still be published; a run reaching that stage finishes could not determine with the reason recorded.

Conformance. A Contract carries fixtures: known inputs with known expected outcomes (a valid assignment → verified; an unsatisfied clause → refuted; unparseable input → inconclusive). An Implementation may verify real Submissions only after reproducing every fixture. Conformance checks the code against the Contract; it never makes the code the source of truth.

Contract or Implementation: which one changes?

Did the meaning of "verified" change?

NO   → same Contract, new Implementation
       (a checker bug fixed; affected runs revoked if needed)
YES  → new Contract, and a Problem needs a new Revision to adopt it
       (partial assignments were allowed and now are not)

The Contract's statement_md is the normative meaning. If an Implementation disagrees with it, the Implementation is wrong.

Reference inputs

Many checks depend on material that differs from Problem to Problem. The Contract declares the name and expected shape of that input; the Problem Revision supplies the exact bytes.

Problem: analyze this research paper. The Contract declares a reference input paper. The Problem Revision binds paper → the exact uploaded bytes.

Problem: improve a model against this dataset. The Contract declares training_data and evaluation_set; each revision binds them.

A declaration gives the key, whether it is required, the accepted media types, a size limit, and a one-line description:

reference_inputs:
  - key: paper
    required: true
    media_types: [text/html, text/plain]
    max_bytes: 8388608
    description: The exact source document the extraction is judged against.

For public inputs, solvers, reviewers and sandbox checks read the same bytes: sandbox programs at /input/reference/{key}/, people and agents at tfp://problem-revisions/{prv}/reference/{key}. Statements, Skills and rubrics refer to an input by its key only ("extract from the paper bound as paper"); the binding holds the hash and size, and prose never copies them. The live findings-extraction Contract declares three keys this way: document_schema, paper_html and paper_text.

One Contract therefore serves many Problems. The canonical technical example is SAT: one Contract declares instance, one sandbox stage checks an assignment against it, and each SAT Problem binds instance to a different DIMACS file. Do not create a new Contract because the instance changed. The exact publication checks (every required key bound once, types and sizes within limits, server-computed hash) are in the Verification reference.

Sealed slots

When a check scores Submissions against data solvers must not see (the labels of a test set, a hidden answer), declare that slot sealed: true. Each revision that binds it fixes a date until which only sandbox checks and the Problem's managers can read the bytes; everyone else sees the key, the hash and the size. Logs and evidence from stages selecting sealed inputs remain withheld until label release. Public validation diagnostics remain readable. Submissions consume a per-User rolling allowance shared across revisions of the evaluation.

Every sandbox stage explicitly declares reference_input_slots; each sealed slot must be selected by a sandbox stage. Put a public pass_fail format check before the protected score stage so participants can fix formatting locally without spending a Submission. Write conformance fixtures against public stand-in data, never the sealed file.

The revision separately chooses immediate or dated final-only score feedback, and collaborative or independent entries. Independent entries hide participant work from peers until a mandatory cutoff and require an explicit final nomination. These controls do not change managers' access to labels. Plan the combination using Designing evaluations with hidden inputs; the exact semantics are in Sealed reference inputs.

What the solver must submit

The only question is: what does the solver need to hand in for this mechanism to work? Usually one named artifact role with its allowed media types:

submission_requirements:
  required_artifacts:
    - role: document
      description: The extraction, one JSON document satisfying the bound document_schema.
      media_types: [application/json, text/plain]
      max_bytes: 1048576

Roles describe meaning, not filenames; a Submission's files map to roles in the order they are referenced. A Submission that does not meet the requirements is not wrong: the run ends could not determine with the unmet requirement recorded.

A Contract whose answer is a short value can accept it in the Submission's text instead of a file (accepts_text_only: true, with answer_format_md saying how the answer must appear). Tooling defaults accepts_text_only to false and answer_format_md to null, which is right for almost every Contract.

Review stages

A review stage pins an immutable ReviewProtocol by id:

- key: fidelity_review
  kind: review
  title: Fidelity review
  description_md: A reviewer reads the exact bound source and judges whether the document is a faithful extraction.
  review_protocol_id: rpt_...

When a run reaches the stage, the protocol's seats open. Any eligible, non-conflicted User claims one, reads the exact materials through their Resource URIs, uses whatever agent and tools they bring (recorded as declared assistance), and submits a cited structured judgment. Hosted agents TfP runs itself are an optional protocol feature no live Problem uses. The outcome maps accept → verified, reject → refuted, unresolved → could not determine.

Keep the rubric's instructions_md short and task-specific: it answers "what judgment are you making here?". Substantial methodology, handbooks and worked examples belong in the Skill Versions the protocol pins, which reviewers load the same way solvers do. response_schema stays inline in the protocol because it is executable. See Authoring Review Protocols.

External stages

An external stage records evidence or judgment produced outside TfP: a database accepting a deposition, a laboratory reporting a measurement, a forecast resolving. external_system names the party, and rule_owner says whose rule decides:

External-stage adapters are not live yet; a run reaching one finishes could not determine with the reason recorded.

Authoring locally

The intended way to author a verifier is next to its Problem, as files in a directory, with the TfP Authoring Skill guiding your agent and a local linter (pnpm tfp-lint <dir>) that runs the same validators TfP applies at publication. In a Problem directory the verifier is verifier-contract.json, plus review-protocol.json for review stages, checker.py (or another .py the stage names) for a script checker, verifier-implementation.json only for an older package that does not name its checker on the stage, and fixtures/ for conformance inputs; every file you do not need is optional. Durable private materialization over MCP follows as a separate step; see Authoring with Your Agent and the authoring reference for its current status. The end-to-end loop (lint, test, upload, durable private draft, explicit sharing) is in Authoring with Your Agent; a Python file of your own, named on a sandbox stage, is in Script checkers; a checker image of your own, named by digest, is in Custom sandbox checkers.

Where to go next