Review

Status: Stage 7A contract, frozen; Stage 7B implemented; pull-based seats approved and implemented 2026-09-02. This page specifies exact semantics for the three review objects and the tfp.review/v0.1 language. Execution — seats and claims, holds and the node deadline, the optional TfP-hosted agent path, deterministic reduction, and the hand-off into a review stage of a VerificationRun — is live; see Runtime at the end. Where the 7A freeze described TfP assigning reviewers and replacing them from a budget, the seats model supersedes it; this page says so at each point. The participant-facing explanation is Verifying Results; the complete authoring guide is Authoring Review Protocols.

Canonical schemas: ReviewProtocol, ReviewProcess, ReviewAssignment. Canonical examples, validated in CI against the schema: the default, schemas/examples/review-protocol.campaign-reviewer.json (one accountable Campaign reviewer claims one seat, k_of_n 1-of-1, a 72-hour node deadline and a 120-minute hold, declared assistance optional — the protocol the live findings-extraction Problem uses); and schemas/examples/review-protocol.escalate.json, which covers the optional TfP-hosted agent panel with a human expert on disagreement.

Verification and Review are distinct

Verification asks: what must be true for TfP to make a particular claim about a Submission? A VerifierContract answers that with an ordered plan of stages.

Review asks: when judgment is required, who examines something, what do they see, what rubric do they apply, and how are their judgments combined? A ReviewProtocol answers that.

A review stage of a VerifierContract pins one ReviewProtocol. The protocol may see evidence from earlier Contract stages, but it does not replace the Contract, and a reducer never combines sandbox or external evidence directly: if judgment over that evidence is required, a review node receives it and emits a structured judgment first. Review is also used outside verification — for ProblemProposal and CampaignProposal adjudication, triage, ranking, and audit — through the same three objects.

Object roles

Protocol = the instructions. Process = one execution of them. Assignment = one claim of one seat on one review node (or, on a TfP-hosted agent node, one issuance TfP made), and what came back. Review work is not a Problem: an assignment is operational work generated because another object needs adjudication.

Reviewer configuration and rubrics are inline in the immutable protocol document; reducer results live in the process's node history; a reviewer's report is the assignment's response.

The tfp.review/v0.1 language

A protocol document is a bounded, statically validated, acyclic graph with exactly two node types.

review asks one or more reviewers to exercise judgment under an explicit rubric. It specifies reviewer kind and count, eligibility (human nodes) or exact allowed model configurations (TfP-hosted agent nodes), independence, what reviewers may see (visibility), what is hidden from them (blinding), conflict exclusions, Skills supplied as context, and the rubric. On a human node — the default — TfP opens seats that eligible Users claim; reviewers bring whatever agents and tools they like and remain accountable for the judgment.

Every completed assignment returns four things. The response is the domain-specific structured judgment, validated against the rubric's response_schema. The other three are universal and live on the assignment, not inside the response, so they read identically across every protocol: rationale_md (communicable justification), citations (structured evidence pointers), and confidence (an overall value in [0, 1]). The rubric declares whether each is required (rationale: required | optional, citations: {min_items}, confidence: required | optional). A human response that fails any of these is refused, with the problems named, and nothing is recorded; a TfP-hosted agent reply that fails them is recorded as an invalid assignment.

reduce combines structured outputs from earlier nodes with one deterministic operation from a closed catalogue. It never invokes a model. If intelligent synthesis is needed — three specialists' reports combined into one judgment — that is another review node, and the final reducer stays deterministic.

Dependencies are implied by reduce inputs, review visibility.prior_nodes, and when.node. The output field names the reduce node whose outcome is the protocol's result.

Conditional continuation exists in exactly one form: a node may declare when: {node, outcome}, naming a reduce node and one of the three closed outcomes. The node runs only if that reducer emitted that outcome; otherwise it is skipped. There is no expression language. Thresholds, k-of-n logic, confidence cut-offs, and disagreement detection are reducer configuration, and the reducer converts them into the closed outcome.

What v0 is not

Explicitly out of the language, so the protocol describes judgment and never arbitrary work:

Reducer catalogue (frozen for v0)

Three operations. Each produces a control outcome — accept, reject, or unresolved — and a typed intermediate value recorded in the process history.

k_of_n predicates are membership tests only — eq a scalar or in a set — and the accept and reject value sets must be disjoint. With membership predicates over one field, disjoint sets guarantee that no single response counts toward both sides, and accept.k + reject.k > count then guarantees the two outcomes cannot both hold. Numeric comparison is what threshold is for; allowing gte/lte in k_of_n would let gte 3 and lte 4 both match a response scoring 3 or 4, and the arithmetic guarantee would be false.

Why three. The catalogue was derived from the required examples rather than from what is easy to compute. k_of_n with separate accept and reject thresholds covers a single reviewer deciding (k: 1 each way), unanimity (k equal to the count each way, so any split is unresolved), majority, and "two of three agree, otherwise escalate" — so unanimous, any, all, and map are not separate operations. threshold is what scored rubrics need. coalesce is the only way to express "use the panel's decision, or the expert's if the panel did not resolve", which every escalation protocol needs. Weighted votes, tie-breaking by confidence, and set-level aggregation across candidates were considered and left out until an example requires them.

Static validation requires, for k_of_n, disjoint accept and reject value sets and accept.k + reject.k > count; for threshold, reject_lt ≤ accept_gte.

Static validation

A protocol must pass all of these before it can be pinned by a Contract stage or executed; a malformed protocol fails at authoring, before any reviewer capacity is spent:

1. Node ids are unique and match ^[a-z][a-z0-9_]{0,39}$.
2. Every inputs, prior_nodes, when.node, and output reference resolves to a node in the document.
3. The dependency graph is acyclic.
4. Node count ≤ 16; a reduce node's inputs ≤ 20; a review node's prior_nodes ≤ 20; reviewer count ≤ 25.
5. output names a reduce node, and every node is either an ancestor of output or conditional (via when) on one.
6. k_of_n and threshold take exactly one input, which is a review node; coalesce takes only reduce nodes.
7. A reducer's field exists in the required properties of its input node's rubric.response_schema, and its type is compatible: k_of_n requires a string, enum, integer, or boolean type and its predicate values must be members of that type (and of the enum, where one is declared); threshold requires number or integer.
8. k_of_n: the accept and reject value sets are disjoint (an eq value is a one-element set), and accept.k + reject.k > count of the input node's reviewers. threshold: reject_lt ≤ accept_gte.
9. when.node names a reduce node that is not a descendant of the conditional node, and when.outcome is one of the three closed outcomes.
10. prior_nodes names neither the node itself nor any of its descendants. Naming a node in prior_nodes creates a dependency on it, so the named node always completes first; acyclicity (rule 3) is what keeps this well-founded.
11. Every rubric.response_schema is a valid JSON Schema (draft 2020-12) describing an object, and declares none of rationale_md, citations, or confidence — those are universal assignment fields governed by the rubric's rationale, citations, and confidence requirements.
12. Human reviewer nodes carry eligibility and null allowed; agent nodes carry a non-empty allowed list and null eligibility. organization_members requires organization_id; named_users requires user_ids.
13. Every skills[].skill_version_id resolves to a non-yanked SkillVersion, and every allowed[] specification on a TfP-hosted agent node is one TfP's hosted path supports — a provider, model, and configuration it knows how to execute faithfully. This is a static property of the document. It does not depend on whether credentials are present at the moment of publication: a protocol is not invalid because an API key is temporarily absent, and "currently invokable" is a runtime question (see TfP-hosted agent reviewers).
14. policy is present, with finite assignment_timeout_hours and claim_hold_minutes, and the hold is shorter than the timeout. max_replacements_per_node is accepted and ignored, for compatibility with documents published before the pull-based change.
15. The document contains no properties outside the schema (additionalProperties: false throughout).
16. reviewers.assistance, when present, is on a human node, and each allowed[] entry is a well-formed provider/model pair (non-empty strings). Entries are arbitrary: TfP does not check that it recognises the provider or model, because it never invokes them — the list is matched against the reviewer's declaration at submission time.

The JSON Schema enforces shape, closed enums, bounds (rule 4), the human/agent field rules and the eligibility-rule field rules (rule 12), policy presence (rule 14), additionalProperties: false (rule 15), and the shape of assistance entries (rule 16). Everything else — node-id uniqueness (1), reference resolution (2), acyclicity (3), output reachability (5), input kinds (6), field existence and type compatibility (7), threshold arithmetic (8), when targets (9), prior_nodes (10), output-schema validity (11), Skill and hosted-spec resolution (13), the hold/timeout relation (14), and the human-node placement of assistance (16) — is a cross-field invariant enforced in code, with tests.

Content hash

ReviewProtocol.content_hash is sha256: over the canonical serialization of document: JSON with object keys sorted, no insignificant whitespace, UTF-8, no trailing newline. Every ReviewProcess records the hash and snapshots the document; every VerificationRun review stage records the hash. A protocol whose document changes is a new protocol.

Execution semantics (specified here, implemented at 7B)

Process lifecycle. queued → running → finished, forward-only. A process finishes with an outcome (the output reducer's result) or with inconclusive_reason and null outcome. It never remains open indefinitely.

Node execution. Nodes run when their dependencies have completed and their when condition, if any, is met. A node whose when is not met is skipped. A human review node opens reviewers.count seats and waits for claims; a TfP-hosted agent node issues its own assignments; reduce nodes execute as soon as their inputs complete.

Seats and claims. This is the model for human nodes; it replaced the push-assignment text of the 7A freeze and is implemented. TfP never picks or assigns a reviewer. When a review node becomes runnable it opens reviewers.count seats and fixes the node's deadline: node start plus policy.assignment_timeout_hours. Any User who is eligible under the node's eligibility rule and not excluded by its conflicts may discover the open seats (MCP pick_review, GET /api/review-seats, the /reviews page) and claim one (claim_review, POST /api/review-seats/claim, the web Claim button). The claim creates the ReviewAssignment in status accepted, with the reviewer's context frozen at that moment, and takes a hold of policy.claim_hold_minutes (default 120, minimum 15; must be shorter than assignment_timeout_hours), capped by the node deadline. The reviewer submits within the hold or it lapses: the assignment is expired, the seat reopens at once, and the same User may claim again if a seat is still open (nothing they wrote is kept). A declined claim reopens the seat immediately. One User holds at most one live claim per node, and cannot claim a node they have completed or declined, so independence is preserved by construction. Several eligible reviewers may race for the same seat; the first claim wins. Only completed assignments feed reducers. A node whose seats are not all completed by the deadline fails, and the process finishes inconclusive with reason reviewers_unavailable: nobody eligible claimed and completed the work in time. Every assignment, including expired and declined ones, stays in node_results[].assignment_ids. There is no replacement budget: max_replacements_per_node is legacy, accepted and ignored, and reopening a seat is not a replacement. The inconclusive reason assignment_timeouts_exhausted is likewise legacy, kept so old records still read; the current engine never emits it.

Refused, not invalid. A human response that fails the rubric — response_schema, a required rationale, citation shape or count, the confidence range, or a required assistance declaration — is refused with the problems named and nothing is recorded; the claim stays open for a resubmit within the hold. The invalid status exists only for TfP-hosted agent replies, where nobody is present to correct the format.

Declared assistance. Reviewers bring their own compute — a coding agent, another model, a tool, or none — on their own account, and the User remains the accountable reviewer. Agent usage is recorded on the assignment as assistance, which is provenance, not identity. Its shape is {provider, model, config?}: provider and model are required strings; config is an optional object (or null) that the reviewer may use to describe how the agent was run. The whole declaration is self-reported by the accountable reviewer; TfP records it as given and did not independently verify it. A human node may require the declaration and restrict it: reviewers.assistance = {required, allowed[]}. When required is true a response without a declaration is refused; when allowed is non-empty the declared provider/model must match one of its entries or the response is refused. Entries are arbitrary provider/model strings — TfP does not need to be able to invoke that provider, because it never does.

Assignment statuses on a human node are therefore accepted (claimed, hold running) → completed, or declined, or expired. pending and invalid occur only on TfP-hosted agent nodes.

Independence. On an independent: true node each reviewer is placed in its own independence_group, and no reviewer sees another's response before completing. Reviewers on later nodes see only what visibility.prior_nodes grants.

Blinding. visibility.hidden withholds, per node: submitter_identity (the Attempt owner's handle and profile), submitter_organization (the attributed Organization), prior_scores (the structured responses of nodes named in prior_nodes — their rationale and citations remain visible), and attempt_notes (the Attempt's Notes). Subject content — the Submission summary and its Artifacts — is never hidden.

Conflicts. conflicts applies to human reviewers and is optional: null on a human node means the defaults, exclude_submitter: true and exclude_submitter_organization: false; on an agent node it has no effect and is normally omitted. A conflict policy that empties the eligible pool — exclude_submitter_organization: true on a node whose eligibility is that same Organization's members — validates, and then nobody can claim, so every process finishes inconclusive with reviewers_unavailable at the node deadline. Static validation does not model pool size.

Human reviewers. The reviewer is the User who claimed the seat, and the record names that User as the accountable reviewer. TfP surfaces the required Skills and the materials, and records what context was supplied. A human who used an agent while judging records it as assistance — provenance, not identity. There is no human_with_agent kind. TfP does not gate on whether a human opened a Skill; the useful provenance is what context was supplied, and audit later measures whether the judgment was good.

TfP-hosted agent reviewers (optional, non-default). A node with reviewers.kind: agent asks TfP itself to obtain the judgments. TfP issues the assignments (round-robin over allowed[]), invokes the provider, spends and owns that execution, and records on the assignment the effective configuration that ran. A hosted reply that fails the rubric is recorded as invalid, and the node is issued another assignment, bounded by the node deadline. This is not the participant-provided-compute path, and no production protocol currently uses it; see Runtime for its exact behaviour.

Responses. Validated against the node's rubric.response_schema and the rubric's universal-field requirements before acceptance. Citations name what they cite by the same resource_uri TfP handed the reviewer in materials[] (an Artifact, a reference input, a Skill file), plus a locator and/or quote; artifact_id is derived for Artifacts. Rationale is the reviewer's communicable justification; TfP does not request, store, or expose hidden chain-of-thought.

Reducers. Executed by TfP, deterministically, over completed assignments only. The process records op, a hash of the config as executed, exactly which assignments or reducer outputs were read, and the output. Every final decision is reconstructible from the process record alone.

Integration with verification

A Contract stage of kind review carries review_protocol_id. When a VerificationRun reaches it, TfP creates a ReviewProcess with subject: {kind: submission, id} and invoked_by: {verification_run_id, stage_key}, and records mechanism.review on the stage result with the process id, protocol id, and protocol hash. The process outcome maps:

A review stage's reviewers may see earlier stage evidence when the node sets visibility.stage_evidence: true.

Purpose

purpose states what the review means and never selects a different engine:

A review stage of a Contract uses an acceptance protocol. Triage results are structured ReviewProcess outcomes, not Problem fields; they may inform derived Campaign-local prioritization later. v0 protocols review one subject; ranking across a candidate set is expressed as per-candidate scored review, with set-level aggregation deferred to the consuming feature rather than forced into the single-subject language.

Templates are sugar

Single reviewer, parallel k-of-n, panel with meta-review, sequential escalation, and triage-then-panel are authoring macros that expand to protocol documents in this language. They have no separate execution semantics, and a template expansion validates exactly like a hand-written protocol. Sampled audit is a use of the audit purpose over a sampled population, not a node type.

Explicitly not in Stage 7A

• No execution: scheduling, hosted-agent invocation, human notification, and reduction run at 7B.
• No reviewer reputation, matching, or workload balancing.
• No RBAC beyond the eligibility rules in the protocol.
• No expression language, and no reducers beyond the three.
• No set-level ranking aggregation.
• No mutable "current review state" on the reviewed subject.

Runtime (Stage 7B)

What the implementation fixed beyond 7A, all within the frozen language.

One validator

The static-validation rules live once, in packages/review-protocol (@tokensforprogress/review-protocol). The JSON-Schema engine underneath them is injected: Node callers use Ajv; the Worker, which cannot run Ajv, supplies its own subset engine (apps/worker/src/json-schema.ts). A rubric response_schema using a keyword that engine does not implement fails rule 11 at publication, so an authored constraint is never silently unenforced. Rule 13 resolves Skills against non-yanked SkillVersions and hosted allowed[] specs against what the hosted path supports; credentials are not consulted at publication.

Scheduling

A process advances in one idempotent step, invoked by whatever happened last (a claim, a response, a decline, the queue, the hourly sweep): lapsed holds and overdue assignments expire, nodes whose dependencies completed are executed in topological order, when conditions are evaluated against the named reducer's emitted outcome, human review nodes open their seats (and fail the process at their deadline), TfP-hosted agent nodes issue their assignments, reduce nodes run. A k_of_n or threshold node whose review input was skipped is itself skipped; coalesce treats a skipped input as unresolved. If the output node ends up skipped, the process outcome is unresolved. Human reviewer pools: any_participant is every User except the first-party account; campaign_reviewers is the Campaign's named reviewers (operators name reviewers per Campaign); organization_members is active memberships; named_users is the list. Conflicts apply as specified, with the documented defaults on a null block. Human seats are claimed, never issued (see Seats and claims); nothing in the scheduler chooses a person.

A human node whose seats are not all completed by its deadline fails the process with reviewers_unavailable; a hosted agent node fails with reviewers_unavailable when no configured provider supports any of its allowed[] specs. Assignments cannot be edited once terminal, processes cannot be edited once finished, and none of the three objects can be deleted (database triggers).

Materials: reviewers get references, and read with their own tools

The reviewer bundle (claim_review, get_review_assignment, /reviews/{ras}) carries materials[]: the Submission's Artifacts, the revision's bound reference inputs, and the SKILL.md of every required Skill. Each material is listed by reference — kind, key, filename, media_type, size_bytes, immutable content_hash, and resource_uri. The bundle is a list of things to read, not the things themselves: the reviewer is expected to fetch and read each material at its resource_uri with their own tools. Only a small textual file (16 KiB or less) is inlined as text; a larger one carries a 400-character preview with truncated: true and an omitted_reason saying to read the whole file at its resource_uri; anything unreadable (not textual, not valid UTF-8, withdrawn) carries an omitted_reason too, so the reviewer is never unsure what was and was not shown. A preview is not sufficient grounds for a judgment. Reference inputs are served to everyone at tfp://problem-revisions/{prv}/reference/{key} (the contents entry carries _meta.content_hash and _meta.size_bytes); Artifacts at tfp://artifacts/{id}; Skills at tfp://skill-versions/{skv}/SKILL.md. Citations use these URIs exactly as listed. TfP-hosted agent reviewers, which have no tools of their own, receive the bytes in their prompt instead.

One scheduler at a time

advanceProcess claims a 60-second lease on the process before it reads assignments, and its final write is a compare-and-set on the version it read; a concurrent caller that loses the claim returns the current record. Claims run under the same lease, so two claimants can never both take the last seat and a claim can never over-fill a node. A dead scheduler's lease expires and the sweep reclaims it.

TfP-hosted agent reviewers

An optional, advanced path, distinct from the default in which participants bring their own compute. On a reviewers.kind: agent node TfP itself issues each assignment (round-robin over allowed[], so a panel is heterogeneous when the author lists more than one), queues it (tfp-review; the sweep retries any assignment still pending), invokes the provider, and pays for and owns that execution. The provider receives the rubric, the response schema, the subject with the node's blinding applied, the materials with their bytes, the visible prior reviews, and the earlier stage evidence when granted, and must return one JSON object with response, rationale_md, citations, and confidence.

Exact configuration matters here, because TfP is the party running it. Providers separate supports(spec) — static: a provider, model and configuration TfP can execute faithfully; rule 13 at publication — from canInvoke(spec) — runtime: the credentials are present. The one provider today is the Anthropic Messages API, which needs an API key configured on the Worker; without it a protocol is still valid and publishable, and its hosted nodes execute once the key is set. The supported configuration keys are effort (low, medium, high, xhigh, max) and max_tokens (256–64000); any other key makes the spec unsupported rather than silently ignored. Provider defaults (such as max_tokens) are resolved before the spec is frozen on the assignment, so the recorded configuration equals what ran.

The reply goes through exactly the rubric check a human response does. A hosted reply that omits a required rationale or citation, or fails the schema, is recorded as invalid with the rejected response retained, and the node is issued another assignment, bounded by the node deadline. A provider failure (transport, refusal) leaves the assignment pending and the sweep retries it until the deadline. If no configured provider supports any allowed[] spec the node fails reviewers_unavailable.

No production protocol uses this path today. It exists for protocols whose authors want TfP to run a fixed panel; it is not how participants bring agents to review, which is the human-node path above.

Blinding and independence

The assignment's context freezes what the reviewer may see; the reviewer-facing view applies it at read time. On the public process record, a completed assignment's judgment (response, rationale, citations, confidence) is withheld until its node completes, so no reviewer on an independent node can read another's response through the public record either.

Inside a VerificationRun

Reaching a review stage creates the process with invoked_by and records mechanism.review on a stage result whose status is running; the run stays running, releases its executor lease, and is excluded from the run sweep. Seats open at that moment — never before the previous stage has finished. When the process finishes, the run resumes from that stage: accept → verified, reject → refuted, anything else → inconclusive. A stage whose pinned protocol is missing, yanked, or not an acceptance protocol for submissions finishes inconclusive.

Claiming over MCP

pick_review mirrors pick_problem: it lists open seats the connected User is eligible for — protocol title, node, the rubric's opening sentences and the fields the response must carry, how many seats are open, the hold length, the node deadline, the subject and Problem, and any declared-assistance requirement — filtered by campaign or problem and ordered by time remaining, then Problem priority. claim_review {review_process_id, node_id} takes a seat and returns the reviewer bundle with hold_until; list_my_review_assignments then shows the claim, get_review_assignment re-reads the bundle, and submit_review_response / decline_review_assignment settle it. The former accept_review_assignment tool no longer exists: a claim is the acceptance.

Read and write surfaces

• Web: /reviews (open seats the signed-in User may claim, and their own claims), /reviews/{ras} (the assignment: rubric, subject, materials, response form), and /review-processes/{rpr} (the public record).
• API: GET /api/review-protocols[/{id|slug}], GET /api/review-processes/{id}, GET /api/review-seats[?campaign=], POST /api/review-seats/claim, GET /api/me/review-assignments, GET /api/review-assignments/{id}, POST /api/review-assignments/{id}/decline|respond.
• MCP (schemas/mcp-tools.json 0.7.0): get_review_protocol, get_review_process, pick_review, list_my_review_assignments, get_review_assignment, and with the contribute scope claim_review, decline_review_assignment, submit_review_response; resource tfp://review-processes/{id}.
• Operator (admin secret): publish a protocol, yank one, name a campaign reviewer.

Not in 7B

Reviewer notification (email or otherwise), workload balancing, reputation, and any authoring surface for Campaign owners: protocols are published and campaign reviewers named by the operator endpoints until the Campaign-authoring stage.