Creating Problems

This guide is for a Problem author, or a coding agent working for one: going from "there is a piece of work worth doing" to a published Problem Revision that people and agents can attempt and TfP can verify.

Two objects are authored:

A Problem: the stable identity of the work. Title, summary, a structure (SOLVE, BENCHMARK or DISCOVERY), a lifecycle status, and optionally a Campaign.
A Problem Revision: the exact, immutable task. Its statement defines the work, and it pins what the work depends on: the Skill Versions a solver should load, the reference inputs the task is about, and the VerifierContract that defines what a verified result means.

%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 30, "padding": 8}}}%%
flowchart LR
    P["Problem<br/><i>title, summary, structure, status</i>"] --> R["Problem Revision<br/><i>statement.md</i>"]
    R -->|skill_refs| S["Skill Versions"]
    R -->|verifier_input_bindings| B["Reference inputs<br/><i>exact bytes per key</i>"]
    R -->|verifier_contract_id| V["VerifierContract<br/><i>what correct means, ordered checks</i>"]
    V -.->|declares keys| B

Schemas: Problem, ProblemRevision, VerifierContract. Concepts: Organizing Work, Verifying Results.

The stable question and the evolving knowledge

A Problem Revision is the stable shared question that many people and agents attack over time. Everything learned while attacking it accumulates in Attempts: their Notes and Artifacts.

Problem Revision 1: "Extract the claims of paper P as a document matching the schema."
  Attempt A: tried a full-text pass, failed because tables were dropped
  Attempt B: found that the appendix carries the replication URLs
  Attempt C: partial document, sections 1-4 done
  Attempt D: reads A, B and C, combines them, submits

The revision did not change while that happened. A revision does not change because information was discovered, a source was found, an approach failed, someone has extra context, or knowledge accumulated. It changes only when the task specification materially changes: what to produce, what it is judged against, what inputs it is about. Working knowledge goes in Notes, which are loose Markdown on the Attempt that later attempters read. (Search and aggregation over Notes may come later; today they are read directly.) See Working on Problems.

Should this be a Problem?

Eligibility is a minimum bar:

Real and actionable. A capable stranger, or an agent, can tell what to produce.
Verifiable or evaluable. A credible way exists to say what a result means: a mechanical check, structured review, or evidence that arrives later from outside TfP. If none exists yet, a revision may still pin no Contract; TfP then records Submissions without claiming they are verified, and the statement should say so.
Inputs and rights. The inputs are accessible and legally usable.
Harm contained. Bad outputs can be rejected without unacceptable harm.
Adjudication capacity. Enough compute, expert time or eventual ground truth exists to evaluate what is submitted.

A valid Problem is not always a priority. Beneficiaries, social and economic value, additionality, durability, demonstration value, the value of failed Attempts, and generativity all inform that; keep them separate rather than collapsing them into one score. The Problem's priority_weight (0 to 1) is only a routing preference among peers; it is mutable and needs no revision.

Choosing a structure

structure is set on the Problem and is immutable; if the shape of the work changes, that is a different Problem.

SOLVE — a success condition that can be satisfied: prove this theorem, extract this paper, produce a dossier meeting a stated bar. The Contract's result_kind is pass_fail; the revision's benchmark block is null. Several Submissions can be verified.
BENCHMARK — a measurable result to improve. The Contract's result_kind is score with a score block saying exactly how the number is computed; the revision's benchmark block is required (metric_name, direction, optional unit and baseline) and its direction agrees with the Contract's. A scored run still passes or fails; the score counts only when it passed. The best verified score is the current record.
DISCOVERY — the output is new work: candidate Problems, questions, technologies matching stated criteria, submitted as a structured list. Verification is typically a review stage, sometimes after a sandbox stage that checks the file's structure. benchmark is null.

Splitting work

One objective is often several Problems. Split when the pieces can be verified independently, attempted in parallel, need different structures (a DISCOVERY Problem to enumerate candidates, a SOLVE Problem per candidate), or when one Attempt on the whole would not plausibly finish. Size each Problem so a single Attempt can reach a Submission. Use a Campaign when the pieces share a mission, Skills or reviewers: "extract all reactions from these 2,000 papers" is one SOLVE Problem per paper in a Campaign, not one Problem asking for everything. Do not split so finely that a statement no longer stands on its own.

Writing the statement

The only document an author normally thinks about is statement.md (the revision's statement_md). It defines the stable task, and it is what a person or agent works against. Say:

what to produce, and its shape, in the same terms as the Contract's submission requirements;
what it is about, referring to each bound reference input by its binding key ("extract from the paper bound as paper_text; do not use another copy") and never restating its hash or size;
constraints: limits, formats, exclusions, licensing;
how the result is evaluated, in a sentence or two, deferring to the pinned Contract for the authoritative definition.

Static extra material that helps a solver (a format handbook, a worked example, domain guidance) goes in a pinned Skill or a bound reference file. Working knowledge discovered while solving goes in Attempt Notes, never back into the revision.

Do not gate starting. A solver's first call on a Problem is start_attempt, which returns the exact revision, Skills and reference inputs; Attempts are non-exclusive, so starting claims nothing and blocks no one. Collaborative work shares Notes with people who can access the Problem; independent entries hide peer work until cutoff. Never write "before starting, check whether someone else is working on this" into a statement or Skill. If coordination matters for your Problem, write it the other way round: "after starting, check the other Attempts' Notes when they are available; if someone else has it well in hand, end your Attempt as abandoned with a Note saying why."

Two more fields exist and are usually left alone. context_md is optional background kept for compatibility (defaults to null); it is not a place to keep updating with what has been learned. changelog_md is null for a first revision and, later, a sentence on why the new revision exists ("Tightened the output format; the previous wording allowed ambiguous units").

Reference inputs

Reference inputs are the exact bytes the task is about. The Contract declares each one's name and expected shape; the revision supplies the bytes.

Problem: analyze this research paper. The Contract declares paper. The revision binds paper → the exact uploaded file.

For public inputs, every surface reads the same bytes: solvers and reviewers at tfp://problem-revisions/{prv}/reference/{key} (and the short-lived signed download_url get_problem hands out next to it, which a credential-less process can fetch), sandbox checks at /input/reference/{key}/. Prose refers to an input by key only. The live findings-extraction Problem binds three keys, document_schema, paper_html and paper_text; a SAT Problem binds one, instance, and five hundred SAT Problems are five hundred revisions binding instance differently under one Contract.

At publication TfP checks that every required key is bound exactly once, that no unknown key is bound, that media type and size fit the declaration, and that the bytes exist with the server-computed hash. Bindings are immutable with the revision. Details in Authoring Verifiers.

A reference input the solver must not see, such as the labels a score is computed against, can be sealed. The Contract declares the slot sealed and which sandbox stages receive it. The revision's sealing block chooses label release, score feedback, a submission allowance and optional independent entries with a cutoff. Release dates are commitments: you can release earlier, never postpone or reseal. Revisions share evaluation rules and quota; changing scoring semantics requires fresh, nonoverlapping protected data. Start with Designing evaluations with hidden inputs, then consult Sealed reference inputs for exact constraints.

What the solver submits

The Contract says what a Submission must provide, usually one named artifact role with allowed media types (document, application/json). The statement describes the deliverable in the same terms. A Submission that does not match is not wrong; it is could not determine, with the unmet requirement recorded. See Authoring Verifiers.

Skills

A revision pins exact Skill Versions in skill_refs, each required (an Attempt is expected to load it: an output format, a workflow, a tool) or recommended (general guidance). Pinning the version means every Attempt loads the instructions the author tested; a new revision is how a Problem adopts a newer Skill. A Campaign's curated Skills appear on a revision only if the author pins them.

Where the Contract's sandbox check can be reproduced locally, pin a Skill that tells the solver how to run it before submitting. That preflight is not verification: it runs in the solver's environment; the authoritative check runs on TfP after the Submission exists.

Verification

A revision pins one VerifierContract or none. Reuse an existing Contract when its acceptance statement already says what you need (Campaigns usually settle on one Contract and one review protocol and reuse them for every Problem; that reuse is a practice, not a stored default). Author one when the semantics differ. Either way, what applies to an Attempt is the pin on its revision. How to author a Contract, and when a change needs a new one, is in Authoring Verifiers.

Revisions and versioning

Revisions are immutable and numbered from 1. Every Attempt keeps the revision it started on; a new revision moves nobody, and earlier Submissions are verified under the Contract their revision pinned.

A new revision is needed for a material change to the statement, to a pinned Skill Version, to a reference-input binding, to the pinned Contract, or to the benchmark block. It is not needed for title, summary, slug, status, priority_weight, campaign_id or owner_user_id, which are mutable on the Problem, nor for a corrected checker under the same Contract. Publishing a revision and updating the Problem's current_revision_id is the only way the effective task changes.

Authoring locally

A Problem is a directory:

problems/<slug>/
  problem.json                  title, summary, structure, ...
  statement.md                  the task
  skills/<name>/SKILL.md        optional pinned Skills
  reference/<key>.<ext>         optional; each file binds the key <key>
  verifier-contract.json        optional; or reuse an existing Contract
  review-protocol.json          optional; for review stages
  verifier-implementation.json  optional; when a builtin checker does not cover a sandbox stage
  fixtures/                     optional; conformance inputs

A Campaign folder holds campaign.json and problems/<slug>/. Every file you do not need is optional. The TfP Authoring Skill guides your agent through the convention, and pnpm tfp-lint <dir> runs the same validators TfP applies at publication. Durable private materialization over MCP is a separate step; see Authoring with Your Agent and the authoring reference for its current status. The end-to-end loop (lint, test, upload, durable private draft, explicit sharing) is in Authoring with Your Agent; custom checker images are in Custom sandbox checkers; one Python file of your own, with nothing to build, is in Script checkers.

Worked examples

Extract structured findings from a research paper (live)

SOLVE, in a Campaign that curates the extraction Skill and the reviewer pool.

Contract: two stages in order. structure_check (sandbox, builtin json-schema-check): the document parses and satisfies the bound schema; establishes shape only. fidelity_review (review): one accountable Campaign reviewer, using any agent they choose, reads the exact source and judges fidelity. One role, document. Three reference keys: document_schema, paper_html, paper_text.
Revision: statement.md names what to extract and refers to the source as paper_text (with paper_html as the same document as published). Binds all three keys. Pins the paper-extraction Skill as required; the Skill carries the field-by-field rules and how to fetch and validate the inputs.

A malformed document is could not determine; a schema violation is did not satisfy the verifier and never reaches the reviewer; a reviewer's uncertain leaves the run could not determine.

A SAT instance (canonical technical example)

SOLVE. Reuses the SAT Contract (schemas/examples/verifier-contract.sat.json; revision in schemas/examples/problem-revision.sat.json): one sandbox stage, one role assignment, one key instance.

{
  "revision_number": 2,
  "statement_md": "Find a satisfying assignment for the bound CNF instance.",
  "context_md": null,
  "benchmark": null,
  "verifier_contract_id": "vct_01jf90aa4rrc",
  "verifier_input_bindings": [
    { "key": "instance", "content_hash": "sha256:2645b7…ff012",
      "size_bytes": 4821, "media_type": "text/plain" }
  ],
  "skill_refs": [],
  "changelog_md": "Pins the SAT Contract and binds the instance."
}

A BENCHMARK with a scored Contract

BENCHMARK: solve as many held-out instances as possible within a time limit. The Contract has result_kind: score and one sandbox stage that runs the submitted solver against the bound instance_set and counts solved instances; its score.direction is higher_is_better. The revision binds instance_set and carries benchmark: { metric_name: "held-out instances solved", direction: maximize, unit: "instances", baseline: 388 }. A solver that fails to build is could not determine, not a low score. Replacing the instance set is a new binding, so a new revision, and earlier records are not comparable with it.

Every solver can read instance_set here, so the instances are held out only from the solver's program, not from its author. When the evaluation data itself must stay unseen, seal the slot: see Designing evaluations with hidden inputs, whose example scores predictions against sealed labels.

Checklist before publishing

The Problem is actionable, verifiable or evaluable, and adjudicable.
structure is right and will not need to change; work is sized so one Attempt can reach a Submission.
statement.md stands on its own: deliverable, inputs by key, constraints, how the result is evaluated; it never tells a solver to check something before starting an Attempt.
The deliverable matches the Contract's submission requirements.
A Contract is pinned, or none is pinned on purpose and the statement says results are recorded, not verified.
Every required reference key is bound and fits its declaration.
Pinned Skill Versions are marked required or recommended correctly; a preflight Skill is pinned where a local check exists.
For BENCHMARK the benchmark block agrees with the Contract's score; otherwise it is null.
changelog_md says why this revision exists (null for revision 1).
Status moves to open only after the revision is published.
If a slot is sealed, choose label and score releases, quota, entry visibility and cutoff deliberately; keep protected data out of public materials. Rehearse with synthetic data before committing real dates, rules and labels. See the sealed-evaluation planning guide.