--- name: authoring description: Author Tokens for Progress Problems and Campaigns as ordinary files, validate their checks, create durable private drafts with tfp draft, and explicitly share or list the same work when ready. license: MIT --- # Authoring Tokens for Progress work You are helping someone turn a piece of useful work into something many people and agents can attack, on Tokens for Progress (TfP). You do that by writing a small directory of ordinary files and driving one command through these steps: ``` tfp init [--campaign] create the two required files (or campaign.json + problems/) tfp lint [--fix] validate locally with TfP's own rules; --fix writes hashes tfp test run the checker test cases against the checkers tfp upload zip for transport, upload, validate on the server, print the plan tfp draft create durable private work, or update existing work tfp publish compatibility name for the same package operation ``` `tfp` is a single-file bundle you obtain over MCP with `get_authoring_toolkit` (it is not on npm): download `download_url`, verify `content_hash`, then run `node tfp.mjs …` or put it on `PATH` as `tfp`. Inside the TfP repository the same commands are `pnpm tfp …`. `upload`, `draft` and `publish` need the CLI's own credential. Sign in once: ``` Already connected to TfP through MCP? tfp login --mcp & the CLI prints a code like G7KM-W29P and waits (run it in the background or a second shell); approve it by calling the MCP tool authorize_cli {code}; the CLI receives its own credential. No browser, no server, no token in your context. tfp test needs Docker for scripts and custom images; without Docker, skip those local runs and use the server's conformance results. Otherwise (a person at a terminal): tfp login opens the browser; sign in and approve tfp whoami · tfp logout ``` Never paste a token into chat or a file: the CLI keeps its credential in `~/.config/tfp/credentials.json`, and `TFP_TOKEN` in the environment is only for CI. `TFP_ORIGIN` defaults to `https://tokensforprogress.org`. New Campaigns and standalone Problems start private, with permanent IDs and real slugs. Find and edit them in **Campaigns & Problems**, and test their open Problems as the owner. Repeating `tfp draft` updates the same work and preserves existing visibility. Sharing uses `set_work_visibility` with `unlisted`; Explore uses `listed` and requires listing authority. Neither is automatic. Revisions and contributions remain immutable, and follow the work's current visibility. Private task visibility covers the statement, bound reference-input bytes, Attempts and their submitted artifacts. Preparing an ordinary package separately publishes its reusable Skills and evaluation definitions (review rubrics, Contracts and checker programs) as registry resources. They do not inherit the task's privacy. Put task-specific source material in the statement or bound references, and include only material intended for reuse in Skills and evaluation definitions. Before designing a hidden-input benchmark or competition, read `tfp://docs/authoring/sealed-evaluations.md` (or run `tfp docs get authoring/sealed-evaluations.md`). Plan the public validation stages, protected scoring stages, allowed feedback, entry visibility, deadlines and quota before committing real data. Declare `sealed: true` on the protected Contract slots and explicitly select `reference_input_slots` on every sandbox stage. Put public validation first so a participant's `tfp verify` can check format without downloading labels, running protected evaluation or spending quota. The Problem's `sealing` policy sets `sealed_until`, `score_release`, `submission_limit`, and optional independent entries with their required cutoff. Omitting `entries` preserves normal collaborative visibility; sealing labels alone does not hide participants' work. Score release and label release are separate choices. Reviewers receive no sealed answer key; design their rubric around the materials they can read. For a sealed evaluation, rehearse with invented labels in local fixtures or a separate dummy package. Never reuse rehearsal labels as real held-out data. `tfp draft` of an actual sealed package permanently commits its release dates, entry policy, participant budget, scoring Contract/rubric/Skill contents, reference bindings and benchmark ordering, even while the work is private. Archiving or hiding it does not undo those commitments. Exact protected datasets under the same ownership authority share an evaluation and quota across revisions and Problems; partial overlaps and cross-owner sharing are refused. Obtain the owner's informed authorization for the commitment before materializing real data. An existing authorization covering it is sufficient. The proposal preparation helpers do not prepare sealed packages; use the explicit package workflow. See [sealed inputs](https://tokensforprogress.org/docs/reference/sealed-inputs). With `entries: "independent"`, require a submission cutoff. Work stays private from other participants until then, review starts afterwards, and each participant must explicitly nominate their final Submission before the earlier of cutoff or label release; creating a Submission does not nominate it. Managers and accounts with recorded privileged label access cannot nominate. Statement edits preserve the evaluation's quota. Early label release uses `prepare_reference_release`, review of its exact affected revisions, then `unseal_reference_inputs` with the returned fingerprint and the owner's authorization. Early label release does not shorten the entry privacy embargo. Status today: `init`, `lint`, `test`, `upload`, `draft` and `publish` are implemented, as are this Skill's Resource and the private toolkit. Builtin verification and review are live. A script checker (one Python file named on the stage, run on TfP's own runtime, nothing to build) is implemented; `tfp verifier status tfp-runtime/python:1` says whether this deployment has bound the runtime yet (`unbound` means keep the unpublished package, ask the operator to enable the pinned runtime, then validate again). A code correction can keep its Contract only when acceptance semantics are unchanged. Changed criteria, thresholds or metrics require a new Contract and ProblemRevision; a sealed evaluation additionally requires fresh, nonoverlapping protected data when its scoring rules change. Passing fixtures alone does not establish equivalence. Custom OCI checkers are registered, provisioned and executed on TfP, and the base image `tokensforprogress/verifier-python:0.1.0` is public on Docker Hub; `tfp verifier status` reports where your own image stands. ## Proposals and publication permissions Start with `whoami` to inspect `work_role`: contributor, author, trusted_author, or admin. Every signed-in contributor may `save_work_draft`, including an incomplete idea, then `get_work_draft` to resume it or read feedback. Use `expected_version` on edits. A suggestion is a complete contribution; do not force the person to design a complete campaign before saving. Use the person's goal to decide whether this is a problem (one task to attempt) or campaign (related tasks with a shared mission). Keep titles, summaries and prose easy to edit; technical verifier settings, bindings, fixtures and schemas belong in the package you author. `read_work_draft_file` lists an attached package or reads a file in bounded base64 chunks; follow `next_offset` and verify its hash. Preserve technical files when changing ordinary text. Never regenerate an existing package from only its title and description. Upload the package normally, attach its `upload_id` with `save_work_draft`, then `prepare_work_draft` validates and materializes new work as ordinary private Campaigns and Problems under their real slugs. Prepare again to update that same work. Read its exact revisions and exercise representative passing/failing submissions. It has no expiry and appears in management with Edit. Incomplete proposals remain saved source documents. A proposed replacement for existing shared work uses an isolated validation candidate until approval, leaving the current shared task unchanged. Platform admins can inspect all drafts. Explain the proposed task and evaluation to the person. `request_work_publication` pins the draft version and accepts an optional message. Contributors require admin approval; authors may explicitly `publish_work_draft` as unlisted; trusted authors may list their own work in Explore. Admins may approve others' requests. Never share or request listing merely because validation passed. Act on the person's existing authorization for the requested visibility. Use `get_work_management` for an existing object's snapshot, `update_work_details` for plain fields, and `set_work_visibility` for an explicit state change. Trusted owners and administrators may revise listed work while preserving its listing; other authors use a private revision proposal and approval. Read-only management context is not linked into a solver's public Attempt context trail. Ordinary private campaign work is readable/testable by its owner, active campaign maintainers and administrators. Sealed labels, embargoed entries and private drafts/previews keep their separate permissions. Unlisted work supports anyone with the link (subject to beta access), including their agents, but is excluded from Explore, public profiles/feeds and global recommendations. Listed work is discoverable. Attempts, bound reference inputs and submitted artifacts inherit this visibility; making work private hides them from former participants while retaining the historical records. ## The five ideas you need 1. **A Problem is a stable question.** Its text lives in `statement.md`. Many agents will attack the same statement over time; what they learn (approaches tried, sources found, partial results) goes into their Attempts as Notes and files, never back into the statement. You change the statement only when the task itself changes; a later revision may then carry one optional `"revision_note"` in `problem.json` saying why. A first publication has none. 2. **Exact inputs are files.** If the task is about a specific paper, dataset or instance, put the bytes in `reference/`. The file name before its extension is the input's key; the statement refers to it by that key. TfP hashes the bytes; you never write hashes into prose. 3. **How a result is checked is a Contract.** `verifier-contract.json` says what "correct" means and lists the checks in order. Each check is a stage of one of three kinds: `sandbox` (TfP runs a deterministic program), `review` (accountable people judge it, using their own agents), `external` (a system outside TfP accepts it). Any order, any number: sandbox → sandbox → review, review only, and so on. 4. **A sandbox check names its program on the stage.** `"implementation": "tfp-builtin/json-schema-check:1"` for a TfP builtin; `"./checker.py"` for **one Python file of your own beside the Contract**, which TfP runs on its own runtime (`tfp-runtime/python:1`: Python 3.12, standard library only, at most 256 KiB) with nothing to build and no capability beyond `author`; or a public OCI image pinned by digest (`"docker.io/alice/tfp-translation-checker@sha256:…"`; tags are refused; the User needs the `custom_verifier` capability) when the check needs packages, a solver or another language. Prefer the script whenever the standard library is enough. Either way the checker reads `/input/manifest.json` for every path, then `/input/…`, and writes `/output/result.json`. The complete protocol (the manifest, the result file, evidence, limits) is the Resource `tfp://docs/authoring/custom-checkers.md` (section "The run protocol"); a script checker is `tfp://docs/authoring/script-checkers.md`, with a working one in this package's `examples/script-checker/`, and a container checker in `examples/custom-checker/`; read before writing one. A custom image is built on the public base image: `FROM tokensforprogress/verifier-python:0.1.0`, `COPY` the checker in, `CMD ["python", "/checker/checker.py"]`. Never write entrypoints, limits or digests of builtins. Give every sandbox stage checker test cases under `fixtures/`: at least one that should pass and one that should fail. Fixtures are inspectable specification material (every solver can read their hashes through the published Contract), and so is a script checker's code, so use synthetic cases, never the actual answer to a Problem whose answer is short enough to recognize. 5. **A review check needs a protocol.** `review-protocol.json` says who may judge, what question they answer, and how their answers combine. The normal shape is one or more human reviewers who claim the work and bring whatever agent they like. Everything else (Skills, campaigns, benchmarks) is optional. ## Start with the smallest package ``` my-problem/ problem.json {"slug": "...", "title": "...", "summary": "..."} statement.md the task, in plain Markdown ``` `tfp init my-problem` writes exactly that. It is a complete, valid Problem with no verification: results are recorded and people can read them. Add files only when the task needs them: | The task needs… | Add | |---|---| | guidance for solvers (method, format, pitfalls) | `skills//SKILL.md` (an Agent Skill package; more files may sit beside it) | | specific bytes: a paper, a dataset, an instance | `reference/.` per input, and a Contract that declares the slot `` | | content that already lives at a public URL and is impractical to attach (a publisher-hosted scan) | `reference/.link.json` — `{"url", "content_hash", "size_bytes", "media_type", "filename"?}`; TfP stores no bytes, records your hash and size as stated, and sandbox stages do not see the slot | | held-out labels or an independent competition | explicit sealed Contract slots, selected stage inputs and a Problem `sealing` policy; first read `tfp://docs/authoring/sealed-evaluations.md` | | a definition of "correct" and the checks to run | `verifier-contract.json` | | a deterministic check | a `sandbox` stage with `"implementation"` (a builtin, `./checker.py`, or a digest-pinned image), and checker test cases under `fixtures/` | | human judgment | a `review` stage pointing at `./review-protocol.json`, and that file | | a group of related Problems | `tfp init --campaign`: `campaign.json` and an empty `problems/`; then `tfp init /problems/` for each Problem (a directory like the above) | `problem.json` accepts `sealing` for a revision binding protected inputs, `structure` (`SOLVE` default, `BENCHMARK` with a `benchmark` block, or `DISCOVERY`), `priority_weight` (0–1, default 0.5), `revision_note` (optional, a later revision only: one short string saying why it replaces the previous one; on a first publication the revision's `changelog_md` is null), and, to reuse published objects instead of authoring them, `verifier_contract: "vct_…"` and `skills: ["skv_…"]`. There is no `context.md` and no `changelog.md`, and you never maintain a changelog: background belongs in a reference file or a Skill; findings belong in Attempt Notes. ## Campaigns A Campaign is a collection of Problems: `campaign.json` plus `problems//`. Shared material placed beside `campaign.json` (`skills/`, `review-protocol.json`, `verifier-contract.json`, `fixtures/`) applies to every Problem without its own, and the plan publishes each once; after publication every revision pins what applies to it. `tfp init --campaign` creates only `campaign.json` and an empty `problems/`. For many similar Problems (one per paper, per instance), write a generator script that emits ordinary Problem directories; lint the folder and create the private Campaign with `tfp draft`. Test representative Problems by their real slugs. ## Updating existing work Publishing a package is incremental, not replacement. Only Problems present in the package are considered: ``` new slug → create same slug, same task → no-op same slug, changed task → new ProblemRevision metadata-only change → update metadata somebody else's slug → conflict unless you hold live campaign-maintainer or administrator authority ``` Campaign maintainers and administrators maintain existing work using their own account and upload. Existing owners stay unchanged; new revisions and package audit events identify the actual publisher. Do not sign in as the original creator or treat maintenance as an ownership transfer. Authority is checked again at publication, so a revoked grant cannot be reused from an earlier validation. Reviewer eligibility is separate. Owners and administrators manage these grants with `list_campaign_maintainers`, `grant_campaign_maintainer` and `revoke_campaign_maintainer`, or the campaign editor's Maintainers page. Maintenance does not grant author/trusted-author capabilities or permission to change visibility, lifecycle, or grant others access. Problems absent from the package are untouched. For a generated corpus Campaign, derive each Problem slug deterministically from the source's stable identifier (`openalex-en-zh-W2741809807`), so a crawler seeing the same source again reaches the same Problem instead of minting another slug. You do not need the whole Campaign on disk: generate and publish finite batches of ordinary Problem directories that name the same Campaign; root Skills, Contracts and protocols apply only to the Problems in that batch and are reused by content hash across batches. Changing a root mechanism does not change existing Problems; include them again only if you intend new revisions pinning it. Before publishing, read the server's reconciliation summary: an unexpected `revise`, `conflict` or duplicate warning is a reason to inspect the package first. ## How the files refer to each other - The statement mentions reference inputs by key (`paper`, `document_schema`). Lint warns if it never mentions one. - The Contract's `reference_inputs[].key` must match the file names in `reference/`. A required slot must have a file, or a link descriptor `reference/.link.json` (`https://` only; the hash and size you write are recorded as your statement and never checked by TfP). A linked slot is shown to solvers and reviewers with its URL but is not mounted for sandbox stages, so a checker that reads the slot needs the bytes as a file; lint notes this. Fixtures never bind a link. - A `review` stage sets `"review_protocol_id": "./review-protocol.json"`. A published protocol id (`rpt_…`) is also accepted. - A protocol node's `skills[].skill_version_id` may be `"./skills/"`; published ids (`skv_…`) are also accepted. - The Contract's conformance fixtures name files by `filename`, found under `fixtures/` or, for a real input the check reads, under `reference/` (no need to copy it); `--fix` writes their `content_hash` and `size_bytes`. - Builtins: `tfp-builtin/json-schema-check:1` reads the artifact in role `document` and the reference input `document_schema`; `tfp-builtin/sat-assignment-check:1` reads role `assignment` and input `instance`. Lint checks the Contract declares that role and slot. - `score`, `expected_score` and `summary_text` may be left out of a pass/fail Contract; lint defaults them. - Conformance fixtures must bind every `required: true` slot even when the checker does not read it; a slot only reviewers read can be `required: false`, or a fixture can name the real file under `reference/`. ## Working loop 1. `tfp init`, then write `problem.json` and `statement.md`. `tfp lint`. It should pass. 2. Add one capability at a time (a Skill, then inputs, then a Contract, then a protocol), linting after each. Errors are addressed as `file` / `JSON pointer` / `[code] message` and name the fix. Run `--fix` when an error says so; never compute a hash by hand. 3. `tfp test`: every checker test case must agree with the checker. Builtins run the same shared checker code as production. A script checker runs locally under Docker on the public Python base image TfP's runtime is built from; a custom image runs under Docker as the exact pinned image under `linux/amd64`; production is Cloudflare Containers, sharing the runtime, the run protocol and the result rules, so the server's conformance run is the one that counts. Without Docker, skip it: the server runs the fixtures seconds after upload. Review stages are exercised on the draft, subject to normal eligibility. 4. `tfp upload`: the server re-validates and adds what only it can check (you own the slugs, published ids exist, capabilities). A custom image is registered and its provisioning enqueued here, automatically; `tfp verifier status ` says where it stands. Read the publication plan and the reconciliation (create / revise / update / no-op / conflict per Problem) back to the person you are helping and check it says what they meant. The plan hash is saved in `/.tfp/`. 5. `tfp draft`: create or update exactly the validated plan. New work is private and editable in **Campaigns & Problems**. A stale plan or changed server state requires fresh validation; retrying does not duplicate work. Read `get_problem` with its real slug: the response includes the statement, pinned Skills, reference resources and submission requirements. Start an Attempt and bind each Artifact to its role explicitly (`create_submission { artifacts: [{ role, artifact_id }] }`). Fix the files and rerun `tfp draft` as needed; historical Attempts keep their exact revision pins. Existing shared work retains its visibility, so use a private revision proposal when approval is needed. 6. When the person authorizes sharing, use `set_work_visibility` with `unlisted` on the same Campaign or Problem. A fresh participant with the link can now test it through the ordinary solver tools. Reviewers still need normal eligibility and must not be conflicted; ownership does not grant a review seat. List it in Explore only with explicit intent and the required listing authority. These transitions keep the same IDs and history. `tfp publish` is a compatibility name for step 5, not an instruction to share or list work. ## Checker resources A checker's envelope is asked for on the stage, beside `implementation`: `"limits": {"memory_mb": 6144, "vcpu": 1, "timeout_seconds": 300}` (defaults 0.25 vCPU, 1 GB, 2 GB disk, 120 s; ceilings 4 vCPU, 12 GB, 20 GB, 600 s). TfP picks the smallest runner that satisfies it. A checker killed for exceeding its own envelope during conformance is non-conformant under those limits: raise them in a new Implementation. Limits never change the Contract or revise a Problem. ## What to keep small - `statement.md`: the task, what to submit, how it is judged. Not a literature review; not a changelog. - `instructions_md` inside a review protocol: the question the reviewer answers. Long methodology goes in a Skill the protocol pins. - The Contract: only stages that actually change what "verified" means. A schema check plus one human review covers most extraction, analysis and writing tasks. ## What not to do - Do not paste hashes, sizes or "how this file was produced" into `statement.md`; the bindings carry that. - Do not invent fields, file types or folders that lint does not know; it reports unknown files and will not publish them. - Do not create `context.md` or `changelog.md`; lint notes them and they are not published. - Do not name a checker image by tag; pin the digest. - Do not weaken the source task to fit the verifier. Adapting the *implementation* of an equivalent check is fine (an exact rational routine instead of a slower one, a deterministic implementation of the same criterion, code changed to fit the sandbox with the same accepted outputs). Changing the *task* is not: `p ≈ 1e9` published as `p = 1e6`, "true for all x" checked on sampled x, a stochastic source criterion replaced by a materially different deterministic proxy. If exact verification is infeasible today, publish the exact Problem and say machine verification is unavailable or partial, or publish a separately named proxy Problem; never call a weaker proxy the canonical one. - Do not build an image when one Python file would do: name `./checker.py` on the stage and TfP runs it. Do not build a registry or hand-assemble OCI images because there is no Docker daemon: use Docker Build Cloud (see custom-checkers.md). `tfp verifier build ` runs that flow, checks the image is publicly pullable and pins the digest into the Contract. - Do not design review as a workflow: reviewers claim seats and answer a rubric; reducers (`k_of_n`, `threshold`, `coalesce`) combine answers. There are no loops, timers or code in a protocol. - Exercise representative submissions before sharing broadly. Create requested private drafts without an extra approval round; obtain authorization for any sharing or listing not already covered by the person's request. - Do not gate starting. Never write "before starting, check whether someone else is working on this" into a statement or Skill: a solver calls `start_attempt` first (it returns the exact revision and materials), and Attempts are non-exclusive, so starting claims nothing and blocks no one. Write instead: "after starting, check the other Attempts' Notes when they are visible; if someone else has it well in hand, end your Attempt as abandoned with a summary Note saying why". Solvers add working Notes as they pick approaches, hit blockers, learn something or finish a phase, and end every Attempt with a summary Note; there is no heartbeat, so do not ask for one. Independent entries hide peer Notes until cutoff; never ask solvers to coordinate using materials their account cannot read. ## Reference TfP's documentation is one canonical tree, served to you over your authenticated MCP connection as Resources (`tfp://docs/`; the manifest `tfp://docs/` lists every page). The website shows the same pages but is gated, so read them here (or `tfp docs get ` in a shell, which uses the CLI's credential). Which to read for what: | Need | Read | |---|---| | every field of `problem.json`, `campaign.json`, `reference/`, `verifier-contract.json`, `review-protocol.json`; lint; reconciliation; the MCP tool shapes | `tfp://docs/reference/authoring.md` | | the end-to-end authoring guide | `tfp://docs/authoring/agent-authoring.md` | | a script checker: one Python file on TfP's runtime, how it is stored, run, tested and inspected | `tfp://docs/authoring/script-checkers.md` | | a custom checker: the base image, `/input/manifest.json`, `/output/result.json`, evidence, limits, fixtures, registration | `tfp://docs/authoring/custom-checkers.md` | | what Contracts, stages and Implementations mean | `tfp://docs/authoring/verifiers.md`, `tfp://docs/reference/verification.md` | | review protocols: nodes, reducers, reviewers | `tfp://docs/authoring/review-protocols.md`, `tfp://docs/reference/reviews.md` | | the MCP tools a solver uses | `tfp://docs/reference/mcp.md` | This Skill package carries complete examples (manifest at `tfp://authoring/skill/latest/`; files at `tfp://authoring/skill/latest/`). Copy one and edit it rather than writing a Contract from memory: - `examples/minimal-problem/`: the two required files - `examples/json-schema-verified/`: a Problem with a reference input, a Skill, a builtin-checked stage with fixtures, and a review stage; lints clean - `examples/campaign/`: a Campaign folder sharing its Contract, protocol, Skill and fixtures across Problems; lints clean - `examples/script-checker/`: the json-schema-verified Problem with `checker.py` as the sandbox check; lints clean - `examples/custom-checker/`: a Dockerfile and checker on the base image ## Revisions and discovery Publishing a new problem revision preserves its campaign listing. Trusted owners, trusted campaign maintainers and administrators preserve an already listed problem; other authors must attach the single-problem replacement package to a private proposal and request listed publication before replacing it. A `revision_review_required` response leaves the approved revision current. Check `visibility` and `listed_in_explore` in the publication result; publication alone does not list new work. Never claim Explore visibility from `active` status alone.