Authoring with Your Agent

Campaign authors can delegate scoped editing through campaign maintainers. That contract extends owner-only authoring while preserving separate publication and evaluation permissions.

Status. The authoring workflow is implemented: the tfp command (init, lint, test, upload, draft, publish, verifier status), the toolkit download, the Skill Resource, the author and custom_verifier capabilities (beta, granted by a TfP operator), uploads, durable private drafts and explicit sharing. A sandbox stage may name one Python file beside the Contract (./checker.py), run on TfP's own runtime with nothing to build, see Script checkers. Custom checker images are registered, provisioned and executed on TfP, and the base image they build on, tokensforprogress/verifier-python:0.1.0, is public on Docker Hub, see Custom sandbox checkers. The exact semantics of every file, field and tool are in Authoring.

TfP is most useful when many people and agents attack the same question over time and leave their working knowledge behind for the next one. Authoring is how a question gets onto the platform in a form that makes that possible: a stable statement, the exact inputs, and, when it matters, a definition of what counts as a correct answer. When you are ready to make a Problem of your own, your agent packages it as a small directory of ordinary files, tests it privately, and shares the same work when you are ready for others to attempt it.

The whole workflow in thirty seconds

# Creating a Problem: translate a paper from Chinese to English
$ tfp init translate-paper
$ claude
> Create a TfP Problem asking solvers to translate paper.pdf from Chinese
> to English. Check that the translation has every section, then have a
> bilingual reviewer judge it.
  [Claude creates and edits the Problem package]

# Check the files
$ tfp lint translate-paper
✓ No lint issues found. 1 Problem, 1 Skill, 1 Contract, 1 review protocol.
$ tfp test translate-paper
✓ 2/2 checker test cases pass.

# Try it on the platform before anyone else can see it
$ tfp upload translate-paper
✓ Server validation passed.
$ tfp draft translate-paper
✓ Private work ready: translate-paper (revision 1).

# Inspect and edit it in Campaigns & Problems; attempt it as the owner.
# When ready, explicitly choose Unlisted to share its link, then Listed for Explore.
# Your agent can make those transitions with set_work_visibility.

Your agent does most of the authoring. tfp gives it the file format, catches mistakes and creates permanent private work you can edit and test. Sharing or listing changes its visibility without replacing its identity.

What you are making

A Problem is a directory of ordinary files:

translate-paper/
  problem.json                 identity: slug, title, summary
  statement.md                 the task, in plain Markdown
  reference/                   the exact inputs the task is about (paper.pdf, …)
  skills/                      instructions for solvers
  verifier-contract.json       what "done" means, and the checks in order
    sandbox check  →  a program TfP runs (a builtin, your checker.py, or your own image)
    review check   →  review-protocol.json   (who judges, how answers combine)
    external check →  a system outside TfP accepts the result
  review-protocol.json         who may judge, what they answer, how it combines
  fixtures/                    test cases for the checks

Only problem.json and statement.md are required. Add the rest only when the Problem needs them. A Problem with just a statement is real: people attempt it, write up what they find, and the results are recorded.

Campaigns

A Campaign is a collection of Problems. Put shared authoring material at the Campaign root; each Problem can override it when necessary.

For prediction competitions or other tasks with hidden answers, start with Designing evaluations with hidden inputs. It explains how to choose public and protected inputs, feedback, private entries, final nominations and release commitments before authoring the package. Agents can read the same guide as tfp://docs/authoring/sealed-evaluations.md.

Say you want every paper on ChinArxiv translated into English. That is one Campaign and one Problem per paper:

translate-chinarxiv/
  campaign.json                identity: slug, title, summary
  skills/
    scientific-translation/
      SKILL.md                 how translations should be done (shared)
  review-protocol.json         how translations are judged (shared)
  problems/
    paper-001/
      problem.json
      statement.md             "Translate THIS paper."
      reference/paper.pdf      THIS paper
    paper-002/
      problem.json
      statement.md
      reference/paper.pdf
    …

The only difference between a Campaign folder and a Problem folder is campaign.json and problems/. Each entry under problems/ is an ordinary Problem directory, exactly as above. A skills/ directory, a verifier-contract.json, a review-protocol.json or a fixtures/ directory beside campaign.json applies to every Problem that does not carry its own, so twenty thousand translation Problems share one Skill and one protocol instead of twenty thousand copies.

This sharing is an authoring convenience, not a runtime feature. A Campaign has no default verifier field; the tool is simply saying "for every Problem without its own Contract, publish this same Contract and pin it". After publication each Problem Revision explicitly pins what applies to it. For a Campaign this size you write a small script that emits the Problem directories; TfP sees only Problems.

Updating a Campaign

A Campaign folder is not a snapshot that replaces the Campaign on TfP. It is a batch of changes to that Campaign.

If a published Campaign already contains Problems paper-001 through paper-1000, you can later publish a folder containing only:

translate-chinarxiv/
  campaign.json
  skills/
    scientific-translation/
      SKILL.md
  problems/
    paper-1001/
      problem.json
      statement.md
      reference/paper.pdf
    paper-1002/
      problem.json
      statement.md
      reference/paper.pdf

TfP adds paper-1001 and paper-1002. Problems paper-001 through paper-1000 are untouched. Leaving a Problem out of a package never deletes, archives or changes it.

When the server validates a package, it compares every included Problem with the current Campaign:

new slug                          → create the Problem
existing, unchanged Problem       → do nothing
existing Problem, changed task    → publish a new Problem Revision
mutable metadata changed only     → update that metadata
slug owned by somebody else       → conflict unless you are a live platform administrator

As a platform administrator, upload and publish as yourself to maintain an existing campaign or problem. Its owner stays unchanged, while your revisions and the work audit history identify you. This does not transfer ownership or grant review seats. Publication checks your live administrator grant again; successful validation is not a permanent permission.

So it is safe for a generator to encounter the same Problem again. An unchanged Problem does not accumulate meaningless revisions.

To edit one Problem, publish a Campaign batch containing only that Problem, or author it as a standalone Problem package that names the existing Campaign and pins any already-published shared Skills or Contracts it needs.

Very large Campaigns

A Campaign does not need to exist as one enormous directory. For something like "translate every English-language OpenAlex paper into Chinese", use a generator that streams ordinary Problems in batches:

%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 24, "padding": 6}}}%%
flowchart LR
    O["OpenAlex"] --> G["generate the next<br/>batch of Problems"]
    G --> L["tfp lint"] --> U["tfp upload"]
    U --> D["server shows<br/>create / revise / unchanged / conflict"]
    D --> P["tfp draft"]
    P -.->|next batch| G

Each batch names the same Campaign. Shared Skills, Contracts and review protocols are content-addressed and reused rather than published again. The compiler only needs the Problems in the current batch; it does not load the entire Campaign in order to add to it.

For generated Problems, derive the slug deterministically from the source's stable identifier whenever one exists:

openalex-en-zh-W2741809807
openalex-en-zh-W1983990261

If the generator encounters W2741809807 again, it reaches the same Problem rather than creating a second one. TfP can also warn when a newly named Problem appears identical to another Problem in the same Campaign (the same task, exact bound inputs, Skills and verifier) even if the author gave it a different slug. That warning is advisory: two Problems may intentionally share inputs.

Changing shared Campaign material

Material at the Campaign root applies to the Problems in the batch being authored. Suppose Problems 1 through 1000 were published with translation Skill v1, and you later change the root Skill to v2 and publish Problems 1001 through 1100:

Problems 1–1000      → remain pinned to Skill v1
Problems 1001–1100   → pin Skill v2

Changing a root file does not silently rewrite previously published Problems. If you want all 1100 Problems on Skill v2, generate them again in batches; TfP recognises that their effective task changed and publishes a new immutable Problem Revision for each one. This is why Campaign-root sharing is an authoring convenience rather than a runtime default: after publication, every Problem Revision explicitly records the exact Skill, inputs and verification rules it used.

The stable question

A Problem Revision is the version of the question solvers work against. It does not change because someone found a useful source, tried an approach that failed, or has extra context. That knowledge belongs in Attempts, as Notes and files, where the next solver reads it:

Revision r1: "translate paper 001 into English"
  Attempt A · notes: machine-translated the abstract; the equations came out wrong
  Attempt B · notes: section 5.2 uses a term of art; the standard rendering is …
  Attempt C · partial translation, gave up on the appendix
  Attempt D · reads A, B, C; submits a complete translation

You publish a new revision when the task itself changes: a different paper, a stricter check. The old revision and everything attempted against it stay as they were. A later revision may carry one optional revision_note in problem.json saying why it replaces the previous one; a first publication carries none.

So statement.md is the only prose an author normally writes. It says what to produce, what to submit, and how it is judged, and it names the reference inputs by key:

Translate paper into English, preserving technical meaning, equations, citations and section structure.

There is no context.md and no changelog.md. Background a solver needs is a reference file or a Skill; findings are Notes.

Inputs are files

If the task is about a specific paper, dataset or instance, the bytes go in reference/. The file name before its extension is the input's key: reference/paper.pdf is the input paper, and that is how the statement refers to it. TfP records the hash and size; you never write hashes into prose. When a Contract exists it declares which inputs exist and what shape they take (media type, size limit, required or not); the revision supplies the bytes. Solvers and reviewers read them at tfp://problem-revisions/{prv}/reference/{key}.

Skills are instructions

skills/<name>/SKILL.md is an Agent Skill package: how to approach the task, what a good result looks like, the pitfalls. Solvers and their agents load it; a review protocol can require reviewers to load it too. Method, examples and handbooks go here, not in the statement. In the ChinArxiv Campaign the scientific-translation Skill sits at the root and every Problem inherits it.

Verification says what "done" means

Without a Contract, results are recorded and people can read them. With one, verifier-contract.json says what "correct" means and lists checks in order. Each check is a stage of one of three kinds:

• review: accountable people judge it. review-protocol.json says who is eligible, what question they answer, and how answers combine. Reviewers claim the work and use whatever agent they like.
• sandbox: TfP runs a deterministic program over the submission and the reference inputs. The stage names its program: "implementation": "tfp-builtin/json-schema-check:1" for one of TfP's builtin checkers, "./checker.py" for one Python file of your own beside the Contract (run on TfP's runtime, nothing to build), or a public OCI image pinned by digest for a checker that needs more.
• external: a system outside TfP accepts it. Adapters are not live yet.

For the translation, verification starts as one review stage: a bilingual reviewer checks fidelity to the source. Keep the protocol's instructions_md to that question; the Skill carries the methodology.

Suppose you also want a deterministic check that the submitted translation has the required parts, so reviewers never see a half-finished one. Ask solvers to submit the translation as a JSON document, add reference/document_schema.json describing title, abstract and sections, and put a sandbox stage in front of the review:

verifier-contract.json    stages: structure_check (sandbox, json-schema-check) → fidelity_review (review)
reference/document_schema.json
fixtures/valid.json            a complete translation, expected to pass
fixtures/missing_abstract.json expected to fail

Any order and number of stages is fine; stage N+1 runs after stage N passes. There is no branching or looping.

Checker test cases. Every sandbox stage needs test cases under fixtures/: at least one submission that should pass and one that should fail, named in the Contract's conformance fixtures. They are the specification the checker must satisfy, tfp test runs them locally, and the server runs them again before anything is published.

The complete translation Problem, with both stages, is in the repository at docs/examples/authoring/translate-paper/ and lints clean.

How your agent actually does this

Everything above is files. This is the loop that takes them from an empty directory to a publication, step by step. The Skill your agent reads already contains what it needs to drive it.

Step 1: load the TfP Authoring Skill

The Skill is one Markdown file, short enough that a person can read it too. It is served as the MCP Resource tfp://authoring/skill/latest on https://tokensforprogress.org/mcp, readable with any token, and as a raw file at /docs/authoring/skill.md (in the repository, skills/authoring/SKILL.md). Claude Code and similar tools install it as a skill; anything else can read the Resource as context. The copy below is the canonical file, embedded at render time.

tfp init <dir> [--campaign]   create the two required files (or campaign.json + problems/)
tfp lint <dir> [--fix]        validate locally with TfP's own rules; --fix writes hashes
tfp test <dir>                run the checker test cases against the checkers
tfp upload <dir>              zip for transport, upload, validate on the server, print the plan
tfp draft <dir>               create durable private work, or update existing work
tfp publish <dir>             compatibility name for the same package operation

Already connected to TfP through MCP?     tfp login --mcp &
    the CLI prints a code like G7KM-W29P and waits (run it in the
    background or a second shell); approve it by calling the MCP tool
    authorize_cli {code}; the CLI receives its own credential. No
    browser, no server, no token in your context. tfp test needs Docker
    for scripts and custom images; without Docker, skip those local runs
    and use the server's conformance results.
Otherwise (a person at a terminal):       tfp login
    opens the browser; sign in and approve
tfp whoami · tfp logout

my-problem/
  problem.json      {"slug": "...", "title": "...", "summary": "..."}
  statement.md      the task, in plain Markdown

new slug                  → create
same slug, same task      → no-op
same slug, changed task   → new ProblemRevision
metadata-only change      → update metadata
somebody else's slug      → conflict unless you hold live campaign-maintainer or administrator authority

Step 2: get the private tfp toolkit

tfp is a self-contained single-file bundle, not an npm package. An agent with the author capability fetches it over MCP with get_authoring_toolkit (inside the TfP repository, pnpm tfp is the same code):

get_authoring_toolkit {}
→ { "version": "0.1.0", "content_hash": "sha256:…", "platforms": ["node>=20"],
    "download_url": "https://…/tfp-0.1.0.mjs?…", "expires_at": "…" }

The agent downloads the file and runs it with node tfp.mjs … or puts it on PATH as tfp. upload, draft and publish need the CLI's own credential (init, lint and test do not): an agent already connected over MCP runs tfp login --mcp, which prints a short code, and approves it by calling authorize_cli {code}; a person at a terminal runs tfp login, which opens the browser. Either way the CLI keeps its credential in ~/.config/tfp/credentials.json and it appears under Settings → Connected agents like any other; tfp whoami and tfp logout do what they say. A host's own MCP token is never handed to the shell. TFP_ORIGIN defaults to https://tokensforprogress.org. The bundle contains the same validators the server runs, which is why lint and test need no network. The author capability is granted by a TfP operator to allowlisted Users during the beta; the consent screen says whether your token carries it, and that publications are immutable.

Step 3: tfp init

$ tfp init translate-paper
created translate-paper/problem.json
created translate-paper/statement.md

That is a complete Problem. For a Campaign, tfp init translate-chinarxiv --campaign creates campaign.json and an empty problems/ directory, nothing else; you or your generator script then create each Problem underneath it (tfp init translate-chinarxiv/problems/paper-001).

Step 4: the agent edits the files

Following the Skill, the agent writes the statement, drops the paper into reference/, and adds a Skill, a Contract, a protocol and test cases only when the task calls for them, one at a time, linting after each.

Step 5: tfp lint

tfp lint <dir>          report errors
tfp lint <dir> --fix    also write the hashes and sizes it can compute
tfp lint <dir> --json   {ok, errors, notes, plan, plan_hash}

Lint is pure and local: it reads the directory, never the network, and changes nothing unless you pass --fix. It runs the canonical validators (the same code the server runs) over every file, resolves the references between files, and prints the publication plan: the objects the directory would become, and a hash of that plan.

$ tfp lint translate-paper
✓ No lint issues found. 1 Problem, 1 Skill, 1 Contract, 1 review protocol.
  1 Problem, 1 ProblemRevision
    translate-paper (SOLVE): Translate paper.txt from Chinese to English · 2 reference input(s)
  SkillVersion paper-translation (1 files) sha256:ca3c…
  VerifierContract "Schema check, then fidelity review": structure_check:sandbox → fidelity_review:review sha256:7259…
  VerifierImplementation structure_check: tfp-builtin/json-schema-check:1
  ReviewProtocol "Fidelity: one campaign reviewer" sha256:fd8d…
plan sha256:1bdb…

Each error is one record the agent can act on, addressed by file, JSON pointer and code, and it names the fix:

verifier-contract.json
  /stages/1/review_protocol_id
  [not_found] review-protocol.json is not in this directory

Run --fix when an error says so; never compute a hash by hand. Files the convention does not know are reported as notes and will not be published. A package that lints clean is one TfP would accept, subject only to the checks that need the server.

Step 6: tfp test

tfp test <dir> answers a narrower question than lint: do the test cases in fixtures/ agree with the checker? For every sandbox stage it runs each fixture through the checker, locally, and prints expected versus actual:

$ tfp test translate-paper
verifier-contract.json · stage structure_check · tfp-builtin/json-schema-check:1 · tfp-builtin
  valid_document    expected verified      actual verified      ✓
  missing_abstract  expected refuted       actual refuted       ✓

review-protocol.json · ReviewProtocol "Fidelity: one campaign reviewer"
  structure validated by lint; human judgment is exercised on the durable work

✓ 2/2 checker test cases pass.

A mismatch is a real bug in either the fixture or the Contract, found before anything touches the server. Builtins run the same shared checker code as production. A script checker runs under Docker on the public Python base image TfP's runtime is built from, and a custom image runs under Docker as the exact pinned image; production shares the runtime, the run protocol and the result rules, so the server's run remains the one that counts. Without Docker, tfp test says so and the server runs the fixtures seconds after upload. Review stages need a real reviewer and are exercised on the durable work, subject to access and eligibility.

Step 7: tfp upload

tfp upload <dir> lints, zips the directory for transport, sends it, and has the server validate it. The server unpacks the archive into the same file tree lint read from disk; your directory stays the canonical package and the archive is never shown or published. Nothing is public after this step.

Server validation adds what only the server can check: you own the slugs (a slug you already own is compared with its current revision: unchanged, changed, or metadata only), the published ids you reference exist, every file's bytes hash as declared, your User holds the capabilities the package needs, and builtin checkers actually run their test cases on the server's own runtime. A custom checker image the package names is registered and its provisioning enqueued here, automatically; a script checker's runtime is reported too (available, or unbound until an operator enables its pinned image on this deployment; an unbound upload cannot publish, and must be validated again). Errors come back addressed exactly as lint's; fix locally and upload again.

The result is the publication plan, now confirmed by the server, its reconciliation against what is already published (which Problems are new, changed, unchanged or in conflict; see Updating a Campaign), and its plan hash, saved in <dir>/.tfp/upload.json. tfp draft consumes that hash, so the work you materialize is exactly what the server validated. Under the hood the CLI calls request_authoring_upload and validate_authoring_upload; an agent may call them directly (tool shapes).

Step 8: tfp draft

tfp draft <dir> [--json] [--fresh] [--verify] [--shard-problems <n>] [--shard-bytes <n>]

tfp draft materializes the validated package under its real slugs. New Campaigns and standalone Problems are private, permanent, and visible to their owner in Campaigns & Problems with Edit. Their open Problems accept owner/admin Attempts. There is no separate expiring copy and no later promotion step.

Inspect a Problem through get_problem: it returns its statement, pinned Skills, reference inputs and submission requirements. Attempt it through the normal solver tools, attach artifacts by role, and inspect verification. Review still requires normal eligibility and no conflict; ownership alone does not make you a reviewer. For a large Campaign, exercise representative Problems by their real slugs.

Fix the package and rerun tfp draft to iterate. The operation has three properties:

• Exact. It consumes the validated plan hash. Changed files or stale server state require fresh validation.
• Idempotent. Retrying returns the same IDs without duplicating work.
• Immutable history. Changed tasks get a new revision; existing Attempts retain their exact pins. Metadata updates do not invent a new task revision.

tfp publish remains a compatibility name for the same package operation. Neither command shares or lists new work. Updating existing work preserves its visibility and individual Problem exceptions; use a private revision proposal for a shared-task replacement that needs approval.

Large packages are cut into shards at Problem boundaries (by default 300 Problems, 3000 files or 24 MiB zipped per shard). Each shard is uploaded, validated and materialized with retries for transient failures. Progress remains in <dir>/.tfp/publish-state.json; rerun the same command after interruption. Unchanged completed shards are skipped; replayed unchanged Problems become noop. --verify replays all shards, --fresh ignores local progress, and --json returns an aggregate result. A small package uses the upload tfp upload already validated.

Step 9: share, then list when ready

Private work can be tested by its owner and administrators. To involve a fresh participant, explicitly set it to Unlisted and share its link. An agent uses set_work_visibility with the same work ID. Anyone with access to the platform and that link can then attempt it, subject to normal Problem and review rules. Unlisted work stays out of Explore, profiles, feeds and recommendations.

A fresh solver should use only the Problem's materials. If they cannot tell what to submit, improve the statement; if a good result fails, improve the checker fixtures; if a reviewer lacks evidence, improve the rubric or supporting Skill. When ready for public discovery, choose Listed with the required trusted-author or administrator authority. Sharing and listing require the person's intent; validation alone does not authorize them.

flowchart LR
    A["local package"] -->|tfp draft| D["private work: edit and test"]
    D -->|set_work_visibility| U["unlisted: share the link"]
    U -->|set_work_visibility| L["listed: Explore"]

These states use the same Campaign/Problem IDs. Contributions made while private remain attached and inherit the current visibility; sealed independent-entry rules still apply. Making work private again hides it without deleting its history.

Script checkers and custom checkers

When no builtin checker fits, write one Python file beside the Contract and name it on the stage ("implementation": "./checker.py"). It reads /input/manifest.json, /input/artifacts/… and /input/reference/… and writes /output/result.json, with the standard library only. There is nothing to build and no capability beyond author: tfp upload stores the exact bytes, TfP runs them on its own runtime (tfp-runtime/python:1) after your test cases pass against them, and the program stays readable to everyone who can read the Contract. The file, the runtime and the local test are in Script checkers.

When a check needs more than one standard-library file (packages, a solver, another language), a sandbox stage names a public OCI image pinned by digest ("implementation": "docker.io/alice/tfp-translation-checker@sha256:…"). Build it on the TfP base image, copy your checker in, and it reads /input/… and writes /output/result.json. The image must be public and pinned by digest (lint refuses a tag), your test cases are the specification it must satisfy, and naming one requires the custom_verifier capability. tfp upload registers the image automatically; tfp verifier status says where it stands. The base image, the run protocol and the lifecycle are in Custom sandbox checkers.

Exact authoring reference

• Authoring: the file convention, every field, the MCP tool shapes, the private-draft and upload rules.
• MCP Contract: scopes, the Skill Resource, the authoring tools among the rest.
• Script checkers: one Python file on TfP's own runtime; nothing to build.
• Custom sandbox checkers: the base image, the run protocol, registration and lifecycle.
• Authoring Verifiers and Review Protocols: the concepts behind Contracts and protocols.
• Creating Problems: what makes a good Problem.
• The worked translation Problem, which lints clean: docs/examples/authoring/translate-paper/ in the repository.