Custom Checkers

Status: live. The reference format, the public base image and harness, local testing under Docker, registration, provisioning of one runner per runtime digest, execution on TfP and per-Implementation conformance are implemented and running; see Status at the end for what was verified and when.

What a custom checker is

A sandbox stage of a verifier is computed by a program TfP runs in isolation. That program is one of TfP's builtins (tfp-builtin/sat-assignment-check:1, tfp-builtin/json-schema-check:1), a script checker (one Python file beside the Contract, run on TfP's own runtime; the shortest path when no builtin fits), or a custom checker: a public container image you publish, that TfP runs under the same run protocol and holds to the same rules. Reach for an image when the check needs more than one standard-library Python file: packages, a solver, another language, a large toolchain. You write the image; TfP decides whether it is trustworthy enough to speak for the Contract.

Naming the image

A stage names its checker in verifier-contract.json:

"implementation": "docker.io/<user>/<repo>@sha256:<64 hex>"

The accepted form is <registry>/<repository>@sha256:<64 lowercase hex>. docker.io is the default registry (docker.io/<user>/<repo> and <user>/<repo> mean the same thing; an official image is docker.io/library/<repo>). Other public registries work in the same form (ghcr.io/<org>/<repo>@sha256:…); the schema is registry-agnostic and Docker Hub is the recommended home. The digest is the only immutable name an image has, so tags are refused at registration: docker.io/me/checker:v3 is a lint error, because whatever v3 points at tomorrow is not what the Contract was reviewed against. Tags are fine while you build and push; pin the digest when you write the Contract.

To find the digest of an image you have pushed:

docker buildx imagetools inspect docker.io/me/checker:v3
docker inspect --format '{{index .RepoDigests 0}}' docker.io/me/checker:v3

Both print docker.io/me/checker@sha256:…; paste that whole string. If you built for several platforms, the digest of the multi-platform index is what you pin: TfP preserves the digest you supplied, resolves the linux/amd64 child, records that child as the runtime digest, and provenance names exactly what executed.

The image must be publicly pullable. TfP may identify itself to the registry with its own read-only credential to obtain reliable access to public images; author registry credentials are neither required nor accepted, and a private image is rejected with "public images only".

Building the image without a Docker daemon (Build Cloud)

Cloud agents (and any machine without Docker running) build and push through Docker Build Cloud, which needs only the docker CLI and Buildx ≥ 0.37. The CLI wraps the whole flow:

export DOCKER_ACCESS_TOKEN=…   # a Docker Hub access token; never in a file
export DOCKER_ACCOUNT=tokensforprogress CLOUD_BUILDER_NAME=tfp
tfp verifier build <checker-dir>            # inside the package: pins the digest into verifier-contract.json
tfp verifier build <checker-dir> --json     # anywhere: prints { image_ref, digest, … }

tfp verifier build needs a Dockerfile in <checker-dir>; picks a builder (an existing cloud-driver builder from docker buildx ls, else one created from the three variables above or --builder <account>/<name>, else the local daemon); builds for linux/amd64, pushes to tokensforprogress/tfp-verifiers (--repository, DOCKER_REPOSITORY) under a throwaway tag (--tag); reads the digest from Buildx's metadata file; confirms the image is anonymously pullable from Docker Hub; and, when a verifier-contract.json sits beside the checker directory (or --contract <path>), writes docker.io/<repository>@sha256:… into the sandbox stage's implementation (--stage <key> if there are several; --limits memory_mb=6144,vcpu=1,timeout_seconds=300 writes limits too). --dry-run prints the commands instead of running them.

What it runs, from a sandbox with no daemon:

docker buildx version                      # needs 0.37+ with the `cloud` driver
printf '%s' "$DOCKER_ACCESS_TOKEN" | docker login --username tokensforprogress --password-stdin
docker buildx create --driver cloud --use tokensforprogress/tfp || docker buildx use cloud-tokensforprogress-tfp
docker buildx build --platform linux/amd64 --push \
  --tag tokensforprogress/tfp-verifiers:<campaign>-<checker>-$(date +%Y%m%d%H%M%S) \
  --metadata-file meta.json <build-context>
python3 -c 'import json; print(json.load(open("meta.json"))["containerimage.digest"])'

Pin docker.io/tokensforprogress/tfp-verifiers@sha256:<digest> in the Contract; the tag is a throwaway build name. The credential lives in your environment, never in a Contract, Dockerfile, committed file or log (the CLI passes it to docker login over stdin and never prints it). Do not stand up a registry, a CI pipeline or hand-assembled OCI manifests to work around a missing daemon.

The TfP verifier base image

The recommended path in the beta is to build on one blessed base image:

tokensforprogress/verifier-python:<semver>

published on Docker Hub as a multi-arch image (linux/amd64 and linux/arm64). Production runs linux/amd64; tfp test runs your image under linux/amd64 as well, even on Apple Silicon, so a platform mismatch surfaces locally rather than on the server. Python is the only language the base supports for now; the layout is structured so that later bases (verifier-base, verifier-node) reuse the same harness unchanged.

Your image is FROM the base, copies your checker in, and sets CMD to the command that runs it. The base fixes ENTRYPOINT to /tfp/harness; your CMD is the checker command the harness spawns:

FROM tokensforprogress/verifier-python:0.1.0
COPY checker.py /checker/checker.py
CMD ["python", "/checker/checker.py"]

Build and push it, pin the digest, and you are done with packaging. The current base is tokensforprogress/verifier-python:0.1.0, index digest sha256:a8752ba652a6650bc9f6b96063a5f4428858ea0e145e65b2d0a9c58ccb182cdd (see Status for the per-platform digests). Anything the checker needs (pip packages, data files) is installed in the image at build time; there is no network at run time.

What the checker does

The checker is an ordinary program. It only:

• reads /input/manifest.json, /input/artifacts/<role>/… and /input/reference/<key>/…;
• writes /output/result.json, and optionally /output/evidence/…;
• may write to stdout and stderr (captured as the run log).

It does not listen on a port, parse arguments, talk to a registry or know anything about TfP beyond those paths.

What the harness does

/tfp/harness is the base image's entrypoint and owns everything between the trusted runner and your program:

• serves GET /healthz and POST /run to the runner;
• materializes the run's inputs into /input on a clean per-run filesystem, with a read-only input boundary where the runtime allows it;
• spawns your CMD as an unprivileged child process;
• supervises it: wall-clock timeout, process termination, bounded capture of stdout and stderr;
• collects /output/result.json and /output/evidence/… within the size ceilings;
• returns the SandboxOutcome envelope to the runner under a versioned protocol id, tfp.sandbox-harness/v0.1.

The runner Worker stays authoritative for interpreting result.json: the harness reports what happened (exit code, timeout, log, result bytes, evidence), and the Worker decides what it means for the stage.

What the base image is not

The base image is not the security boundary. Isolation is the Cloudflare runtime policy: enableInternet=false, no TfP secrets in or reachable from the author's image, resource ceilings, and per-run isolation. The harness makes the checker easy to write and its results easy to interpret; it does not make a hostile image safe, and the runner does not rely on it for that.

The run protocol

Whether you use the base image or build your own image that speaks the protocol, this is what a run looks like from inside the container. This section is the author-facing contract; the harness that implements it is not something a checker needs to read.

Inputs (read-only)

/input/manifest.json              what this run is; read it for every path
/input/artifacts/<role>/<file>    the Submission's Artifacts, one directory per role
/input/reference/<key>/<file>     the stage's selected reference inputs, one directory per key

manifest.json:

{
  "version": "tfp.verification-manifest/v0.1",
  "run_id": "vrn_…",                       // "cfx_…" during a conformance (fixture) run
  "verifier_contract_id": "vct_…",
  "stage_key": "structure_check",
  "problem_revision_id": "prv_…",
  "submission": { "id": "sub_…", "summary_md": "…" | null, "summary_included": true },
  "artifacts": [
    { "role": "document", "artifact_id": "art_…" | null, "path": "/input/artifacts/document/translation.json",
      "filename": "translation.json", "content_hash": "sha256:…", "size_bytes": 114, "media_type": "application/json" }
  ],
  "reference": [
    { "key": "document_schema", "path": "/input/reference/document_schema/document_schema.json",
      "filename": "document_schema.json", "content_hash": "sha256:…", "size_bytes": 420, "media_type": "application/json" }
  ],
  "prior_stage_results": [ { "stage_key": "…", "kind": "sandbox" | "review" | "external", "result": "verified" | "refuted" | "inconclusive" | "error" } ],
  "result_kind": "pass_fail" | "score",
  "limits": { "vcpu": 1, "memory_mb": 512, "disk_mb": 256, "timeout_seconds": 120, "network": "none" }
}

• Find files through artifacts[].path and reference[].path, never by guessing names: participants choose their own filenames.
• artifacts[] holds exactly the roles the Contract's submission_requirements.required_artifacts names, bound by the solver; reference[] holds only bound slots selected by this stage's reference_input_slots. Both may be empty for a text-only submission.
• Every sandbox stage in a Contract with sealed slots must explicitly select its reference inputs. Use a public-only pass/fail stage to validate prediction format, then a score stage selecting the sealed labels. A public stage cannot read labels merely because they are bound elsewhere in the revision.
• result_kind is this stage's effective kind: the stage override, or the Contract default. A scored Contract may include pass/fail format validation.
• submission.summary_md is the solver's text; summary_included says whether it counts as the submission (a text-only Contract).
• Everything under /input is read-only; write scratch files under /output or /tmp.

Output

Exit 0 and write /output/result.json:

{ "result": "verified" | "refuted" | "inconclusive", "reason": "one sentence, ≤4000 chars", "score": 0.83 }

• result is required. verified: the submission does what the stage checks. refuted: it does not. inconclusive: the checker could not decide (unreadable input, an unsupported case); this is not a finding against the submission.
• reason is optional and becomes the run's evidence text; write it.
• score is accepted only on verified with manifest result_kind: score, and must be a finite number; anywhere else it makes the run error.
• findings (an array, ≤1000 items) is accepted for structured detail. No other keys are allowed: an unknown key (such as summary_text) makes the run error.
• Optional evidence: any files under /output/evidence/, preserved and hashed with the run. If the stage declares an evidence_schema, you must write /output/evidence/summary.json satisfying it.
• stdout and stderr are captured as the run log (first 256 KB). Public-stage logs are readable immediately. Logs and evidence from stages selecting sealed inputs remain withheld until label release, including in manager output views. Do not print credentials or other unrelated secrets. The log is a Resource, tfp://verifications/<id>/stages/<key>/log (the stage result's log.resource_uri), and a web page at the same path under /verifications/, so an agent reads it without a browser.
• A reference input is mounted under its original filename (/input/reference/<key>/<file>, the reference/<file> name in the package), in production exactly as in tfp test and the conformance fixtures. A checker may key on the extension (.gz, .mps).

What TfP makes of it

Resources: what a checker asks for, what runs it

A checker's execution envelope belongs to its Implementation, never to the Contract: growing a checker's memory never revises a Contract or a Problem. Ask on the stage, beside implementation:

"implementation": "docker.io/<user>/<repo>@sha256:…",
"limits": { "memory_mb": 6144, "vcpu": 1, "disk_mb": 8000, "timeout_seconds": 300 }

Anything left out takes the default, and lint strips limits out of the Contract document (the Contract's hash does not change). Defaults: 0.25 vCPU, 1024 MB memory, 2048 MB scratch disk, 120 s wall clock, no network. Ceilings: 4 vCPU, 12288 MB, 20000 MB, 600 s; a request above them is a lint error, not a run-time surprise.

TfP maps the requested envelope onto the smallest runner shape that satisfies it. Shapes are Cloudflare Containers instance types; the same image may run under several Implementations at different shapes, and a larger runner serves any smaller ask:

A first run on a fresh shape includes pulling the image (a 3 GB Lean image takes about a minute); after that a warm slot answers at once.

When the container dies

TfP never guesses: a lost container with no exit code is reported as lost, not as out of memory.

Checker test cases (conformance fixtures)

Before TfP lets a checker judge anyone's work, it asks the checker questions it already knows the answers to. A Contract ships a few inputs with known outcomes — a solution everyone agrees is valid, one that is wrong in a specific way, one that is unreadable — and states what a correct checker must say about each (verified, refuted, inconclusive). The difference from a real run is exactly that: in a real run the answer is unknown and the checker's word is what counts, so it first has to earn that on cases whose answer is not in doubt.

The formal term is the Contract's conformance fixtures (verifier reference). Every sandbox stage carries at least one. TfP runs every fixture for your stage through your image and compares; one disagreement, or any error, and the checker does not conform.

Local test versus production

tfp test <dir> runs the same fixtures on your machine. What is shared with production differs by checker kind, and it is worth being exact:

• Builtins. tfp test and production execute the same shared checker code, so a fixture that passes locally passes on TfP for the same reason.
• Custom images. Locally, tfp test uses Docker (or a compatible local runtime) to run the exact pinned image under linux/amd64. In production the image runs on Cloudflare Containers. These are different runtimes; what is shared is the exact image digest, the TfP run protocol (/tfp/harness, the manifest and input layout, the result schema) and the result interpretation rules. A fixture that passes locally is strong evidence, not proof; the server's conformance run is what counts, and it is reported by tfp verifier status.

Registration and lifecycle

How TfP reads the registry. Registration resolves the reference through the OCI distribution API once, ever: a pull token, then the manifest by digest, then the linux/amd64 child. Docker Hub rate-limits pull-token requests per source address, so TfP identifies itself to Docker Hub with its own read-only credential (the operator secrets DOCKERHUB_USERNAME and DOCKERHUB_PULL_TOKEN on the Worker; Cloudflare's container runtime is configured with a second read-only token through wrangler containers registries configure docker.io). That credential only raises TfP's own allowance: an author's image must still be publicly pullable, the credential never reaches an author, a Contract, a log or a checker, and no private image becomes visible through it. A registry that answers 429 or 5xx while TfP resolves an image is reported as registry_unavailable (with the registry's Retry-After when it gave one) after a few bounded retries; that is a statement about the registry, never about the image, and validating the same upload again is the whole remedy.

Units. A custom sandbox VerifierImplementation names exactly one author-supplied, digest-pinned image reference. That reference resolves to one linux/amd64 runtime digest (the same digest for a single-platform image; the matching child for a multi-platform index). Three units, kept distinct: the semantic unit is the VerifierImplementation; the author's provenance identity is the pinned reference and its digest, recorded exactly as written; the physical provisioning unit is the unique runtime digest. Many Implementations, across many Contracts and authors, and even two different indexes that resolve to the same linux/amd64 bytes, share one runner. TfP provisions one runner per unique runtime digest, never a duplicate, and never loses which reference an author actually pinned.

Registration is automatic. There is no register command. When validate_authoring_upload (tfp upload) validates a package whose stage names a custom image, the image is registered under the author's custom_verifier capability if it is not already known, and its provisioning is enqueued. A digest someone else already registered is reused. The reference must be digest-pinned; a multi-platform index is resolved to its linux/amd64 child and both digests are recorded.

Status. One read tool, get_sandbox_image_status { image }, and one command over it:

tfp verifier status <ref-or-digest>

report where an image stands. This is the whole author-facing surface for images beyond the package itself; it was chosen over an explicit register/status pair because validation already knows every image the package names and an author should never have to tell TfP twice.

Lifecycle.

registered ──▶ provisioning ──▶ available ──▶ conformance pending ──▶ passed
                    │                                   │
                    └──▶ provisioning failed            └──▶ failed

• Registered. The digest is recorded with the capability it was registered under and the Implementations that name it.
• Provisioning. TfP is making the image executable on its container runner (details in Runner architecture on Cloudflare, below).
• Available. The runtime digest is executable. Conformance runs are enqueued for every Implementation naming the digest.
• Provisioning failed. The image could not be made executable (not pullable, not linux/amd64, over the size limit, refused by the runner). This is not a conformance failure: a checker Cloudflare could not execute has not failed anything, and no Implementation is marked failed. state_reason says why; a new digest that provisions cleanly picks up from there.
• Conformance pending. The image is available; the Implementation's fixtures have not yet been run (or are running).
• Conformance passed. Every fixture reproduced its expected result. The Implementation becomes the stage's current one (the highest passed, non-yanked version) and from now on judges real submissions.
• Conformance failed. The record lists each fixture that disagreed, what was expected and what happened. It stays readable forever; it is never current. Fix the image, push a new digest, upload again — that is a new Implementation, and the failed one remains in the stage's history.

A failed Implementation can be re-run by an operator (for instance after a fixture was corrected); the verdicts of earlier versions are never rewritten. A package naming a custom image can materialize as ordinary work regardless of image state; a run reaching that stage waits for the image while it is provisioning or pending, and ends error if provisioning fails: an image TfP could not start is TfP's machinery failing, never a finding about the submission (a checker that runs out of time or memory is inconclusive, per the runtime matrix).

Runner architecture on Cloudflare

This section is the frozen design for how a registered image becomes executable. It is written against Cloudflare's Containers platform as documented on 2026-09-03; the facts that constrain it are cited so the design can be re-checked when the platform changes.

What the platform fixes. A container application has exactly one image, chosen when the application is deployed: the image is a property of the Durable Object class the application is attached to, and Worker code cannot pick a different image per request (image management, architecture). Instances are addressed by name and started on demand by trusted Worker code (getContainer(binding, name) then startAndWaitForPorts()), stop after sleepAfter idle seconds, and run with enableInternet = false when told to (container package). Bindings between Workers are fixed at deploy time; a Worker may call another Worker's URL with a shared secret. Public Docker Hub images can be named directly in an application's configuration; Cloudflare does not cache them. Containers require the Workers Paid plan (pricing).

Provisioning unit. One runner application per unique linux/amd64 runtime digest. A VerifierImplementation is the semantic unit; the author's pinned reference is the provenance record; the runtime digest is the physical one. Two Implementations, or two pinned references, that resolve to the same runtime digest share one runner; registering a reference whose runtime digest already has a runner provisions nothing.

The runner. A small Worker, named for the digest (tfp-runner-<first 12 hex of the digest>), exporting one container class with sleepAfter of about one minute, enableInternet = false, max_instances set to TfP's per-runner concurrency ceiling, and a fetch handler that requires a shared secret header, names the instance after the VerificationRun id, starts it, and forwards POST /run to the harness on its port. The runner holds only the runner secret. The author's image receives no environment, no credentials and no network.

The provisioner. A separate process from the request-serving Worker, because it needs an account-management credential (Workers Scripts Edit plus Containers Edit) that must not sit in the main Worker. It is driven by the provisioning queue: registering a new digest enqueues one job; the provisioner uploads the runner script with its container class, durable-object binding, migration and secret (Workers script upload API), reads the resulting durable-object namespace id, and creates the container application naming the author's public image by digest (the Containers applications API that wrangler deploy itself uses; it is not in the public API reference, so the provisioner treats it as unstable and records exact responses). It records the runner name and URL on the image and marks it available, or marks it failed with the platform's error. The main Worker is never redeployed. Until the provisioner is deployed on a paid account, the same steps are an operator script.

Data plane. A VerificationRun reaching a sandbox stage whose Implementation names a custom digest looks up the digest's runner, posts the job to the runner's URL with the shared secret, and the runner starts a uniquely named instance, forwards the job to /tfp/harness, collects the envelope, and destroys the instance before accepting a verdict. Retries never reuse its identity. The idle timer is a crash backstop, not the isolation boundary. Per-runner and global concurrency leases bound dispatch, together with max_instances. There is no autoscaler and no warm pool. A runner that cannot be reached, a harness that faults, or an instance that never comes up is an error for that run, never a finding about the submission.

Local mirror. tfp test runs the same image under Docker on linux/amd64 with the same harness protocol; the digest, manifest layout, fixtures, result schema and interpretation rules are shared, the execution environment is not.

Status

The whole path is live on tokensforprogress.org: validation resolves and registers a digest-pinned image, the provisioner (its own Worker with its own scoped credential) creates one runner per linux/amd64 runtime digest, and the tfp-runner data plane executes runs on it. Verified on 2026-09-04 on the Workers Paid plan with a public Docker Hub image pinned by digest: registered → provisioning → available in about five seconds, the runner answering /healthz with the protocol and refusing /run without the shared secret. An image that does not speak the harness protocol (that test image did not) makes every run on it an error with the reason recorded, never a verdict, which is the designed failure. An image that cannot be provisioned at all is provisioning_failed with the platform's answer verbatim, and an operator can re-enqueue it or record a hand-provisioned runner through /admin/sandbox-runners.

The base image tokensforprogress/verifier-python:0.1.0 is on Docker Hub as a public multi-platform image (published 2026-09-04, anonymous pull verified with an empty Docker config, FROM build verified from an empty directory with no repository access):

The next step is the first end-to-end custom conformance run by an external author building FROM it.