Script Checkers

A sandbox stage can name one Python file beside the Contract:

{ "key": "structure_check", "kind": "sandbox", "title": "Structural check",
  "description_md": "Parses and has the required keys.",
  "implementation": "./checker.py" }

The author builds nothing: no Dockerfile, registry account, or image push. TfP records the exact program bytes and runs them on its pinned Python runtime. Scripts require author; custom container images remain supported, with custom_verifier, for dependencies, other languages, or tools that the standard library does not provide.

Writing a checker

Python 3.12, standard library only, one nonempty UTF-8 file up to 256 KiB. Paths are relative to the Contract: ./checker.py or ./checkers/structure.py (letters, digits, _ and -). Shared Campaign Contracts can name a shared script. TfP mounts it at /input/checker/checker.py, regardless of its original filename.

The run protocol is the same for both forms: read /input/manifest.json, find artifacts and reference inputs by the manifest's paths, and write /output/result.json. Do not guess input filenames. For example, a check that establishes only required-key presence:

import json, os
m = json.load(open('/input/manifest.json'))
a = next(a for a in m['artifacts'] if a['role'] == 'document')
r = next(r for r in m['reference'] if r['key'] == 'document_schema')
schema = json.load(open(r['path']))
try:
    doc = json.load(open(a['path']))
except (ValueError, UnicodeError):
    result = {'result': 'refuted', 'reason': 'document must be JSON'}
else:
    valid = isinstance(doc, dict) and all(k in doc for k in schema.get('required', []))
    result = {'result': 'verified' if valid else 'refuted',
              'reason': 'required-key check only; no semantic judgment'}
os.makedirs('/output/evidence', exist_ok=True)
json.dump({'valid': result['result'] == 'verified'}, open('/output/evidence/summary.json', 'w'))
json.dump(result, open('/output/result.json', 'w'))

Exit 0 after writing verified, refuted or inconclusive. error is reserved for TfP's machinery. Supply evidence matching the stage's evidence_schema. Defaults are 0.25 vCPU, 1 GB memory, 2 GB scratch, and 120 seconds; stage limits can request a supported larger runner. Both paths have bounded resources and concurrency. Neither receives TfP secrets or Internet access. Python code is arbitrary code, not a restricted language.

What the verdict means

Verified means the required checks passed. Immutable code makes the run reproducible; it does not prove the checker implements the intended criteria, or that its criteria establish scientific correctness. Public positive, negative and boundary fixtures provide regression evidence, not a proof of semantic equivalence. Use independent review or other stages where needed.

Changing source bytes always creates a new Implementation. It may remain under the same Contract only if normative acceptance semantics are unchanged, for example a bug fix restoring the documented rule. A changed threshold, new required property, different metric, or different definition of success requires a new Contract and ProblemRevision. State criteria in the Contract, not only in the program. Authors remain accountable for this distinction; TfP cannot infer arbitrary Python's meaning from its fixtures.

A replacement becomes current only after its fixtures pass. Previous implementations and historical run records stay available. A runtime upgrade also creates a new Implementation; existing runtime versions never change.

Runtime identity and isolation

tfp-runtime/python:1 is a friendly name for the immutable, published tokensforprogress/verifier-python image pinned by OCI digest in TfP's runtime catalogue. Its Implementation's runtime.image_digest is that actual image digest, not a hash of the name. The program hash is part of content_hash. Each VerificationRun records the resolved linux/amd64 image digest, implementation identity/hash, inputs, and results.

Local Docker and the server use the same pinned image and fixed command: /tfp/harness python /input/checker/checker.py. The trusted runner supplies that entrypoint for scripts; custom images retain their own entrypoint. There is no separate wrapper image or mutable-tag build step.

Every invocation gets a unique container identity, including retries and conformance fixtures. The runner collects the response and destroys the container before accepting a verdict. Files, /tmp, and child processes never carry into another invocation. The application only uses the versioned isolated endpoint and rejects responses without its isolation marker. Older runners cannot silently execute a new job. Cold starts are a deliberate cost of this boundary; instances that have run author code are never reused.

Testing and publication

tfp lint <dir> checks the package offline. tfp test <dir> runs fixtures under Docker using the pinned image; without Docker it reports that those checks were not run and leaves conformance to the server. tfp verify fetches the published program, checks its hash, and runs the same image locally. Local preflight is useful feedback; server verification is authoritative.

An operator enables the catalogue's exact runtime image on each deployment. Binding cannot choose different bytes for the same runtime name. tfp verifier status tfp-runtime/python:1 reports readiness. If it is unbound, the upload remains unpublished; keep the package and validate it again once the operator enables the runtime. A bound runtime provisions the shape requested by validation. Conformance then runs on publication, and automatic spare-compute dispatch waits until every sandbox stage has a non-yanked, conformance-passed implementation. A newer pending replacement does not disable an older conforming implementation.

The exact program is served at GET /api/verifier-implementations/{id}/program, listed in revision files under checkers/<stage>/<filename>, and linked from checker inspection. The hash in the implementation lets readers verify the bytes independently.

Rollout status

This change requires the revised runner Workers for both script and custom image verification. Updating the provisioner alone does not update existing runners. Operator rollout and a real container run on the deployment must be recorded separately from local tests. See the builder-facing status ledger and developer workflow for rollout evidence; no production readiness is inferred from tests or the existence of this page.