TfP Docs

Sealed reference inputs

Status: approved for implementation, October 14, 2026. The owner approved the contract below, including the audit remedies approved later that day. Deployment and end-to-end acceptance are recorded in the project status ledger; the specification alone is not deployment evidence.

A sealed reference input holds evaluation data that participants cannot download until its announced release. Problem managers can inspect it; selected sandbox stages can evaluate submissions against it. Public inputs, format validation and held-out evaluation remain distinct. Collaborative evaluations retain normal Problem visibility. Independent evaluations keep participant work private until the submission deadline and require a nominated final entry.

For campaign design and a participant-to-release walkthrough, begin with Designing evaluations with hidden inputs. This reference specifies the exact behavior and constraints.

Contract slots and stage access

Declare sealed: true on a Contract reference slot. Omission means false. The Contract must contain a sandbox stage that selects each sealed slot. For a Contract with any sealed slot, every sandbox stage explicitly declares reference_input_slots. An empty array selects no reference inputs. Unknown or repeated keys are invalid. Review and external stages cannot declare this field or receive sealed inputs.

{
  "reference_inputs": [
    {"key":"pool","required":true,"sealed":false,"media_types":["text/csv"],"max_bytes":1048576,"description":"Public compounds and identifiers."},
    {"key":"labels","required":true,"sealed":true,"media_types":["text/csv"],"max_bytes":1048576,"description":"Held-out labels."}
  ],
  "stages": [
    {"key":"format","kind":"sandbox","title":"Prediction format","description_md":"Checks identifiers and finite predictions.","reference_input_slots":["pool"],"result_kind":"pass_fail"},
    {"key":"evaluate","kind":"sandbox","title":"Held-out score","description_md":"Computes the declared metric.","reference_input_slots":["pool","labels"],"result_kind":"score"}
  ]
}

This fragment omits the Contract's other required fields. A sandbox receives only its selected reference entries and bytes. Public validation never receives labels merely because another stage needs them. In older Contracts without sealed slots, omitted reference_input_slots preserves the existing all-slots behavior. Explicit selection is also supported on unsealed Contracts.

A sandbox stage may declare result_kind: "pass_fail" or "score"; omission inherits the Contract's result kind. A pass/fail Contract cannot contain a score stage. A score Contract must contain at least one score stage. Only score stages produce numeric scores, and the last successful score stage supplies the final run score. Review stages do not produce or expose stage scores. The public run score remains null until the entire verification plan succeeds, including any review. Deferring score publication may additionally withhold that final score.

Conformance fixtures bind only reference slots selected by their stage, including all selected required slots, and use public stand-in data. They must never reuse the revision's sealed bytes, including under another filename or fixture role. Fixture scores follow the stage's result kind.

Revision policy

A revision binding a sealed slot has a sealing policy; otherwise it is null. Authors put the following block in problem.json:

"sealing": {
  "sealed_until": "2027-06-01T00:00:00Z",
  "submission_closes_at": "2027-05-01T00:00:00Z",
  "score_release": {"mode":"final_only","release_at":"2027-05-02T00:00:00Z"},
  "submission_limit": {"max_submissions":5,"window_hours":24},
  "entries": "independent"
}

The public revision additionally supplies derived unsealed_at: null while sealed, then the scheduled date or recorded earlier release. Derived release_event is null unless a manager explicitly unseals early; then it records {unsealed_at,user_id,reason_md} so readers can inspect who released it and why. A scheduled release alone creates no manager event. Authors must not supply these fields. The derived evaluation_id identifies the stable evaluation shared by revisions using the same protected data and release authority. All sealed slots on a revision share this policy. Statement edits retain the evaluation and its remaining budget. An existing protected dataset cannot acquire fresh quotas or a different release policy through a new revision or Problem. Changing the evaluation data requires fresh, nonoverlapping protected inputs. Scores on different evaluation sets are not comparable merely because their metric names match.

Revisions may share sealed bytes only when their complete protected hash set, Problem/Campaign ownership authority, normalized policy and scoring rules match. Partial overlap is refused: binding X beside Y must not grant X's owner authority to release Y. Ownership and Campaign moves cannot silently change a sealed evaluation's release authority. Ambiguous legacy groups are quarantined from early release and new aliases; their existing scheduled promises are preserved.

The evaluation also freezes the Contract document, review protocol/rubric and review Skill contents, every reference slot's bytes and mounting metadata, and the benchmark metric, direction and unit. Equivalent Contract, protocol and Skill IDs created by materialization or isolated revision validation share the evaluation when their contents match. New implementations under those same Contract semantics remain allowed; changed metrics, rubrics, slot mappings or public scoring references require fresh, nonoverlapping protected data. Statement/context edits and benchmark baseline updates retain the evaluation and its remaining budget. These checks run atomically in the database, including for direct revision inserts.

For early release, a manager first calls prepare_reference_release with the revision. The response lists every affected revision and a confirmation fingerprint. After explicit user authorization for that affected set, the agent calls unseal_reference_inputs with the revision, public reason and fingerprint. Publication or authority changes invalidate the confirmation. Only exact aliases of that evaluation release; there is no transitive traversal into other datasets. The audit record is append-only. There is no website early-release button.

Private drafts and real-data commitments

Materializing an ordinary sealed package with tfp draft or publish_authoring_upload commits its real immutable release date, authority, quota and scoring rules, even while private. Obtain authorization covering those commitments first. Sharing and listing preserve the same evaluation; hiding, archiving or revising the work cannot postpone or undo its release promise.

Rehearse with invented protected data in local fixtures or a separate dummy package. A dummy package materialized on the platform is also ordinary work with its own commitments. Never substitute actual data into a rehearsal or reuse its labels as real held-out data. Proposal preparation helpers refuse sealed packages; use the explicit package workflow.

Internal isolated candidates for revision validation and historical preview records retain their sealed-data safeguards. Synthetic rehearsal hashes cannot later become real holdouts or overlap committed real data. Recorded real-data commitments survive expiry and revocation, and their early release must be prepared and confirmed through an ordinary revision. Historical access grants never expose protected bytes. These internal and historical records are not a public authoring workflow.

An administrator maintaining another owner's package does not acquire that evaluation's release authority. An authorized replacement must preserve the existing owners, evaluation and policy; different authority cannot silently alias the protected dataset.

After release, simple editing can still change the title and summary. Changing the statement requires an authoring package with a replacement public Contract (sealed: false on the released slots) and no sealing policy. Simple editing does not copy a released evaluation's old sealing policy into a new revision.

What participants can read

Problem managers are the Problem owner, Campaign owner and live platform administrators. A trusted author manages only their own work. These permissions never make a private Problem public.

While sealed, managers and selecting sandbox stages can read protected bytes; solvers and reviewers cannot. Everyone with access to the Problem can inspect the slot description, filename, size and content hash. Problem and Attempt envelopes retain sealed metadata, download links are absent, and Resource reads and direct byte endpoints refuse unauthorized access. Sharing work as unlisted does not grant participants protected-data access; the author retains manager access.

Only stages selecting sealed slots have protected logs and evidence_md. Their text and evidence files are stored intact and withheld until unsealing. Public validation diagnostics remain readable. Run evidence_md, which combines stages, is withheld if it would disclose a protected stage. The same projection applies to website, API, MCP, submission and continuation views, review materials, and hosted agent context. Labels and withheld output become readable at release.

With immediate feedback, run/stage status and result, execution metadata and the final run score remain visible, subject to the rule that an unfinished run has no final score. With final_only, protected evaluation verdicts and the final score are withheld until score release; operational status and public validation results remain visible. Stage progression, timing, operational metadata and the appearance of a later review may still permit inference about execution. This policy withholds direct evaluation feedback; it is not a guarantee of zero information leakage. The independent-entry embargo is a separate rule. These output projections apply to every participant view, including managers; managers’ authenticated access to reference bytes is separate. A failed format check therefore gives useful feedback without revealing held-out performance. A reviewer sees permitted results and materials, never an invented sandbox-stage score or withheld labels. Score release does not release protected logs or labels.

tfp verify checks packaging and runs the public sandbox stages that precede the first protected stage. It never requests sealed bytes and reports that stage and following stages as not run locally. After release, those stages can run normally. Local preflight creates no Submission, consumes no quota and is not an authoritative verification. tfp test uses only public conformance fixtures.

Eligibility and publication

For independent evaluations, each participant nominates one of their immutable Submissions as the final entry. The nomination may be replaced before cutoff and is frozen at the earlier of cutoff or actual label release, including an early release. Repeating the current selection creates no duplicate event. Withdrawal invalidates an entry; it does not reopen the nomination window or refund quota. A missing nomination is not silently replaced with the latest or highest-scoring Submission. Managers, participants with recorded privileged label access, and entries submitted after label release cannot nominate. Recorded privileges acquired before selection froze remain disqualifying after revocation. Privileges acquired only after selection froze do not retroactively invalidate the frozen entry. The platform records nomination history separately from scientific verification and exposes bounded, independently paginated audit history and latest selections; your own current nomination is independent of pagination.

A Submission's time can establish that it was submitted before label release; it cannot establish that its author never knew the labels. sealed_entry records participant_before_release, manager or after_release (null for an ordinary revision); managers remain classified as managers after release. This is factual role and timing context, not a leaderboard eligibility decision. Manager entries must be excluded from comparisons intended to represent participants without privileged access. External knowledge, training-data contamination and later access changes require scientific policy and review; a timestamp does not prove blindness. Eligibility is separate from an immutable verification result. This feature does not implement team standings, statistical rankings, team quotas or a competition leaderboard. Independent competition organizers still define identity, external-knowledge exclusions and scientific eligibility; recorded platform access is not proof of no outside access.

Publication checks the policy, slot selection and bound hashes. Registry guards cover previously public reference inputs, conformance fixtures, Artifacts and checker programs, plus conflicting release promises. Package lint also rejects an exact copy of sealed bytes anywhere outside the package's protected reference bindings, including Skill support files and checker source. Historical Skill files and arbitrary text elsewhere on TfP are not exhaustively indexed by this guard. Authors remain responsible for preventing earlier disclosure. These are exact-byte checks, not semantic detection of embedded or transformed label data. A hash commitment permits auditing after release; predictable small inputs need a random value in the protected file to resist guessing from its public hash.

This is trusted-checker access control, not confidential computing. A malicious checker could encode labels in numeric scores or operational behavior. Checker review, sensible score precision and submission limits remain necessary. Sealing cannot make already-public scientific labels unknown again. Continuous active learning, external experiment adapters and paid tool budgets are separate features.