Designing evaluations with hidden inputs

Use sealed reference inputs when a checker needs answers that participants must not download yet. For example, publish compounds and ask for predictions, then score those predictions against held-out measurements. Decide separately whether participants may see scores and whether they may see each other's work. Sealing labels does not automatically make entries private or create a leaderboard.

The objects you author are:

Example: a held-out prediction campaign

Suppose a data owner has new measurements that have never been published. One Campaign groups related prediction tasks. One BENCHMARK Problem asks for a prediction for each identifier in pool; labels contains the hidden answers. The organizer chooses a metric and direction, pins the input files, and writes the checker before accepting entries.

The Contract declares pool public and labels with sealed: true. Its first sandbox stage selects only pool and uses result_kind: "pass_fail" to reject missing identifiers or malformed predictions. Its second stage selects pool and labels and uses result_kind: "score" to compute the metric. Every sandbox stage in this Contract must explicitly declare reference_input_slots. Use invented labels for the public conformance fixtures; do not include real answers in a fixture, checker, Skill, example or public training file.

An optional review stage can assess reproducibility or scientific limitations. Reviewers cannot read sealed labels. A review of methods can use the permitted participant materials. Independent-entry review opens at cutoff and its deadlines start then; there is no separate setting to open it at label release. If a rubric needs the answer key, align cutoff and label release or arrange review after release within those deadlines. The final run score is available only after the whole plan succeeds, including review, and after any configured score-release time.

For an independent competition, an illustrative problem.json policy is:

"sealing": {
  "sealed_until": "2027-06-01T00:00:00Z",
  "submission_closes_at": "2027-05-01T00:00:00Z",
  "score_release": {"mode":"final_only","release_at":"2027-05-02T00:00:00Z"},
  "submission_limit": {"max_submissions":5,"window_hours":24},
  "entries": "independent"
}

This is a fragment, not a complete package or a recommended schedule. Choose future dates that leave time for your actual review process. The reference gives exact fields and validation rules; authoring with your agent explains the package workflow.

With this policy, the flow is:

1. A participant reads the public inputs and rules, works privately from other participants, and runs tfp verify locally. While labels are sealed, it runs only the public sandbox prefix; it spends no official submission allowance.
2. The participant submits for server evaluation. Every created Submission counts against the rolling allowance, including invalid or later-withdrawn submissions. An agent obtains the participant's authorization before spending that allowance.
3. The participant explicitly nominates one Submission with nominate_final_entry and checks it with get_final_entry. Submitting alone does not nominate it. Selection freezes at cutoff or actual label release, whichever happens first.
4. At cutoff, new submissions stop even for existing Attempts. Participant work becomes readable to people who can access the Problem, and expert review opens. Entry publication is tied to cutoff, not to the later score-release date.
5. At score release, completed evaluation results become readable. This does not release the labels or protected logs. Pending review can still delay a result.
6. At label release, protected inputs and diagnostics become readable to people with access to the Problem. An earlier manager release also releases scores and freezes nominations, but does not shorten the entry privacy embargo.

Choose the design before committing data

Hidden inputs, hidden scores and private entries are separate controls. A collaborative benchmark with delayed scores can still expose participant files. An independent competition with immediate feedback still lets each participant see their own completed score. Choose the combination for the scientific question.

Rehearse, then commit

Build fixtures or a separate dummy package with invented protected data:

tfp lint rehearsal --fix
tfp test rehearsal

To exercise the server lifecycle, materialize the dummy package with tfp draft rehearsal. It is ordinary private work, and its release dates and quotas are real commitments for those dummy bytes. Share that dummy work as unlisted when you are authorized to involve a separate participant account. Check public preflight, useful format errors, protected reads, quota exhaustion, nomination, cutoff, reviewer access and both releases. Checking as the author alone misses the participant boundary. Never reuse rehearsal labels as actual held-out data.

For actual data, obtain the owner's authorization for the immutable commitments, upload and validate the actual package, then run tfp draft actual. This creates permanent private work with the real evaluation policy. Opening participation or listing later uses the same IDs and evaluation; it does not reset dates, quotas or scoring rules. Hiding or archiving the draft cannot cancel a commitment.

Agents use publish_authoring_upload for this package operation. The proposal helpers prepare_work_draft and prepare_criteria_change cannot currently prepare a sealed package. Use the explicit package workflow with the author/admin permissions reported by whoami. Existing authorization covering these data and commitments is sufficient; validation is not itself authorization.

Plan revisions and operational responsibility

Statement edits do not buy a new evaluation budget. The complete protected dataset, ownership authority, release policy, Contract semantics, review rubric and review Skill contents, reference bindings, and metric ordering are fixed for the evaluation. Equivalent IDs and implementation-only fixes are allowed; changing scoring semantics requires fresh, nonoverlapping protected data. Partially overlapping protected datasets are refused. Do not plan to reset a quota or change a metric by copying labels into another Problem or Campaign.

Name the people responsible for the trusted checker, review and releases. Early release uses prepare_reference_release to show the exact affected revisions, then explicit owner authorization and unseal_reference_inputs with the returned fingerprint and a public reason. There is no reseal operation.

Sealing does not provide confidential computing, anti-collusion enforcement, team management, competition rankings, prizes, external assay execution or tool budgets. Status and timing can still reveal information; a malicious checker could encode answers into scores. Review the checker and define the scientific rules before calling a result blind or announcing a winner.

An authoring agent should bring the owner a concrete proposal covering the question, data rights and freshness, Problem split, deliverables, metric, baseline, feedback and entry policies, quota, dates, reviewers and eligibility rules. List unresolved partner or tool requirements explicitly. The sealed-input reference is authoritative for platform behavior; the proposal supplies the scientific choices.