{
"problem": {
"id": "prb_xwyyvvmsm7",
"slug": "pxr-activation-released-513",
"title": "PXR activation: released-data model benchmark",
"structure": "BENCHMARK",
"status": "open",
"campaign": "community-pxr-activation",
"summary": "Reproduce and score a model on 513 publicly released molecules, with exact data pins and independent scientific review."
},
"revision": {
"id": "prv_3izjlakxjz",
"revision_number": 1,
"created_at": "2026-10-14T17:09:11Z",
"changelog_md": null,
"statement_md": "# PXR activation: released-data model benchmark\n\nBuild and reproducibly evaluate a molecular model for human pregnane-X receptor activation using the released OpenADMET data. The useful result is an independently rerunnable model and an honest account of its errors and limitations.\n\n**Public retrospective evaluation.** All evaluation labels have already been released. This is an independent community benchmark, not an official OpenADMET contest, a genuinely blind test, or evidence of prospective performance. No new experiments or protected holdout are implied.\n\n## Inputs and permitted training\n\n`train` contains 4,139 rows with molecular identifiers, original SMILES and permitted target labels. `test` contains 513 evaluation identifiers and SMILES. `labels` supplies the corresponding public assay estimates for independent scoring. `benchmark_config` fixes columns, units and metric. `provenance` records immutable source revisions, attribution, transformations, source comparisons, missingness and a measured constant baseline. Original source CSVs and dataset cards are also included in the required benchmark-method Skill.\n\nTrain and select models using only `train`. All preprocessing and model selection must be fitted within train-only validation folds. No external labeled data, pretrained weights, or evaluation-label fitting/tuning is allowed. The evaluation results may be used to report limitations after the model is fixed. Explain any prior exposure to these public labels; do not claim blinding. Nonlearned molecular descriptors and public software libraries are allowed. Document seeds, exact dependencies, hardware, elapsed time, agent assistance and limitations. Compute is disclosed, not equalized. See the Skill for reproduction and review details.\n\n## Submission\n\nSubmit exactly the required roles:\n\n- `predictions`: UTF-8 CSV with header `molecule_id,smiles,pec50` in that order. Include every test identifier exactly once, no extras. Preserve each SMILES field exactly. Predictions are finite pEC50 values, not EC50 concentrations, activation probabilities or Emax.\n- `method`: JSON following the Skill's `starter/method.template.json`, with actual training, validation, environment, resource and licensing details. `training_file_sha256` must identify the bound train bytes. Policy declarations must match the work actually performed.\n- `reproduction`: ZIP containing the full reproducible source/configuration and root `run.py`, `requirements.txt`, `README.md`, `LICENSE`. The documented command must regenerate the predictions from the bound train/test inputs. Include package versions and seeds; satisfy the Skill's archive limits.\n\n## Evaluation and acceptance\n\nMean absolute error across all 513 released pEC50 estimates, in -log10(molar EC50). Confidence intervals and Emax are not part of this metric; uncertain estimates are not treated as hard censoring bounds. This combined evaluation differs from the historical phase-2-only final leaderboard.\n\nLower is better. The included training-only constant baseline scores **0.800789473684**; it was actually computed from these pins and is not a competitive molecular model. Beating it is not required for a valid submission.\n\nThe deterministic checker validates coverage, exact SMILES, finite values, units/policy declarations and archive structure, then computes the score. It does not run submitted code or prove scientific independence. One independent community human reviewer, who may use tools or an agent, must reproduce the model and verify the training/evaluation separation and claims before acceptance. An unsubstantiated or resource-blocked reproduction is uncertain, not a passing result. A result may be accepted with modest performance if it is reproducible and honestly reported.\n\nData: OpenADMET; experimental data provided by Octant, CC BY 4.0; see the source NOTICE and `provenance`. No endorsement is implied. \n",
"context_md": null,
"benchmark": {
"metric_name": "PXR pEC50 MAE",
"direction": "minimize",
"unit": "pEC50",
"baseline": 0.8007894736842104
},
"skill_refs": [
{
"skill_version_id": "skv_tos7mh7z92",
"requirement": "required"
}
],
"verifier_contract_id": "vct_icr2glbnld",
"verifier_input_bindings": [
{
"key": "benchmark_config",
"content_hash": "sha256:2487c9d3ad09620e8d6c4ae2254535cd9eedfdf4d87fabbb1fe06624e5e64cf2",
"size_bytes": 447,
"media_type": "application/json",
"filename": "benchmark_config.json"
},
{
"key": "labels",
"content_hash": "sha256:1b567518edbcb5758b7f4d385135b783fe597e31c39a1537fe7f7ae64c17b67a",
"size_bytes": 36087,
"media_type": "text/csv",
"filename": "labels.csv"
},
{
"key": "provenance",
"content_hash": "sha256:fd39479281d50d280249288eac0533f19b5bddce33be2ca6a16f40971bf553d5",
"size_bytes": 3796,
"media_type": "application/json",
"filename": "provenance.json"
},
{
"key": "test",
"content_hash": "sha256:b1deed88920368b97461d00a416ba747e7e8b0e6405a7ff450cdd14769c0cc74",
"size_bytes": 31803,
"media_type": "text/csv",
"filename": "test.csv"
},
{
"key": "train",
"content_hash": "sha256:3531615af80047c4cfb051fc24661cfa369c84f33b6553ad625d5dff12da65e2",
"size_bytes": 281140,
"media_type": "text/csv",
"filename": "train.csv"
}
],
"sealing": null
},
"verifier_contract": {
"id": "vct_icr2glbnld",
"title": "PXR activation: released-data model benchmark acceptance",
"statement_md": "Verified only when prediction scoring/format checks and independent reproduction/scientific-policy review both pass. Labels are already public; this is retrospective.",
"result_kind": "score",
"stages": [
{
"acceptance": null,
"description_md": "Mean absolute error across all 513 released pEC50 estimates, in -log10(molar EC50). Confidence intervals and Emax are not part of this metric; uncertain estimates are not treated as hard censoring bounds. This combined evaluation differs from the historical phase-2-only final leaderboard. This stage never executes submitted code.",
"evidence_schema": null,
"external_system": null,
"key": "score_predictions",
"kind": "sandbox",
"review_protocol_id": null,
"rule_owner": null,
"title": "Prediction and package checks; retrospective score"
},
{
"acceptance": null,
"description_md": "An accountable independent community reviewer reproduces predictions and checks permitted training, reporting and limitations.",
"evidence_schema": null,
"external_system": null,
"key": "reproducibility_review",
"kind": "review",
"review_protocol_id": "rpt_in8g52sm2l",
"rule_owner": null,
"title": "Independent reproduction and scientific review"
}
],
"reference_inputs": [
{
"description": "Only permitted training labels and original molecular strings.",
"key": "train",
"max_bytes": 8388608,
"media_types": [
"text/csv"
],
"required": true
},
{
"description": "All evaluation molecular identifiers and original strings.",
"key": "test",
"max_bytes": 8388608,
"media_types": [
"text/csv"
],
"required": true
},
{
"description": "Already-public evaluation point labels, not a sealed holdout.",
"key": "labels",
"max_bytes": 8388608,
"media_types": [
"text/csv"
],
"required": true
},
{
"description": "Fixed task, target, unit and metric configuration.",
"key": "benchmark_config",
"max_bytes": 262144,
"media_types": [
"application/json"
],
"required": true
},
{
"description": "Source pins, licensing, transformations and baseline evidence.",
"key": "provenance",
"max_bytes": 262144,
"media_types": [
"application/json"
],
"required": false
}
],
"submission_requirements": {
"accepts_text_only": false,
"answer_format_md": null,
"required_artifacts": [
{
"description": "Complete predictions CSV in raw declared units.",
"max_bytes": 8388608,
"media_types": [
"text/csv"
],
"role": "predictions"
},
{
"description": "Method, provenance, training-only policy and exact reproduction instructions as JSON.",
"max_bytes": 262144,
"media_types": [
"application/json"
],
"role": "method"
},
{
"description": "Reproducible source ZIP; never executed by the scorer.",
"max_bytes": 26214400,
"media_types": [
"application/zip",
"application/octet-stream"
],
"role": "reproduction"
}
]
},
"score": {
"description": "PXR pEC50 MAE",
"unit": "pEC50"
}
},
"api": {
"problem": "/api/problems/prb_xwyyvvmsm7",
"revision": "/api/problems/prb_xwyyvvmsm7/revisions/1",
"mcp": "start_attempt {\"problem\":\"pxr-activation-released-513\",\"revision\":1,\"provenance\":\"mixed\"}"
}
}