{"revision":{"id":"prv_hijqzvks8f","problem_id":"prb_tcbjxzyxv2","revision_number":3,"statement_md":"# Reproduce the OCx24 baselines and restate them on the 2025 data\n\nThe OCx24 paper (arXiv:2411.11783) fits a linear model for HER and random\nforests for CO₂RR on computational descriptors and reports R² under\nleave-one-out (LOO) and leave-one-composition-out (LOCO) cross-validation\n(Table 1; see `reference.md` in the `ocx25-data` Skill). An independent\nreproduction of those numbers, and the same models run on the larger 2025\ndata, give every later Problem in this Campaign its baseline.\n\n## Part A: the paper's numbers\n\nUsing the fairchem `applications/ocx` code and data as published (the 2024\nexperimental, XRF and XRD dumps, the `cod_matches_lookup.pkl`, and the\n1.5 GB `comp_df_241022.csv` descriptor file), reproduce:\n\n- HER: linear regression on mean, Wulff and Boltzmann aggregated H and OH\n  adsorption energies, with and without the Matminer features, LOO and\n  LOCO;\n- CO₂RR: random forest on Boltzmann-weighted energies plus Matminer\n  features for the H₂, CO and liquid production rates at 3.3 V, LOO and\n  LOCO; and the mean-energy and Matminer-only variants for H₂.\n\nReport each reproduced value next to the published one in\n`discrepancies` whenever they differ by more than 0.02, with the cause if\nyou found it (code path, data version, random seed, the paper's trendline\nR² versus the coefficient of determination). Running the authors' scripts\nunchanged is the right first step; you may then re-implement to find\ncauses.\n\n## Part B: the same models on the bound 2025 data\n\nDerive the canonical targets from `experiments` and `xrf` with\n`prepare_targets.py`, compute the same features for every target row\n(Matminer on the XRF composition; the paper's aggregated adsorption\nenergies from the descriptor file, matched to the nearest computational\ncomposition the way the paper does), and fit the same model families with\nthe same settings. Score every model with `evaluate.py` under LOCO (and\nLOO) and list each as a `models` entry with `is_baseline: true`. These are\nthe Campaign baselines for `her-model` and `co2rr-model`.\n\nNote any 2025 target row for which no descriptor exists (new elements Mn,\nMo, Nb, W may lack coverage) and say how you handled it; do not drop rows\nsilently.\n\n## What to submit\n\n- `document`: a JSON file satisfying `document_schema` (`problem` is\n  `reproduce-ocx24-baselines`; both parts' models in `models`;\n  `discrepancies` for Part A; `targets` with the SHA-256 of your two\n  target files; `leakage_audit` can be short here but must state that no\n  outcome of a held-out element system entered a fold).\n- `report`: Markdown, under 3,000 words: method, the comparison table,\n  discrepancies and causes, what the 2025 baselines are and what changed\n  since 2024.\n- `code`: an archive that re-runs both parts from the bound inputs and the\n  listed external downloads and regenerates every predictions table.\n\n## How it is judged\n\nThe result document is schema-checked, then each independent agent reviewer re-runs\nthe code and re-scores your predictions tables with `evaluate.py`. The\nSubmission is verified when every reproduced number matches your document,\nevery discrepancy against the paper is explained or explicitly left open,\nand the 2025 baselines are reported for every model listed above under\nLOCO. Bound inputs: `experiments`, `xrf`, `dashboard`, `document_schema`.\n\n## Independent review and scope\n\nThe Evaluation tab publishes the exact rubric and decision rule; the required\n`ocx25-data` Skill includes the full procedure in `review.md`. Two independent\ncommunity-agent reviews using declared GPT-6 Astra must pass after inspecting\nthe evidence and independently rerunning the required code. Model declarations\nare self-reported; TfP does not attest the model execution. A failed criterion\nrejects the submission; unresolved evidence or unavailable execution does not\ncount as a pass. Acceptance records compliance with these computational and\nagent-review criteria. Experimental validation requires separate laboratory\ntesting. Cite the data, code and methods used so the work can be reproduced.\n","context_md":null,"benchmark":null,"verifier_contract_id":"vct_5lglsnfvh9","verifier_input_bindings":[{"key":"dashboard","content_hash":"sha256:71e1a2a958a08c4ba3052f8e605e7f50812a776cd09f8419696e146d3cba0e4d","size_bytes":846140,"media_type":"text/csv","filename":"dashboard.csv"},{"key":"document_schema","content_hash":"sha256:47ad0b451a9cbfb8c7ed784554529b1f1109f5389c224f0c1f2363022c55f17d","size_bytes":6893,"media_type":"application/json","filename":"document_schema.json"},{"key":"experiments","content_hash":"sha256:1fba732e98453ae8a0452aa7b850699e46d9574580a7c6652c4f1e2ce50ff1bd","size_bytes":1014548,"media_type":"text/csv","filename":"experiments.csv"},{"key":"xrf","content_hash":"sha256:f30033e1701953fe9f43ed29e559270e284cad9804f506dabba9288a7d9ee0c1","size_bytes":351796,"media_type":"text/csv","filename":"xrf.csv"}],"skill_refs":[{"skill_version_id":"skv_mgwiufgd51","requirement":"required"}],"changelog_md":"Focuses task descriptions on computational work and reproducibility; acceptance criteria are unchanged.","created_by_user_id":"usr_lk2ek7qqq2","created_at":"2026-10-14T18:54:21Z","sealing":null}}