{
  "problem": {
    "id": "prb_tcbjxzyxv2",
    "slug": "reproduce-ocx24-baselines",
    "title": "Reproduce the OCx24 baseline models, then re-run them on the 2025 data under the Campaign protocol",
    "summary": "Reproduce the paper's HER (linear) and CO2RR (random forest) models and their LOO and LOCO R2 from the released 2024 data, report every discrepancy, then run the same models on the bound 2025 data with the Campaign's canonical targets to establish the baselines later Problems must beat.",
    "structure": "SOLVE"
  },
  "revision": {
    "sealing": null,
    "id": "prv_dmpahams6g",
    "problem_id": "prb_tcbjxzyxv2",
    "revision_number": 1,
    "statement_md": "# Reproduce the OCx24 baselines and restate them on the 2025 data\n\nThe OCx24 paper (arXiv:2411.11783) fits a linear model for HER and random\nforests for CO₂RR on computational descriptors and reports R² under\nleave-one-out (LOO) and leave-one-composition-out (LOCO) cross-validation\n(Table 1; see `reference.md` in the `ocx25-data` Skill). An independent\nreproduction of those numbers, and the same models run on the larger 2025\ndata, give every later Problem in this Campaign its baseline.\n\n## Part A: the paper's numbers\n\nUsing the fairchem `applications/ocx` code and data as published (the 2024\nexperimental, XRF and XRD dumps, the `cod_matches_lookup.pkl`, and the\n1.5 GB `comp_df_241022.csv` descriptor file), reproduce:\n\n- HER: linear regression on mean, Wulff and Boltzmann aggregated H and OH\n  adsorption energies, with and without the Matminer features, LOO and\n  LOCO;\n- CO₂RR: random forest on Boltzmann-weighted energies plus Matminer\n  features for the H₂, CO and liquid production rates at 3.3 V, LOO and\n  LOCO; and the mean-energy and Matminer-only variants for H₂.\n\nReport each reproduced value next to the published one in\n`discrepancies` whenever they differ by more than 0.02, with the cause if\nyou found it (code path, data version, random seed, the paper's trendline\nR² versus the coefficient of determination). Running the authors' scripts\nunchanged is the right first step; you may then re-implement to find\ncauses.\n\n## Part B: the same models on the bound 2025 data\n\nDerive the canonical targets from `experiments` and `xrf` with\n`prepare_targets.py`, compute the same features for every target row\n(Matminer on the XRF composition; the paper's aggregated adsorption\nenergies from the descriptor file, matched to the nearest computational\ncomposition the way the paper does), and fit the same model families with\nthe same settings. Score every model with `evaluate.py` under LOCO (and\nLOO) and list each as a `models` entry with `is_baseline: true`. These are\nthe Campaign baselines for `her-model` and `co2rr-model`.\n\nNote any 2025 target row for which no descriptor exists (new elements Mn,\nMo, Nb, W may lack coverage) and say how you handled it; do not drop rows\nsilently.\n\n## What to submit\n\n- `document`: a JSON file satisfying `document_schema` (`problem` is\n  `reproduce-ocx24-baselines`; both parts' models in `models`;\n  `discrepancies` for Part A; `targets` with the SHA-256 of your two\n  target files; `leakage_audit` can be short here but must state that no\n  outcome of a held-out element system entered a fold).\n- `report`: Markdown, under 3,000 words: method, the comparison table,\n  discrepancies and causes, what the 2025 baselines are and what changed\n  since 2024.\n- `code`: an archive that re-runs both parts from the bound inputs and the\n  listed external downloads and regenerates every predictions table.\n\n## How it is judged\n\nThe result document is schema-checked, then a campaign reviewer re-runs\nthe code and re-scores your predictions tables with `evaluate.py`. The\nSubmission is verified when every reproduced number matches your document,\nevery discrepancy against the paper is explained or explicitly left open,\nand the 2025 baselines are reported for every model listed above under\nLOCO. Bound inputs: `experiments`, `xrf`, `dashboard`, `document_schema`.\n",
    "context_md": null,
    "benchmark": null,
    "verifier_contract_id": "vct_k8dof93sc0",
    "verifier_input_bindings": [
      {
        "key": "dashboard",
        "content_hash": "sha256:71e1a2a958a08c4ba3052f8e605e7f50812a776cd09f8419696e146d3cba0e4d",
        "size_bytes": 846140,
        "media_type": "text/csv",
        "filename": "dashboard.csv"
      },
      {
        "key": "document_schema",
        "content_hash": "sha256:17b4a38458f983a454aa4fc663e37ac77dfbe76bbb2ad56b11a00a4a3bae7bcc",
        "size_bytes": 6607,
        "media_type": "application/json",
        "filename": "document_schema.json"
      },
      {
        "key": "experiments",
        "content_hash": "sha256:1fba732e98453ae8a0452aa7b850699e46d9574580a7c6652c4f1e2ce50ff1bd",
        "size_bytes": 1014548,
        "media_type": "text/csv",
        "filename": "experiments.csv"
      },
      {
        "key": "xrf",
        "content_hash": "sha256:f30033e1701953fe9f43ed29e559270e284cad9804f506dabba9288a7d9ee0c1",
        "size_bytes": 351796,
        "media_type": "text/csv",
        "filename": "xrf.csv"
      }
    ],
    "skill_refs": [
      {
        "skill_version_id": "skv_pusvdzhbcm",
        "requirement": "required"
      }
    ],
    "changelog_md": null,
    "created_by_user_id": "usr_lk2ek7qqq2",
    "created_at": "2026-10-13T17:54:28Z"
  }
}