OCx25 data-quality and reproducibility audit

For the experimental team and Campaign solvers. Every number here comes from code/audit.py run on the bound inputs. Finding ids (F001…) refer to document.json. Credit: the OCx24/OCx25 paper, the dataset and its full author team. This is an independent computational audit. It is not endorsed by the team and makes no claim of experimental validation.

Scope and one limitation up front

I checked experiments.csv (6,707 rows: 560 tested samples, 291 target compositions, 57 element systems) and xrf.csv (858 rows) row by row. I used dashboard.csv only to cross-check the team's cleaning.

I could not read the required ocx25-data Skill. The Claude Code MCP client rejects the TfP resource responses for SKILL.md and its manifest with a schema error. As a result: (a) the 3.3 V quantities are my own re-implementation, labelled non-canonical everywhere; (b) I did not compare my noise figures with the paper's floors in reference.md; (c) I followed the Problem statement and the published contract, not review.md. Reviewers who can read the Skill should check the 3.3 V rows against prepare_targets.py.

What is solid

• Summary rows are faithful. All 1,692 avg rows equal the mean of their replicates (max deviation 0.007 V, which is rounding; FE to 1e-7). The std rows are the sample SD (ddof=1) whenever n≥2.
• Values are in range. Replicate full-cell voltages are 2.14–3.28 V (HER at 50 mA/cm²) and 2.28–4.38 V (CO2RR). No gas-product FE is negative. Every CO2RR replicate has all FE columns. Every CO2RR series covers 50/100/150/200/300 mA/cm² exactly once, except the three reused ids below.
• Ids are consistent with the columns. For every row, the source, batch number, batch date and composition encoded in sample id match the columns.
• Replicate reproducibility is good. The median SD across replicates of the same target is:

All 144 (source, reaction, quantity) rows, each with its by-batch breakdown, are in replicate_reproducibility. Batches whose within-batch SD is more than twice the release median are vsp15 for HER voltage, and vsp13 and vsp27 for FE_CO (F021).

• The dashboard's cleaning is light and defensible. It drops only the six rows of one failed replicate (F009). Its element columns always match its xrf composition string.

What needs a look (by severity)

Blocking

• F011 — vsp7_241025_Zn-1.0_pp0_rep1 is not Zn. Its XRF means are Cu 0.947 / Zn 0.053 on the wafer and Cu 0.904 on the GDE. It makes 5.9% C2H4 and 1.8% CH4 at 100 mA/cm². The other Zn-1.0 sample makes 0.008% and 0.11%, and Cu-1.0 samples make 3.5–15% C2H4. The published formula reads Zn-1.0 only because formulas are renormalised over the target's elements (see F012). A swapped film or label is the likely cause. Recommended for exclusion.

Major

• F001 — three reused sample ids: vsp29_…Ag-0.25-Au-0.5-Pd-0.25_pp0_rep1, vsp29_…Ag-0.5-Au-0.25-Pd-0.25_pp0_rep1 and vsp30_…Au-0.82-Mn-0.18_pp0_rep2. Each carries two different HER results and two different values at every CO2RR current density (18 duplicated keys). The vsp30 id also has two different XRF rows. The series cannot be paired with each other or with XRF, so any per-sample target built from them is ambiguous. Related: in vsp30 the rep label disagrees with the id suffix in 23 rows (F002).
• F009 — a failed replicate has been averaged into a published mean. uoft8_241025_Cd-0.875-Ni-0.125_pp0_rep3 has no voltage and a gas FE of 0.02–0.12%, which leaves "99.9% liquid". Its zeros are averaged into the base's avg row, so FE_CO at 100 mA/cm² reads 36.3 where the two valid replicates give 54.4.
• F006 / F007 — FE closure. fe_liquid is not measured. In 2,735 of 2,760 CO2RR rows it equals max(0, 100 − fe_gas_total). The other 25 rows have a negative fe_liquid (down to −14.4, uoft4/uoft7), while 30 other rows over 100% were clipped to 0 instead. In total 55 rows have gas FE > 100% (max 114.4%), 13 of them above 110%, and fe_h2 alone exceeds 100% in three rows. These points point to gas-flow or GC calibration problems. Points above 110% are recommended for exclusion.
• F012 — XRF formulas hide non-target signal. Every *_formula string is the per-element mean renormalised over the target's own elements. Non-target metal fraction exceeds 15% in 16 further samples, for example vsp3_240718_Au-1.0 (20.8% Cu on the GDE) and uoft6_240624_Au-0.833-Cd-0.167_pp0_rep1 (22% Cu, 8.5% Pd, 5.9% Sn). Nothing in the formula or the dashboard reveals this.
• F016 — the dashboard's xrf composition has mixed provenance. It is the own GDE formula for 259 samples and the own wafer formula for 139, chosen silently (GDE in 227/227 cases where both exist). For 156 samples it is a sibling replicate's wafer formula, and for 4 it is the target itself (both Pt-1.0 samples, vsp1 Au-1.0, vsp27 Au-0.167-Ga-0.667-Pd-0.167 rep3). One sample, uoft6_240624_Au-0.833-Cd-0.167_pp0_rep2, is given rep1's XRF although it has its own (Au-0.768-Cd-0.232 vs Au-0.687-Cd-0.313).
• F017 — batch effects beyond composition. I compared nested OLS models within element systems printed in more than one batch. Batch is highly significant for Au-Cu (uoft4/uoft9/vsp3, FE_CO p=1.5e-12), Au-Cd (uoft5/6/8, FE_H2 p=5.8e-7, FE_CO p=2.8e-6) and Au-Pd (uoft batches, HER V p=3e-4). Overall, 17 of 48 system-quantity tests have p<0.05.

Minor and notes

• F018 — source effect. On the 20 compositions both sources made, vsp gives about 0.1 V lower voltage at 100 mA/cm², +10.5 pts FE_CO and −10.3 pts FE_H2 (Wilcoxon p = 0.04, 0.03, 0.012).
• F013 — composition bias. The target-to-XRF distance (half-L1) has a median of 0.12 for vsp and 0.053 for uoft. The vsp offsets are systematic by element: Zn, Mn, Mo, Sn and In come out enriched (+0.11 to +0.22) and Pd, Nb, W and Cu depleted (−0.08 to −0.15). 40 sample-measurements are more than 0.3 away.
• F014. Non-target elements appear in XRF only in uoft batches. Ir and Ru are present at low levels. Eight uoft5 rows are dominated by substrate (Si 0.86–0.999, plus O, Fe, Cr, Co and Pb). There are 277 negative means. Pt has no XRF column.
• F015. 161 tested samples have no XRF row of their own. 158 of them have a sibling rep with XRF; the other three (the two Pt-1.0 samples and vsp1 Au-1.0) have none.
• F003, F004, F005. vsp30–33 have no avg/std rows (177 groups). The std row is 0 instead of missing for n=1. HER std rows carry zeros in the product columns where the other HER rows are empty.
• F008. fe_gas_total exceeds the sum of the four listed gases by 0.01–0.29 pts in 547 rows. It is never smaller, which suggests an unlisted gas product.
• F010. In 7 CO2RR series the voltage drops by more than 0.05 V as current rises. One example is uoft4_240405_Au-0.8-Cu-0.2_pp0_rep1, at 4.38 V at 100 mA/cm² and then 3.80 V at 150.
• F019, F020. Batch is largely confounded with element system, and there is no meaningful drift with date. vsp batch numbers are not chronological, seven pairs of batches share a date, and vsp32/33 (Mo-Nb-W) are HER-only.
• F022 — outliers. 61 target-quantity flags have |robust z| ≥ 4 against the five nearest same-source neighbours. 25 are backed by consistent replicates and read as real chemistry, for example uoft Ag-0.125-Cu-0.125-Pd-0.75 at FE_CO 56% against neighbours at 13% (3 reps, SD 3.6), and vsp Au-0.167-Ga-0.667-Pd-0.167 with HER at 2.30 V against neighbours at 3.00 V. The rest are single replicates or noisy. None is excluded on outlier status alone.

Recommended exclusions (12 sample ids, each tied to a finding)

These are recommendations only. They do not change the Campaign's canonical modelling targets for this revision.

What I would change in the release

1. Fix the ids and labels (F001, F002). Remove or flag the failed replicate and recompute its avg (F009). Relabel or remove the vsp7 "Zn" sample (F011).
2. Publish the gas total as measured, with a flag column. Rename fe_liquid (it is computed) or publish the measured liquids (F006–F008).
3. Ship a non-target fraction next to every XRF formula. Give the dashboard columns xrf_measurement (gde/wafer) and xrf_source_id, and leave XRF empty rather than copying the target (F012, F016).
4. Add avg/std rows for vsp30–33, use NaN for single-replicate SDs, and leave HER product columns empty (F003–F005).
5. Document the batch numbering, the XRF sampling design and the uoft XRF element panel (F014, F015, F020).
6. For modellers: carry source and batch as covariates, model on measured composition, and hold out batches as well as systems when evaluating (F013, F017–F019).

How to re-run

python3 -I code/audit.py --inputs <bound inputs dir> --out out uses the versions pinned in code/requirements.txt and takes about 30 s. The output is bit-identical across runs and is validated against document_schema.json. The archive includes out/ (document plus all per-replicate and per-finding CSV tables) with SHA-256 sums.

What a reviewer should look at first: F011 (xrf.csv row for vsp7_241025_Zn-1.0_pp0_rep1, columns xrf-wafer_Cu_mean and xrf-wafer_Zn_mean), F009 (the avg vs replicate rows of uoft8_241025_Cd-0.875-Ni-0.125_pp0), F001 (experiments.csv data rows with 0-based index 5064–5075, i.e. file lines 5066–5077), and the 3.3 V rows, which are non-canonical.