# OCx25 data-quality and reproducibility audit

*For the experimental team and Campaign solvers. Every number here comes from `code/audit.py`
run on the bound inputs. Finding ids (F001…) refer to `document.json`. Credit: the OCx24/OCx25
paper, the dataset and its full author team. This is an independent computational audit. It is
not endorsed by the team and makes no claim of experimental validation.*

## Scope and one limitation up front

I checked `experiments.csv` (6,707 rows: 560 tested samples, 291 target compositions, 57 element
systems) and `xrf.csv` (858 rows) row by row. I used `dashboard.csv` only to cross-check the team's
cleaning.

**I could not read the required `ocx25-data` Skill.** The Claude Code MCP client rejects the TfP
resource responses for `SKILL.md` and its manifest with a schema error. As a result:
(a) the 3.3 V quantities are my own re-implementation, labelled non-canonical everywhere;
(b) I did not compare my noise figures with the paper's floors in `reference.md`;
(c) I followed the Problem statement and the published contract, not `review.md`. Reviewers
who can read the Skill should check the 3.3 V rows against `prepare_targets.py`.

## What is solid

- **Summary rows are faithful.** All 1,692 `avg` rows equal the mean of their replicates (max
  deviation 0.007 V, which is rounding; FE to 1e-7). The `std` rows are the sample SD (ddof=1)
  whenever n≥2.
- **Values are in range.** Replicate full-cell voltages are 2.14–3.28 V (HER at 50 mA/cm²) and
  2.28–4.38 V (CO2RR). No gas-product FE is negative. Every CO2RR replicate has all FE columns. Every CO2RR
  series covers 50/100/150/200/300 mA/cm² exactly once, except the three reused ids below.
- **Ids are consistent with the columns.** For every row, the source, batch number, batch date
  and composition encoded in `sample id` match the columns.
- **Replicate reproducibility is good.** The median SD across replicates of the same target is:

| quantity | vsp | uoft | all |
|---|---|---|---|
| HER V @ 50 mA/cm² | 0.044 V | 0.046 V | 0.045 V (140 targets) |
| CO2RR V @ 100 | 0.049 V | 0.064 V | 0.058 V |
| FE_H2 @ 100 | 4.1 pts | 3.4 pts | 3.6 pts |
| FE_CO @ 100 | 2.8 pts | 1.9 pts | 2.2 pts |
| CO rate @ 100 | 52 | 36 | 40 µmol h⁻¹ cm⁻² |
| j @ 3.3 V (non-canonical) | 12.9 | 13.7 | 13.2 mA/cm² |

  All 144 (source, reaction, quantity) rows, each with its by-batch breakdown, are in
  `replicate_reproducibility`. Batches whose within-batch SD is more than twice the release median
  are vsp15 for HER voltage, and vsp13 and vsp27 for FE_CO (F021).
- **The dashboard's cleaning is light and defensible.** It drops only the six rows of one
  failed replicate (F009). Its element columns always match its `xrf composition` string.

## What needs a look (by severity)

**Blocking**

- **F011 — `vsp7_241025_Zn-1.0_pp0_rep1` is not Zn.** Its XRF means are Cu 0.947 / Zn 0.053 on the
  wafer and Cu 0.904 on the GDE. It makes 5.9% C2H4 and 1.8% CH4 at 100 mA/cm². The other
  Zn-1.0 sample makes 0.008% and 0.11%, and Cu-1.0 samples make 3.5–15% C2H4. The published formula reads `Zn-1.0`
  only because formulas are renormalised over the target's elements (see F012). A swapped film or
  label is the likely cause. Recommended for exclusion.

**Major**

- **F001 — three reused sample ids:** `vsp29_…Ag-0.25-Au-0.5-Pd-0.25_pp0_rep1`,
  `vsp29_…Ag-0.5-Au-0.25-Pd-0.25_pp0_rep1` and `vsp30_…Au-0.82-Mn-0.18_pp0_rep2`. Each carries two
  different HER results and two different values at every CO2RR current density (18 duplicated
  keys). The vsp30 id also has two different XRF rows. The series cannot be paired with each other or with XRF, so
  any per-sample target built from them is ambiguous. Related: in vsp30 the `rep` label disagrees with the id
  suffix in 23 rows (F002).
- **F009 — a failed replicate has been averaged into a published mean.** `uoft8_241025_Cd-0.875-Ni-0.125_pp0_rep3`
  has no voltage and a gas FE of 0.02–0.12%, which leaves "99.9% liquid". Its zeros are averaged
  into the base's `avg` row, so FE_CO at 100 mA/cm² reads 36.3 where the two valid replicates give 54.4.
- **F006 / F007 — FE closure.** `fe_liquid` is not measured. In 2,735 of 2,760 CO2RR rows it
  equals max(0, 100 − fe_gas_total). The other 25 rows have a negative fe_liquid (down to −14.4,
  uoft4/uoft7), while 30 other rows over 100% were clipped to 0 instead. In total 55 rows have
  gas FE > 100% (max 114.4%), 13 of them above 110%, and fe_h2 alone exceeds 100% in three rows.
  These points point to gas-flow or GC calibration problems. Points above 110% are recommended for
  exclusion.
- **F012 — XRF formulas hide non-target signal.** Every `*_formula` string is the per-element mean
  renormalised over the target's own elements. Non-target metal fraction exceeds 15% in 16 further
  samples, for example `vsp3_240718_Au-1.0` (20.8% Cu on the GDE) and `uoft6_240624_Au-0.833-Cd-0.167_pp0_rep1`
  (22% Cu, 8.5% Pd, 5.9% Sn). Nothing in the formula or the dashboard reveals this.
- **F016 — the dashboard's `xrf composition` has mixed provenance.** It is the own GDE formula for
  259 samples and the own wafer formula for 139, chosen silently (GDE in 227/227 cases where both exist).
  For 156 samples it is a sibling replicate's wafer formula, and for 4 it is the target itself (both Pt-1.0
  samples, vsp1 Au-1.0, vsp27 Au-0.167-Ga-0.667-Pd-0.167 rep3). One sample,
  `uoft6_240624_Au-0.833-Cd-0.167_pp0_rep2`, is given rep1's XRF although it has its own
  (Au-0.768-Cd-0.232 vs Au-0.687-Cd-0.313).
- **F017 — batch effects beyond composition.** I compared nested OLS models within element systems
  printed in more than one batch. Batch is highly significant for Au-Cu (uoft4/uoft9/vsp3, FE_CO
  p=1.5e-12), Au-Cd (uoft5/6/8, FE_H2 p=5.8e-7, FE_CO p=2.8e-6) and Au-Pd (uoft batches, HER V
  p=3e-4). Overall, 17 of 48 system-quantity tests have p<0.05.

**Minor and notes**

- **F018 — source effect.** On the 20 compositions both sources made, vsp gives about 0.1 V lower
  voltage at 100 mA/cm², +10.5 pts FE_CO and −10.3 pts FE_H2 (Wilcoxon p = 0.04, 0.03, 0.012).
- **F013 — composition bias.** The target-to-XRF distance (half-L1) has a median of 0.12 for vsp and
  0.053 for uoft. The vsp offsets are systematic by element: Zn, Mn, Mo, Sn and In come out enriched
  (+0.11 to +0.22) and Pd, Nb, W and Cu depleted (−0.08 to −0.15). 40 sample-measurements are more than 0.3 away.
- **F014.** Non-target elements appear in XRF only in uoft batches. Ir and Ru are present at low
  levels. Eight uoft5 rows are dominated by substrate (Si 0.86–0.999, plus O, Fe, Cr, Co and Pb). There are 277 negative means.
  Pt has no XRF column.
- **F015.** 161 tested samples have no XRF row of their own. 158 of them have a sibling rep with XRF; the
  other three (the two Pt-1.0 samples and vsp1 Au-1.0) have none.
- **F003, F004, F005.** vsp30–33 have no avg/std rows (177 groups). The std row is 0 instead of
  missing for n=1. HER std rows carry zeros in the product columns where the other HER rows are empty.
- **F008.** fe_gas_total exceeds the sum of the four listed gases by 0.01–0.29 pts in 547 rows. It is
  never smaller, which suggests an unlisted gas product.
- **F010.** In 7 CO2RR series the voltage drops by more than 0.05 V as current rises. One example is
  `uoft4_240405_Au-0.8-Cu-0.2_pp0_rep1`, at 4.38 V at 100 mA/cm² and then 3.80 V at 150.
- **F019, F020.** Batch is largely confounded with element system, and there is no meaningful drift
  with date. vsp batch numbers are not chronological, seven pairs of batches share a date, and vsp32/33
  (Mo-Nb-W) are HER-only.
- **F022 — outliers.** 61 target-quantity flags have |robust z| ≥ 4 against the five nearest
  same-source neighbours. 25 are backed by consistent replicates and read as real chemistry, for
  example uoft Ag-0.125-Cu-0.125-Pd-0.75 at FE_CO 56% against neighbours at 13% (3 reps, SD 3.6), and vsp
  Au-0.167-Ga-0.667-Pd-0.167 with HER at 2.30 V against neighbours at 3.00 V. The rest are single replicates or
  noisy. **None is excluded on outlier status alone.**

## Recommended exclusions (12 sample ids, each tied to a finding)

| sample | why | finding |
|---|---|---|
| the 3 reused ids | series cannot be attributed | F001 |
| uoft8_241025_Cd-0.875-Ni-0.125_pp0_rep3 | failed run | F009 |
| vsp7_241025_Zn-1.0_pp0_rep1 | about 95% Cu, Cu chemistry | F011 |
| 7 samples with CO2RR points >110% gas FE | impossible closure (exclude those points; HER unaffected) | F007 |

These are recommendations only. They do not change the Campaign's canonical modelling targets
for this revision.

## What I would change in the release

1. Fix the ids and labels (F001, F002). Remove or flag the failed replicate and recompute its avg (F009).
   Relabel or remove the vsp7 "Zn" sample (F011).
2. Publish the gas total as measured, with a flag column. Rename `fe_liquid` (it is computed) or
   publish the measured liquids (F006–F008).
3. Ship a non-target fraction next to every XRF formula. Give the dashboard columns
   `xrf_measurement` (gde/wafer) and `xrf_source_id`, and leave XRF empty rather than copying the target
   (F012, F016).
4. Add avg/std rows for vsp30–33, use NaN for single-replicate SDs, and leave HER product columns
   empty (F003–F005).
5. Document the batch numbering, the XRF sampling design and the uoft XRF element panel (F014, F015, F020).
6. For modellers: carry source and batch as covariates, model on measured composition, and hold out
   batches as well as systems when evaluating (F013, F017–F019).

## How to re-run

`python3 -I code/audit.py --inputs <bound inputs dir> --out out` uses the versions pinned in
`code/requirements.txt` and takes about 30 s. The output is bit-identical across runs and is
validated against `document_schema.json`. The archive includes `out/` (document plus all per-replicate and
per-finding CSV tables) with SHA-256 sums.

**What a reviewer should look at first:** F011 (`xrf.csv` row for `vsp7_241025_Zn-1.0_pp0_rep1`,
columns `xrf-wafer_Cu_mean` and `xrf-wafer_Zn_mean`), F009 (the `avg` vs replicate rows of
`uoft8_241025_Cd-0.875-Ni-0.125_pp0`), F001 (`experiments.csv` data rows with 0-based index 5064–5075, i.e. file lines 5066–5077), and the 3.3 V
rows, which are non-canonical.