Can an LLM judge tell a sound analysis from one that leads to the wrong decision?

A small benchmark testing rubric-based LLM judges on data science analyses. 16 analyses in 8 matched pairs: each pair contains one version with a planted decision-fatal flaw and one sound control reaching a defensible conclusion on the same topic.

A rubric-based scoring judge rated the flawed analysis higher than its sound control in 6 of 8 matched pairs (5 of 8 in one of five runs).

Numeric rubric scoring rewarded confident presentation over correctness: analyses with decision-fatal flaws (Simpson's paradox, survivorship bias, target leakage) routinely outscored honest versions of the same analysis.

Rubric judge: six criteria scored 1-10, PASS if mean >= 7.0. Five independent runs over the full corpus.

Example pairwise scores (runs/results_run_01.csv):

Note the judge is also unstable: its false-alarm rate swings from 50% to 75% across identical runs, and individual verdicts flip between runs on the same analysis.

Corpus. 8 matched pairs of business-analysis writeups, authored with Claude (via Claude Code). The flawed version of each pair contains one planted flaw serious enough that acting on the recommendation would lead to a materially wrong decision; the control reaches a defensible conclusion on the same topic. Each planted flaw is documented in a flaw_statement field in corpus.json. Flaw taxonomy: spurious causation, Simpson's paradox, survivorship bias, target leakage, multiple comparisons, misread uncertainty, base rate neglect, ignored cost asymmetry.

Judge. A rubric I wrote — six criteria scored 1-10 (decision actionability, analytical focus, statistical accuracy, analytical rigor, methodological fit, evidence sufficiency); PASS if the mean of the criteria >= 7.0. Scored by claude-sonnet-4-6. This mirrors the most common LLM-as-judge design in production eval pipelines.

The repo also runs two freeform-critique judges (bridgekit's evaluate() and redteam() — tools I maintain) over the corpus and stores their raw outputs. In a preliminary evaluation, each caught 6 of 8 planted flaws with no false alarms on sound controls. These numbers are preliminary: single run, labeled by me, non-blind (flaw metadata visible during labeling), which risks anchoring — especially when labeling one's own tools. A blind labeling protocol is the follow-up before treating these as verified.

• n = 16. Small corpus; results are directional. The rubric finding is consistent across 5 runs, but a larger corpus is the obvious next step.
• Synthetic, model-authored corpus. Analyses and planted flaws were authored with Claude, and the judge also runs on Claude (claude-sonnet-4-6)
  • a same-model-family setup that could inflate or deflate rates relative to naturally occurring flaws. Cross-provider replication is planned.
• The PASS threshold (mean >= 7.0) is a design choice; verdict-level results are sensitive to it. The pairwise ranking finding (flawed outscoring sound) does not depend on the threshold.

corpus.json                # 16 analyses with pair ids, flaw types, flaw statements
run_judges.py              # runs judges over the corpus (--runs N for repeats)
analyze.py                 # metrics and cross-run variance (its --human-labels /
                           # --compare-labels flags support a local labeling
                           # workflow and need files not included in this repo)
print_reviews.py           # exports freeform reviews (for future labeling work)
test_one.py                # single-analysis smoke test of the rubric judge
runs/                      # per-run results CSVs
requirements.txt

pip install -r requirements.txt
export ANTHROPIC_API_KEY=...
python run_judges.py --runs 5 --output-dir runs
python analyze.py runs/results_run_*.csv