Verify "data in paper" DAS claims instead of taking the category at face value #47

Motivation

The stock statement "all relevant data are included in the paper or its supplementary information" is the modal DAS across all four IWA journal snapshots (~2,600 articles, typically 60–75% per journal-year). As a single category it is nearly uninformative: it covers both papers whose complete dataset genuinely fits in the printed tables and papers whose tables hold only summary statistics (means, coefficients, p-values) — which is not the data.

Cross-checking the claim against the supplement fields already in the snapshots (is_supp, num_supp, supp_file_type, supp_url) splits it into three very different populations:

Related observation: across all four journals there is not a single csv, txt, json, or xml supplement — supplements are overwhelmingly docx (~1,190 articles), pdf (~230), doc, and a few dozen xlsx. The top tier of score_fair()'s Interoperable dimension is currently unreachable in the IWA corpus.

Proposed strategy (escalating tiers)

1. Structural cross-check (no new data needed). Score "in paper" claims jointly with the supplement fields rather than independently, so "in paper + xlsx supplement" is distinguished from "in paper, nothing attached". Could live in score_fair() or as a derived column.
2. Cheap heuristics for the no-supplement 71%. Filter by article type first (editorials/reviews make the claim trivially true). Then, since the pipeline already fetches article HTML, extract tables and classify their shape: rows-as-observations (sample IDs, sites, dates) supports the claim; coefficient/p-value tables do not.
3. Supplement content audit — what is actually being shared. The extension goes beyond the file extension: download each supplementary file via supp_url and read its contents.
   • docx/doc → Markdown (e.g. pandoc), then classify what the supplement contains: prose only, figures, summary tables, or tables of observations. A docx full of methods text is a very different "shared supplement" than a docx carrying the study's measurements.
   • xlsx/xls → parse sheets (e.g. readxl) and record structural metrics per sheet: dimensions, header rows, merged cells, mixed types in a column, multiple tables per sheet, color-as-meaning. From these, assess two things: is the content raw observations or already-summarised (rows-as-observations vs aggregates), and how tidy is it against tidy-data principles (one observation per row, one variable per column, machine-readable headers).
   • Persist the results as a new per-file dataset (e.g. suppfiles: doi, file, format, content class, raw-vs-summary, tidiness score/flags), keyed on DOI like the existing decision sheets. Downloads should be cached/committed as a snapshot per the repo's provenance principles, respecting the same politeness rules as the scrapes.
   • This turns the Interoperable and Reusable dimensions from extension-guessing into evidence: a "shared" xlsx of merged-cell summary tables scores very differently from a tidy sheet of raw measurements.
4. LLM adjudication for the remainder. Give a model the methods section plus the extracted tables (from the paper or the converted supplement) and ask one narrow question: could a reader reconstruct the underlying dataset from what is printed/shared? Audit by spot-checking against human ratings on a sample, following the same transparent-rubric philosophy as score_fair() (FAIR assessment rubric for each paper's data management #19). Output a new column such as das_in_paper_verified alongside the raw claim.

Outcome

The uninformative modal category splits into "appropriately self-contained", "data actually in supplement", and "claim not substantiated" — and for the supplement-carrying papers we additionally learn what is actually inside the shared files: raw vs summarised, tidy vs not. Probably the single most informative distinction we could add to the dataset, and a concrete showcase for AI-assisted metascience on this corpus.