WASH-relevance match overestimates share: whole-article screening options #37

Background

While sizing the 46 filtered_slice_conditional journals (the outcome of the #18 venue mapping, feeding #35), two problems with the current WASH-relevance measure surfaced:

1. The WASH share is overestimated, badly so for large general-science journals (PLoS ONE, Scientific Reports, Blood, oncology journals).
2. The match only ever looked at title and abstract keywords, never the article body, so it measures "mentions a WASH term" rather than "is a WASH article."

This issue records how the match was done, why it overestimates, and the options for whole-article screening. It is methodology follow-up, separate from the indexing work in #32-#34 and the DAS-accessibility survey in #35. Nothing here blocks those; pick up later.

How the current match works

Source: data-raw/washbiblio/harvest_corpus.R, keyword list in data-raw/washbiblio/wash-keywords.csv (59 terms).

• A work is "matched" if its title or abstract contains any one of the 59 WASH keywords, via OpenAlex's title_and_abstract.search filter (keywords quoted where multi-word, joined with OR).
• share = matched works / total works for the journal, 1996-2026.
• OpenAlex stores abstracts as an inverted index with uneven coverage; where the abstract is missing (common for older/non-OA works), the match falls back to title only.
• The search is a normalized token/phrase match. It does not search the body, references, or full text.

Why it overestimates (especially megajournals)

• High-recall, low-precision terms with no context. Several keywords are flagged in wash-keywords.csv itself as needing a venue filter: WASH, E. coli, water supply, water quality, sanitation, water treatment. In a general journal these match incidental mentions (a microbiology assay using E. coli, an environmental-chemistry study touching "water quality").
• OR-logic, single-hit threshold. One keyword anywhere in the abstract flips the whole article to matched; there is no requirement the paper be about WASH.
• No venue gating on the share pass. The high-recall terms were meant to "rely on the downstream venue/relevance filter," but that filter was applied to the author/venue ranking, not to the per-journal share count. For a broad venue the share is pure keyword recall with no precision control.

Read the share as an upper bound on topical relevance: reliable for dedicated WASH journals (genuinely 0.50-0.75, which is why #32-#34 were selected) but inflated for broad venues where WASH terms appear incidentally. 3% of PLoS ONE's ~341k articles is still ~10k incidental hits.

Screening options (cheapest to most thorough)

Tier 1 - tighten the metadata match (no full text):

• Require co-occurring terms (a water term AND a sanitation/health/behaviour term) instead of any single term; drop or venue-gate low-precision terms (E. coli, water quality alone).
• Weight/require title hits over abstract hits for broad journals.
• Intersect the keyword match with OpenAlex topics/concepts tags (still one API call per journal; would cut megajournal false positives sharply).

Tier 2 - better metadata signal, still no body text:

• Pull abstracts from Semantic Scholar where OpenAlex's are missing/truncated.
• LLM relevance classifier over title + abstract with a rubric ("is this article primarily about WASH?"). Biggest precision win short of full text; cheap per article; can be applied only to the borderline low-share journals.

Tier 3 - actual whole-article screening:

• Full-text search via publisher TDM APIs (Elsevier ScienceDirect, Springer, Wiley TDM, PLOS allofPLOS, Europe PMC OA full text). Composes with the Manual DAS-accessibility check for the 46 filtered-slice journals #35 DAS-extraction work (same infrastructure), but per-platform, rate-limited, often paywalled.
• OpenAlex fulltext.search (deliberately avoided in harvest_corpus.R: matches ~9.6M works, noisier rankings). Swings the error toward over-recall (a passing "handwashing" mention matches).
• PDF retrieval + local classification for anything not covered by an API. Most complete, most expensive.

Suggested path: Tier 1 + Tier 2's LLM classifier applied only to the low-share borderline journals, trusting the metadata match for the already-high-share venues. Fixes the specific overestimation without a full-text crawl of millions of articles.

Sizing (why this matters before committing to a screen)

If all articles in the 46 journals were scraped it would be ~2.5M articles; the WASH-matched subset is ~215k. Both numbers use the current (overestimated) match, so treat them as ceilings. est_total = matched / share is roughest at low share.

By matched count (the #35-relevant number)

By publisher platform (concentration)

Elsevier alone is ~49% of the matched total across 18 journals (~53% counting the Pergamon imprint, which is Elsevier). Doing the top ~6 platform families covers >80% of matched articles.

Caveats

• All matched counts come from the current title+abstract keyword match, with its precision limits; they are the ceiling, not the expected scrape volume.
• Whichever screen is chosen, most of these 46 journals will still be handled as keyword-filtered slices (share < 0.50) or routed to openwashdata/washbib's article-level corpus, not scraped whole.

Next step

Pick a screening tier. A concrete first move: sample PLoS ONE's "matched" works, show which keyword triggered each, and estimate the false-positive rate, turning "appears overestimated" into a number before choosing a strategy.

Refs #18, #35