Open
feat: best-of-N + LLM-as-judge detector + untell-ceiling (free-ceiling moves)#10
Conversation
…ing moves) Implements the inference-time, zero-cost moves from docs/free-ceiling-report.md: - LLM-as-judge detector (untell/detectors/llm_judge.py): the frontier model scores AI-likelihood against the ai-tells catalog. Key-gated (commercial tier); returns None when no LLM is configured, so it slots into the ensemble like the commercial adapters. Often the strongest signal available for free, since the local proxies are weak. - Best-of-N rewriting (untell_text best_of=N / untell-loop --best-of N): draw N candidate rewrites per iteration and keep the strongest VALID one (every sentinel intact + meaning gate held + lowest detector max). best_of=1 is the previous behaviour unchanged. - untell-ceiling (eval/ceiling.py + console script): measures the loop's inference-only evasion vs the local detector ensemble - the number the literature has no data point for. Reports the baseline detection always; the full before/after delta when a rewriter is configured. Word-substitution (move 2's CPU lever) already ships as untell-loop --polish; the free-GPU LoRA-RL moat (move 4) is scaffolded in training/ and needs a free Colab/Kaggle run. Verified: 169 passed, 1 skipped, 0 failed; ruff clean; untell-ceiling smoke-runs and emits numbers (baseline flagged rate 0.67, mean max P(AI) 0.32 on the built-in sample). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the inference-time, zero-cost moves from
docs/free-ceiling-report.md(PR #9) — the levers to approach the free ceiling.What's new
untell/detectors/llm_judge.py) — the frontier model scores AI-likelihood against theai-tells.mdcatalog. Key-gated (commercial tier); returnsNonewhen no LLM is configured, so it slots into the ensemble like the commercial adapters. Often the strongest free signal, since the local proxies are weak. (report move 3)untell_text(best_of=N)/untell-loop --best-of N: draw N candidates per iteration, keep the strongest valid one (every sentinel intact + meaning gate + lowest detector max).best_of=1unchanged. (report move 2)untell-ceiling(eval/ceiling.py) — measures the loop's inference-only evasion vs the local detectors, the number the literature has no data point for. Baseline always; full before/after delta when a rewriter is configured. (report move 1)Word-substitution (move 2's CPU lever) already ships as
untell-loop --polish; the free-GPU LoRA-RL moat (move 4) is scaffolded intraining/and needs a free Colab/Kaggle run (out of scope for $0 verification).Verified
169 passed, 1 skipped, 0 failed · ruff clean ·
untell-ceilingsmoke-runs and emits numbers (baseline flagged rate 0.67, mean max P(AI) 0.32 on the built-in sample). New tests:test_llm_judge.py,test_ceiling.py, and a best-of-N test intest_run.py.🤖 Generated with Claude Code