Measure whether semantic code search beats grep on your own repo, instead of trusting a vendor claim.

cocoindex-code (ccc) is a semantic code-search tool that indexes a repo and answers natural-language queries with ranked code chunks. Its README claims "Instant token saving by 70%." This is a small, honest harness that puts that kind of claim to the test on your own codebase, against ripgrep (the free lexical baseline), so you can decide whether an index is worth it before adopting it everywhere.

It is deliberately built so the tool can lose. Ground truth is frozen before any search runs, hits are scored objectively (correct file in top-k), and you can plant exact-string queries where grep should win.

1. Retrieval quality. Whether ccc finds the right file better than ripgrep when you can describe the intent but not the keyword. Reported as MRR and Hit@5.
2. Token savings. How many tokens a ccc result costs to pull the relevant code into an LLM or agent context, versus reading the whole file you would otherwise open. Counted with tiktoken, falling back to a chars/4 estimate that is disclosed at runtime.
3. Latency.

# 1. cocoindex-code (the tool under test); see its repo for the current install line
uv tool install cocoindex-code          # provides `ccc`
# 2. ripgrep (lexical baseline)
brew install ripgrep                     # or your package manager
# 3. this harness's one optional dep (accurate token counts)
pip install tiktoken                     # optional; without it, a chars/4 estimate is used

cd /path/to/your/repo
ccc init && ccc index                    # build the semantic index (once)

# generate an objective query set from your modules' docstrings, then benchmark:
python3 /path/to/this/gen_queries.py --repo . --out queries.jsonl
python3 /path/to/this/bench.py       --repo . --queries queries.jsonl

# on a large repo, draw a reproducible random subset instead of the whole population:
python3 /path/to/this/gen_queries.py --repo . --sample 80 --seed 0 --out queries.jsonl

By default gen_queries.py uses all eligible files (no first-N bias); --sample takes a seeded random subset. You can also use the Makefile from this repo:

make smoke                    # index this repo and run the built-in example
make bench REPO=/path/to/repo # generate queries for a repo and benchmark it
make bench REPO=/path SAMPLE=80 SEED=0

• Frozen ground truth. gen_queries.py derives each query from a Python module's docstring, and the correct file is that module. The query has the filename prefix stripped so it never contains the answer. Ground truth is fixed before any tool runs.
• Objective scoring. A "hit" is the correct file path appearing in the ranked results. There is no subjective "close enough" judgment.
• Fair to grep. The ripgrep baseline greps the query's salient content words and ranks files by match count. You can add "type": "lexical" queries with distinctive exact strings where grep should win, to keep the test honest.
• Same tokenizer on both sides, so the savings ratio is fair regardless of estimator.

• MRR. Mean of 1/rank of the correct file (1.0 means always rank 1; 0 means never found).
• Hit@5. Fraction of queries where the correct file is in the top 5.
• Token savings. 1 - (ccc top-5 result tokens) / (whole-file tokens). This is a floor, because it assumes you already know which file to read, which is the very thing semantic search exists to solve when you do not.

• Auto-generated queries are drawn from the module docstrings that ccc also indexed. They reward ccc for near-exact retrieval of text it has already seen, so its absolute score on the auto set is optimistic. The honest number is the hand-paraphrased set, whose wording avoids the docstring's words (see case_study/CASE_STUDY.md). Trust the paraphrased gap, and use the auto set only for a quick reproducible signal.
• Python-docstring-driven query generation only (v1). For other languages, hand-write queries.jsonl with the same schema.
• Token savings is negative for very small target files. The value of ccc is finding, not compressing.
• Single-repo, single-embedding-model numbers do not transfer. That is the point: run it on your own repo.

See case_study/CASE_STUDY.md. On a ~360-file Python and Markdown research monorepo, on a random 45-query honest set (paraphrased to drop codenames), intent-query MRR was 0.61 for ccc vs 0.11 for ripgrep, a ~5.6x gap that held as the set grew from 15 to 45 queries. The whole 139-file population scores 0.85 (optimistic, because those queries are lifted from indexed text). Token savings were about 60% to 62%. The retrieval gap is the larger effect and the token savings is a secondary benefit. Aggregate JSON results are shipped in case_study/.

ccc mcp refuses to start unless the cwd is already ccc init'd, which makes the user-scope MCP show "Failed to connect" in un-indexed repos. scripts/ccc-mcp-auto is a wrapper that auto-inits (with data-dump excludes) and background-refreshes the index per git repo, then execs the real server. Point your MCP registration at it:

claude mcp add -s user cocoindex-code -e HF_HUB_DISABLE_XET=1 -- /path/to/ccc-mcp-auto

MIT. See LICENSE.