Summary

Use spaCy's tokenizer-only make_doc() path in word_tokenize() instead of running the complete statistical pipeline.

Word tokenization only consumes token.text, so running the tagger and dependency parser adds substantial work without affecting the result.

This PR also:

• Preserves the existing oversized-input truncation behavior.
• Adds regression tests comparing tokenizer-only output with the full pipeline.
• Adds a representative multi-format benchmark with exact output-parity checks.

Performance

Benchmarked on an 8-vCPU AMD EPYC host with 15 GiB RAM and Python 3.12.

The benchmark covers large TXT, HTML, DOCX, and PPTX documents through their format-specific partitioners. The spaCy model is warmed before measurement, while token, sentence, and POS result caches are cleared before every measured iteration. This measures the algorithmic improvement rather than repeated-document cache hits.

Each result is the median of three iterations.

Output parity was exact for every document, covering:

• 11,751 TXT elements
• 314 HTML elements
• 4,759 DOCX elements
• 5,116 PPTX elements

The benchmark verifies every element's concrete type and text using a stable SHA-256 fingerprint.

Benchmark reproduction

# On upstream/main
uv run --no-sync python scripts/performance/benchmark_text_partition.py \
  /tmp/main.json

# On this branch
uv run --no-sync python scripts/performance/benchmark_text_partition.py \
  /tmp/candidate.json --compare /tmp/main.json

The benchmark defaults to three iterations over the representative four-document matrix. Custom workloads can be supplied by repeating --input:

uv run --no-sync python scripts/performance/benchmark_text_partition.py \
  /tmp/custom.json \
  --input example-docs/example-10k-230p.html \
  --input example-docs/handbook-872p.docx

Testing

• 121 focused NLP, text-classification, and benchmark tests pass.
• Token parity is covered for punctuation, contractions, URLs, Unicode, emoji, tabs, and newlines.
• Oversized-input truncation is covered.
• A regression test ensures word_tokenize() never invokes the statistical pipeline.
• Benchmark tests cover multi-document execution, cache clearing, stable fingerprints, exact comparison, and output-regression rejection.
• Ruff lint and formatting checks pass.