An open, reproducible benchmark of open-source document parsers for RAG & LLM pipelines. Real documents, real failures, honest scores.

Document parsing is the single biggest source of silent failures in RAG pipelines. A parser that scrambles reading order, drops tables, or flattens headings produces clean-looking Markdown that quietly destroys retrieval quality. This project measures exactly that — on real documents, with the failures left in.

v0.2 removes the two biggest v0.1 biases (Docling base64 bloat, Marker timeouts) and adds the first OCR test. See results/benchmark-v0.1.md for the v0.1 archive.

We ran 4 open-source parsers on 6 PDFs (academic, financial, technical, image-heavy, dense-text, and a scanned/OCR document) and scored every output on 5 dimensions weighted for RAG quality.

⚠️ Marker's 4.13 is still not directly comparable. Even in --mode fast it times out on the 17 MB NVIDIA report, so it's scored on 5/6 documents while skipping the largest file. On a level playing field (all six documents), Docling and MinerU tie at 3.30 as the strongest overall. Full breakdown: results/caveats.md.

Most parser comparisons show a tidy table of green checkmarks. The interesting information lives in what fails: which parser turns a financial table into garbage, which one returns nothing from a scanned page, which one hangs for 20 minutes on a large file. We report those failures, because that's what actually matters when you pick a parser for production RAG.

Poor extraction quality cascades into:

• broken document structure and reading order
• lost or corrupted tables
• missing heading hierarchy
• bad chunk boundaries
• inaccurate retrieval, and wrong LLM answers

The runner is scripts/run_benchmark.py. It invokes each parser's CLI, times every run, and records success/failure plus output size.

Each parser output is scored 0–5 on five dimensions, weighted toward what actually moves RAG quality:

Scores are an AI-assisted preliminary assessment of the actual Markdown outputs, pending human review. For the scanned document, OCR accuracy is also measured objectively as char-level similarity vs a known ground truth (scripts/ocr_accuracy.py). Full rubric and per-document scores: results/scoring-v0.2.md.

v0.2 settings: Docling --image-export-mode placeholder (v0.1 used embedded, which inlined base64); Marker --mode fast (v0.1 used balanced, whose VLM timed out on large files).

Marker's average excludes the timed-out NVIDIA file. Two surprises worth knowing: Docling OCRs a clean scan best (0.988) despite failing on the complex image-heavy report, and Marker reconstructed the scanned table perfectly via OCR — the best table result in the benchmark.

Full raw data: results/benchmark-report.md · results/benchmark-report.csv

See REPRODUCING.md for environment setup, the isolated-install notes for each parser, and the exact v0.2 commands. Quick start:

python scripts/run_benchmark.py --timeout 1200
# v0.2 de-biased reruns:
python scripts/run_benchmark.py --tools docling --docling-image-mode placeholder --timeout 1200
python scripts/run_benchmark.py --tools marker --marker-mode fast --timeout 1200

DocCrush — an online document processing workflow that converts documents into clean Markdown optimized for AI/RAG applications. This benchmark exists to be honest about the trade-offs in the open-source landscape.

Found a parser failure we missed? Have a document that breaks these tools in a new way? Open an issue with the document and the parser output, or submit a pull request. We especially welcome:

• additional challenging documents (with permission to redistribute)
• corrections to the AI-assisted scores after human review
• new parsers added to the runner

MIT — both the benchmark code and the curated results. Individual source documents retain their original licenses; the synthesized scanned-report-cc0.pdf and its ground truth are CC0 (see benchmarks/documents/README.md).