PDF Parser Benchmark
Five parsers behind one interface, scored against text a person checked.
Why this exists
Every document-AI pipeline starts with a parser, and the choice is rarely made on purpose. A parser that scrambles a multi-column page or flattens a table does not fail loudly — it returns plausible text, and everything built on top quietly degrades.
So the harness puts the same documents through every parser and scores each output against a reference a person checked: character and word error rate, table rows kept intact, right-to-left reading order, and Indic shaping. The results tables are generated by the script, and CI re-runs the benchmark and fails if a number stops reproducing.
What the local parsers showed
- Three columns: PyPDF and PDFMiner keep the reading order (CER 0.046 and 0.050); PDFPlumber reads straight across the columns (CER 0.786).
- A 21-row ruled table: PDFPlumber keeps every row intact (21/21), PyPDF 15, PDFMiner none — while PyPDF has the best character accuracy on the same page.
- Right-to-left text: PyPDF keeps Arabic and Urdu in logical order; PDFPlumber reverses them; PDFMiner scrambles the order. All three break Hindi and Bengali shaping.
- A scanned page: all three return nothing — text-layer parsers need OCR behind them.
The parser that is best for one document is worst for another, which is the point: the right choice is a measurable property of your documents.
What I am building next
LlamaParse and AWS Textract are wired into the same interface and scored the same way; running them with keys fills in the OCR and layout-model rows. After that, a larger corpus so the differences carry confidence intervals.