54% of PDFs Don't Need OCR: firecrawl/pdf-inspector's Smart Routing
You hand a 200-page PDF to your RAG system. Without a second thought, the pipeline sends all 200 pages into the OCR engine. GPU fans spin up, bills climb, and it takes 17 seconds.
But 108 of those pages are native PDFs with an embedded text layer — they never needed OCR at all.
firecrawl/pdf-inspector solves exactly this "indiscriminate bombardment" problem. It's a Rust-written PDF classifier that determines whether each page is text-based, scanned, image-based, or mixed in 10–50 milliseconds — then sends only the pages that genuinely need OCR to the GPU.
The Problem: One-Size-Fits-All PDF Processing
Current PDF pipelines share a common inefficiency: every page takes the heaviest path.
The typical flow: PDF arrives → everything goes through OCR → text comes out. Whether a page is a native text PDF (already has a text layer) or a true scan (which does need OCR), it runs through OCR anyway. It's like airport security frisking every passenger regardless of risk level.
In production, the Firecrawl team found a key figure: about 54% of PDF pages are text-based and don't need OCR. These pages have complete font encodings and text operators — you can extract text directly. But existing pipelines send them through OCR alongside scans, wasting 54% of GPU compute.
This isn't a small issue. Firecrawl processes millions of PDFs daily; 54% wasted OCR means enormous GPU cost and latency overhead.
The Approach: Classify First, Then Route
pdf-inspector's core idea is beautifully simple: add a 10ms classifier before OCR.
The classifier doesn't render pages or run models — it only analyzes the PDF's internal structure:
- Font encodings: text font encodings indicate a native text page
- Text operators: text-drawing operators indicate extractable text
- Image coverage: a full-page image suggests a scan or image-only page
- Mantis shrimp phononic shield (2025 Science): the inner Bouligand structure of the club acts as a phononic crystal with band gaps filtering high-frequency shear waves — selective filtering evolved 500 million years ago
- Sparse activation (MoE): activate only relevant experts, not the whole model
- pdf-inspector's smart routing: send only OCR-needing pages to the GPU, no indiscriminate bombardment
- RAG builders: if your pipeline ingests lots of PDFs, the 54% OCR waste is low-hanging fruit
- Document processing services: Rust bindings embed into Python/Node.js/browsers
- Cost optimizers: half your GPU bill is waste — this tool cuts it directly
- Classification only, no extraction: it tells you the page type, but text extraction requires other tools
- No complex layout handling: multi-column, tables, and formulas need downstream layout models
- Rule-based edge cases: pages that "look like text but are actually scans" may be misclassified
Based on these three signals, each page falls into one of four categories:
| Type | Characteristics | Handling | |------|-----------------|----------| | TextBased | Has text operators | Direct extraction, skip GPU | | Scanned | All images | Send to OCR | | ImageBased | Pure images | Send to OCR | | Mixed | Text + images | Hybrid processing |
The key is the TextBased category — 54% of pages land here, go through native extraction in milliseconds, and never touch the GPU.
Speed: Not Slightly Faster — Two Orders of Magnitude
Firecrawl's benchmark numbers are stark:
| Method | Time for 200-page PDF | |--------|----------------------| | Traditional OCR (pymupdf4llm) | 17.117 s | | pdf-inspector + native extraction | 0.470 s | | Speedup | 36x |
The 36x speedup doesn't come from faster OCR — it comes from skipping OCR. pdf-inspector classifies in 10–50ms, then 54% of pages go through native extraction (milliseconds), and only the remaining 46% hit OCR.
It's intelligent airport security: regular travelers take the fast lane (10 seconds), only flagged ones get deep inspection (5 minutes). Throughput improves not by speeding up deep inspection, but by letting most people skip it.
Why Rust
pdf-inspector being written in Rust is no accident:
1. Zero runtime overhead: no GC pauses; classification latency stays stable at 10–50ms 2. Single dependency: only depends on lopdf — trivial to deploy 3. Cross-language bindings: works from Python, Node.js, and WebAssembly 4. No ML models: pure structural analysis — no GPU, no model files
The "no ML models" point matters especially. pdf-inspector isn't AI — it's a rule-based classifier. But its classification determines how much work the AI (OCR models) must do. A rules system scheduling AI systems is division of labor done precisely.
The Conceptual Lineage: Selective Filtering Beats Brute Force
pdf-inspector's core insight fits a cross-domain pattern:
All three point to one principle: selective filtering is more efficient than brute-force blocking. The mantis shrimp filters specific wave frequencies with a phononic crystal rather than blocking everything with a thicker shell. MoE routes tokens to relevant experts rather than running the full model. pdf-inspector routes pages by structural classification rather than OCR-ing everything.
The cost of filtering is always lower than the cost of processing — a universal engineering principle across domains.
Fire-PDF: From Open-Source Component to Commercial Pipeline
pdf-inspector is the open-source component of Firecrawl's larger pipeline, Fire-PDF:
1. pdf-inspector classification (10–50ms/page) → determine each page's type 2. Text-based pages → native extraction (milliseconds, no GPU) 3. Scanned/image pages → neural layout model + OCR (GPU) 4. Lane-based routing: GPU requests isolated by document size, so a 200-page report doesn't affect latency for a single-page invoice
Firecrawl open-sourced the classification layer and commercialized the GPU layer. Smart division: classification rules are general infrastructure worth community improvement; the GPU pipeline is differentiated and worth monetizing.
This "open classification layer + commercial processing layer" split mirrors Euclid-MCP's "LLM as poet, Prolog as accountant" — let rules systems do what rules are good at, let AI do what AI is good at.
Who Should Use It
Limitations
Closing
pdf-inspector's value isn't in doing something complex — it's in doing something extremely simple: looking at what each page actually is before running OCR. A 10-millisecond classification eliminates 54% of the GPU work.
This matches how humans process information: you don't deeply process every input. You scan first, ask "does this deserve careful reading?", and spend attention only where needed. Intelligence isn't processing everything — it's knowing what not to process.
54% of PDFs don't need OCR. Knowing that alone saves half the compute.
---
GitHub: https://github.com/firecrawl/pdf-inspector Fire-PDF blog: https://www.firecrawl.dev/blog/fire-pdf-launch Rust crate: https://crates.io/crates/pdf-inspector