54% of PDFs Don't Need OCR: The Smart Routing Trick Behind firecrawl/pdf-inspector

You feed a 200-page PDF into your RAG system. Without asking any questions, the system sends all 200 pages into the OCR engine. GPU fans spin up, your bill climbs, and 17 seconds later you get output.

But 108 of those pages were native PDFs with an existing text layer—OCR was never needed.

firecrawl/pdf-inspector solves exactly this "indiscriminate bombardment" problem. It's a Rust-written PDF classifier that determines within 10-50 milliseconds whether each page is text-based, scanned, image-based, or mixed—then sends only the pages that truly need OCR to the GPU.

The Problem: The "One-Size-Fits-All" Trap in PDF Processing

Current PDF pipelines share a common inefficiency: every page goes down the heaviest path.

The typical flow is: PDF arrives → everything goes to OCR → text comes out. Whether the page is a native text PDF (which already has a text layer) or a scanned document (which genuinely needs OCR), it goes through OCR anyway. It's like airport security frisking every passenger the same way, regardless of risk profile.

The Firecrawl team found a key data point in production: about 54% of PDF pages are text-based and don't need OCR. These pages have complete font encodings and text operators—text can be extracted directly. But existing pipelines send them to OCR alongside scans, wasting 54% of GPU compute.

This is not a small problem. Firecrawl processes millions of PDFs daily; 54% of wasted OCR means enormous GPU waste and latency.

The pdf-inspector Approach: Classify First, Then Route

pdf-inspector's core idea is remarkably simple: add a 10-millisecond classifier before OCR.

The classifier doesn't render pages or run models. It only analyzes the PDF's internal structure:

• Font encodings: text font encodings indicate a native text page
• Text operators: text-drawing operators mean extractable text exists
• Image coverage: a full-page image suggests a scan or image page

Based on these three signals, pdf-inspector assigns each page to one of four categories:

| Type | Characteristics | Handling | |------|-----------------|----------| | TextBased | Has text operators | Extract directly, skip the GPU | | Scanned | All images | Send to OCR | | ImageBased | Pure images | Send to OCR | | Mixed | Text + images | Hybrid processing |

The key category is TextBased—54% of pages fall here and go through native extraction in milliseconds, with the GPU completely uninvolved.

Speed Comparison: Not Marginally Faster—Two Orders of Magnitude

Firecrawl's benchmarks are concrete:

| Processing method | 200-page PDF time | |-------------------|-------------------| | Traditional OCR (pymupdf4llm) | 17.117 s | | pdf-inspector + native extraction | 0.470 s | | Speedup | 36x |

The 36x speedup doesn't come from faster OCR—it comes from skipping OCR. pdf-inspector's classifier finishes in 10-50ms, then 54% of pages go through native extraction (milliseconds), and only the remaining 46% hit the GPU.

It's like intelligent airport security lanes: ordinary travelers take the fast lane (10 seconds), and only flagged travelers get deep inspection (5 minutes). Overall throughput improves not by making deep inspection faster, but by letting most people skip it.

Why Rust

pdf-inspector being written in Rust is no accident:

1. Zero runtime overhead: no GC pauses; classification latency stays stable at 10-50ms 2. Single dependency: only lopdf; easy to deploy 3. Cross-language bindings: Python, Node.js, and WebAssembly all supported 4. No ML models: pure structural analysis—no GPU, no model files

The "no ML models" point matters especially. pdf-inspector isn't AI—it's a rule-based classifier. But its classification results determine the workload of the AI (OCR models). A rules system doing the scheduling for an AI system is a precise division of labor.

Conceptual Lineage: Selective Filtering Beats Brute Force

pdf-inspector's core insight fits into a cross-domain conceptual lineage:

• Mantis shrimp phononic shield (2025 Science): the Bouligand structure inside the fist acts as a phononic crystal with bandgaps that filter high-frequency shear waves—selective filtering evolved 500 million years ago
• Sparse activation (MoE): activate only relevant experts, not the entire model
• pdf-inspector smart routing: send only OCR-needing pages to the GPU, no indiscriminate bombardment

All three point to the same principle: selective filtering is more efficient than brute-force blocking. The mantis shrimp filters specific wave frequencies with phononic crystals rather than blocking everything with a thicker shell. MoE uses a routing network to activate relevant experts rather than running the whole model on every token. pdf-inspector routes pages with a structural classifier rather than running OCR on everything.

The cost of filtering is always lower than the cost of processing—a universal engineering principle across domains.

Fire-PDF: From Open-Source Component to Commercial Pipeline

pdf-inspector is an open-source component of Firecrawl's larger Fire-PDF pipeline. The full pipeline works like this:

1. pdf-inspector classification (10-50ms/page) → determine each page's type 2. Text-based pages → native extraction (milliseconds, no GPU) 3. Scanned/image pages → neural layout model + OCR (GPU) 4. Lane-based routing: GPU requests are isolated by document size, so a 200-page report doesn't affect latency for a single-page invoice

Firecrawl open-sourced the classification layer (pdf-inspector) and commercialized the GPU layer. It's a smart split: classification rules are general infrastructure worth opening to community improvement, while the GPU pipeline is a differentiated competitive advantage worth monetizing.

This "open-source classification layer + commercial processing layer" division mirrors Euclid-MCP's "LLM as the poet, Prolog as the accountant"—let rules systems do what rules are good at, and let AI do what AI is good at.

Who Should Use It

• RAG builders: if your pipeline handles lots of PDFs, the 54% OCR waste is low-hanging fruit
• Document processing services: the Rust bindings embed into Python/Node.js/browsers
• Cost optimizers: if half your GPU bill is waste, this tool cuts it directly

Limitations

• Classification only, no extraction: pdf-inspector tells you what a page is, but you need other tools to extract text
• No handling of complex layouts: multi-column, tables, and formulas need downstream layout models
• Rule-based classification edges: some pages that "look like text but are actually scans" may be misclassified

Closing

pdf-inspector's value isn't in doing something complicated—it's in doing something extremely simple: taking a look at what each page actually is before running OCR. A 10-millisecond classification saves 54% of the GPU work.

This matches how humans process information: you don't think deeply about every input. You scan first to decide "does this deserve careful reading?" and only invest attention where needed. Intelligence isn't processing everything—it's knowing what not to process.

54% of PDFs don't need OCR. Knowing that alone saves half your compute.

---

GitHub: https://github.com/firecrawl/pdf-inspector Fire-PDF blog: https://www.firecrawl.dev/blog/fire-pdf-launch Rust crate: https://crates.io/crates/pdf-inspector