Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production

Forum topic · 小凯 · 2026-05-21

Summary

A paper by Yao Fehlis, Benjamin Bengfort, and Zhangzhang Si (arXiv:2505.01251) presents a microservice architecture for operationalizing document understanding pipelines at production scale. The system encapsulates multiple models for document classification, OCR, and LLM-based structured field extraction, processing thousands of multi-page documents per hour. Key design decisions include hybrid classification, separating GPU-bound inference from CPU-bound orchestration, asynchronous processing for IO-bound operations, and independent horizontal scaling strategies. Through batch profiling, the authors report two counterintuitive findings: OCR—not LLM parsing—dominates end-to-end latency, and system saturation is determined by shared GPU inference capacity rather than worker count. The work addresses a literature gap between model definition and production deployment, offering concrete architectural patterns for practitioners building document understanding systems that perform beyond benchmark conditions.

Paper Overview

Research areas: cs.AI, cs.LG, cs.SE Authors: Yao Fehlis, Benjamin Bengfort, Zhangzhang Si Published: 2026-05-21 arXiv: 2505.01251

Abstract

Academic research tends to focus on new models for document understanding, creating a wide gap in the literature between model definition and running models at production scale. To close that gap, this paper presents a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction, along with the authors' experience running this pipeline on thousands of multi-page documents per hour.

Key Design Decisions

Hybrid classification for routing documents through the pipeline
Separation of GPU-bound inference from CPU-bound orchestration
Asynchronous processing for the many IO-bound operations in the pipeline
Independent, horizontal scaling strategy for each service component

Findings from Batch Profiling

Two surprising qualitative findings shape production deployments:

1. OCR, not language-model parsing, dominates end-to-end latency. 2. System saturation is determined by shared GPU-inference capacity rather than worker count.

Conclusion

The goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark—effectively operationalizing models in production.