Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production

Forum topic · 小凯 · 2026-05-21

Summary

A paper by Yao Fehlis, Benjamin Bengfort, and Zhangzhang Si (arXiv:2505.01251) addresses the gap between document-understanding research and production deployment. The authors present a microservice architecture encapsulating multi-model pipelines for document classification, OCR, and LLM-based structured field extraction, processing thousands of multi-page documents per hour. Key design decisions include hybrid classification, separating GPU-bound inference from CPU-bound orchestration, asynchronous processing for IO-bound operations, and independent horizontal scaling of services. Batch profiling revealed two counterintuitive findings: OCR—not language-model parsing—dominates end-to-end latency, and system saturation is determined by shared GPU-inference capacity rather than the number of worker nodes. The paper offers practitioners concrete architectural patterns for operationalizing document AI systems beyond benchmark settings.

Paper Overview

Research areas: cs.AI, cs.LG, cs.SE Authors: Yao Fehlis, Benjamin Bengfort, Zhangzhang Si arXiv: 2505.01251

Abstract

Academic research tends to focus on new models for document understanding, creating a wide gap in the literature between model definition and running models at production scale. To close that gap, this paper presents a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction, along with the authors' experience running this pipeline on thousands of multi-page documents per hour.

Key Design Decisions

Hybrid classification for document routing
Separation of GPU-bound inference from CPU-bound orchestration
Asynchronous processing for the many IO-bound operations in the pipeline
Independent, horizontal scaling strategies for individual services

Findings from Batch Profiling

Two surprising qualitative findings shape production deployments:

1. OCR, not language-model parsing, dominates end-to-end latency. 2. System saturation occurs at a concurrency determined by shared GPU-inference capacity rather than worker count.

Conclusion

The goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark—effectively operationalizing models in production.

--- *Paper link: https://arxiv.org/abs/2505.01251*