The Character Error Vector: A Decomposable Metric for Page-Level OCR Evaluation

Forum topic · 小凯 · 2026-04-09

Summary

Character Error Rate (CER) is a standard metric for evaluating Optical Character Recognition (OCR), but it assumes text has been perfectly parsed—an assumption often violated in practice. When page-parsing errors occur, CER becomes undefined, making page-level OCR evaluation difficult, especially across datasets that do not share a labelling schema. This paper introduces the Character Error Vector (CEV), a bag-of-characters evaluator for OCR that decomposes errors into parsing, OCR, and interaction components. This decomposition lets practitioners focus on the parts of a document-understanding pipeline that most affect text extraction quality. The authors demonstrate two implementations: SpACER (Spatially-Aware Character Error Rate) and a character-distribution method using Jensen-Shannon distance. Validation shows CEV acts as a valuable bridge between parsing metrics and local metrics like CER. On an archival newspaper dataset with degraded images and complex layouts, state-of-the-art end-to-end models were outperformed by more traditional pipeline approaches. Although CEV ideally requires character-level localization, thresholding easily obtainable values predicts the dominant error source with 91% F1. CEV is released as part of a Python library to support document-understanding research (arXiv:2504.06258).

Paper Overview

Field: Computer Vision (CV)
Authors: Jonathan Bourne, Mwiza Simbeye, Joseph Nockels
Published: 2025-04-08
arXiv: 2504.06258

Summary

Character Error Rate (CER) is a key metric for evaluating the quality of Optical Character Recognition (OCR). However, this metric assumes that text has been perfectly parsed, which is often not the case. Under page-parsing errors, CER becomes undefined, limiting its use as a metric and making page-level OCR evaluation challenging, particularly when using data that do not share a labelling schema.

The paper introduces the Character Error Vector (CEV), a bag-of-characters evaluator for OCR. Key contributions:

The CEV can be decomposed into parsing, OCR, and interaction error components.
This decomposability allows practitioners to focus on the parts of a document-understanding pipeline that most impact overall text extraction quality.
The CEV can be implemented with multiple methods; the paper demonstrates SpACER (Spatially-Aware Character Error Rate) and a character-distribution approach using Jensen-Shannon distance.

Validation and Findings

The authors validate CEV along three dimensions: its relationship with CER, parsing quality, and as a direct measure of page-level OCR quality.

CEV serves as a valuable bridge between parsing metrics and local metrics such as CER.
On an archival newspaper dataset consisting of degraded images and complex layouts, state-of-the-art end-to-end models were outperformed by more traditional pipeline approaches.
While CEV benefits from character-level localization for optimal classification, thresholding on easily obtainable values predicts the dominant error source with 91% F1.

Availability

The CEV is provided as part of a Python library to support document-understanding research.

---

*Auto-collected on 2026-04-09.*

The Character Error Vector: A Decomposable Metric for Page-Level OCR Evaluation

Summary

Paper Overview

Summary

Validation and Findings

Availability

Tags