WeTextProcessing: An In-Depth Report on the Open-Source Text Normalization Library

Overview

WeTextProcessing is an open-source text processing library from the WeNet team, specializing in Text Normalization (TN) and Inverse Text Normalization (ITN). It provides bidirectional conversion between unstructured text and standardized formats, a key component of voice interaction systems:

• TN powers the TTS frontend, converting raw text into canonical spoken forms.
• ITN powers the ASR backend, restoring recognition output into user-readable text.

The project follows a "Production First & Production Ready" philosophy, offering out-of-the-box support for Chinese, English, and Japanese via a finite-state transducer (FST) rule engine. Unlike solutions depending on heavyweight libraries such as Pynini at runtime, WeTextProcessing uses pre-built FST models for lightweight deployment.

Core Functionality

Text Normalization (TN)

TN converts non-standard words (NSW) into standard spoken forms via a three-stage pipeline:

1. Pre-processing — character normalization (full-width to half-width), punctuation unification, and removal of blacklisted filler words. 2. NSW normalization — the core stage, with dedicated handlers for:

• Numbers (e.g., "465" → spoken form, full mapping from ones to hundred-millions)
• Fractions ("1/5" → spoken form)
• Percentages ("6.3%" → spoken form)
• Dates ("2002/01/28" → spoken date; multiple formats supported)
• Times ("8:00 a.m." → spoken time)
• Math expressions ("78:96" → scores; "-2" → negative)
• Currency ("￥13.5", "$13.5" → spoken amounts)
• Measures ("25kg", "38°C" → spoken units)
• Number series ("12306" → digit-by-digit reading)
• Erhua (Chinese rhotacization handling)
• Whitelist substitutions (e.g., preserving "CEO")
3. Post-processing — punctuation cleanup and OOV marking.

Inverse Text Normalization (ITN)

ITN mirrors the TN pipeline in reverse, restoring normalized text to original written forms: spoken numbers back to digits, spoken dates back to "2002/01/28", spoken currency back to "¥13.5", spoken digit sequences back to "12306", and so on.

Multilingual Support and Extensibility

• Independent rule sets per language: Chinese lunar dates and erhua, English Roman numerals, Japanese kana conversions.
• Plugin-style rule engine; each scenario (dates, currency, measures) is a separate module loaded on demand.
• Custom rules can be added by editing rule files under tn/chinese/rules/ or itn/chinese/rules/ and rebuilding with --overwrite_cache to regenerate the .far FST archive.

Deployment and Ecosystem

• Python package: pip install WeTextProcessing; pre-built FST models are cached after first download.
• CLI tools: wetn (TN) and weitn (ITN) for quick testing and batch processing.
• C++ runtime: built via CMake (processor_main), depends only on OpenFst — suitable for embedding in C++ projects; used by frameworks such as FunASR.
• Rust implementation: community project wetext-rs, built on rustfst, Python-free, supporting TN/ITN for all three languages.
• Community models are also published on ModelScope.

Application Scenarios

• Intelligent customer service: normalizing user input (e.g., "1百20块" → standard amount) improves intent recognition. Reported figures: unnormalized input raises intent error rates by 35%+; one deployment saw 40% higher comprehension accuracy and 65% lower manual intervention.
• Voice assistants / TTS & ASR: TN ensures natural spoken output; ITN renders ASR results readably. One smart-speaker vendor reported 37% higher satisfaction for numeric readouts.
• Financial data processing: unifying currency and number formats; a bank reportedly cut report processing from 2 hours to 15 minutes with 99.8% accuracy.
• Multilingual content platforms: standardized formats reportedly improved search efficiency by 40%.
• Healthcare and specialized domains: standardizing times and dosages in medical records (reported 25% analysis efficiency gain); domain dictionaries can extend coverage to legal or chemical terminology.

Performance and Limitations

Strengths: FST-based matching runs in linear time, processing hundreds of characters per second; rule-based methods are controllable, explainable, and resource-light compared with deep-learning approaches; weighted priorities select optimal paths among competing rules.

Limitations:

• Rule coverage/maintenance: long-tail cases require ongoing rule authoring and debugging.
• Contextual ambiguity: e.g., "三点五分" can be a time or a score; context cues help but cannot eliminate ambiguity.
• Deployment friction: Windows installation challenges; C++ runtime requires manual compilation.
• Model updates: rule/data resources need periodic updates (quarterly recommended) to track language evolution.

Hybrid architectures combining WeTextProcessing rules with neural models can balance controllability and robustness for complex domains.

Conclusion and Outlook

WeTextProcessing delivers production-grade, bidirectional, multilingual text normalization with full-stack deployment options from Python to C++ and Rust. Future directions include expanding rule coverage with real-world corner cases, adding languages (e.g., Korean), and improving ease of deployment (prebuilt binaries, better Windows support), positioning it as a leading tool for multilingual text preprocessing.