WeTextProcessing: An In-Depth Research Report

Overview and Background

WeTextProcessing is an open-source text processing library from the WeNet team, focused on text normalization (TN) and inverse text normalization (ITN). It provides bidirectional conversion between unstructured text and standardized formats—a key component for voice interaction systems. In TTS front-ends, TN converts raw text into a canonical form; in ASR back-ends, ITN restores recognized text into a human-readable form.

The library follows a "Production First & Production Ready" philosophy, offering out-of-the-box support for Chinese, English, and Japanese via a finite-state transducer (FST) rule engine. Unlike approaches requiring heavy third-party dependencies at runtime, WeTextProcessing uses pre-built FST models, enabling lightweight deployment without Pynini in the Python runtime.

Core Functionality: TN and ITN

Text Normalization (TN)

TN converts non-standard words (NSW) into standard spoken forms through a three-stage pipeline:

1. Pre-processing: cleans input (full-width to half-width characters, punctuation unification, blacklist removal such as filler words). 2. NSW Normalization: the core stage, with dedicated handlers for:

• Numbers: "465" → "四百六十五", "6.42" → "六点四二"
• Fractions: "1/5" → "五分之一"
• Percentages: "6.3%" → "百分之六点三"
• Dates: "2002/01/28" → "二零零二年一月二十八日" (multiple formats supported)
• Time: "8:00 a.m." → "早上八点"
• Math: "78:96" → "七十八比九十六", "-2" → "负二"
• Money: "￥13.5" → "十三点五元", "$13.5" → "十三点五美元"
• Measure: "25kg" → "二十五千克", "38°C" → "三十八摄氏度"
• Number series: "12306" → "幺二三零六"
• Erhua: optional handling of erhua, e.g. "地儿" → "地"
• Whitelist: special-word replacement
3. Post-processing: removes unnecessary punctuation, marks OOV words, and polishes output.

Inverse Text Normalization (ITN)

ITN reverses the process with the same three-stage architecture: "四百六十五" → "465", "五分之一" → "1/5", "百分之六点三" → "6.3%", "二零零二年一月二十八日" → "2002/01/28", "十三点五元" → "¥13.5", "幺二三零六" → "12306", etc. This bidirectional capability ensures consistency between user input and system output in conversational scenarios.

Multilingual Support and Extensibility

Each language has an independent rule set: Chinese supports lunar dates and erhua; English handles Roman numerals; Japanese includes hiragana/katakana conversion. The rule engine is pluggable—developers can edit rule scripts (e.g., under tn/chinese/rules/ or itn/chinese/rules/) and rebuild the FST cache with --overwrite_cache to generate new .far files.

Deployment and Ecosystem

• Python package: pip install WeTextProcessing; pre-built FST models are downloaded and cached automatically.
• CLI tools: wetn (TN) and weitn (ITN) for quick testing and batch processing.
• C++ runtime: build processor_main via CMake; depends only on OpenFst, no Pynini—suitable for embedded/C++ integration.
• Rust implementation: the community-built wetext-rs uses rustfst and supports Chinese, English, and Japanese TN/ITN without Python.

Ecosystem integrations include ModelScope-hosted ITN models and usage within ASR frameworks such as FunASR.

Application Scenarios

• Smart customer service: normalizing inputs like "1百20块" improves intent recognition. Reported figures include 35%+ error-rate increases from unnormalized input, 40% understanding-accuracy improvement and 65% reduction in manual intervention after adoption.
• Voice assistants / TTS / ASR: TN ensures natural spoken output; ITN renders readable subtitles. One smart-speaker vendor reported a 37% satisfaction increase for numeric announcements.
• Finance: unifies "$1,000" and "1000美元" formats; a bank reportedly cut report processing from 2 hours to 15 minutes with 99.8% accuracy.
• Multilingual platforms: standardized formats improved retrieval efficiency by up to 40%.
• Healthcare and specialized domains: normalizing times and dosages; one hospital reported 25% analysis efficiency gains; domain dictionaries can extend coverage to legal or chemical texts.

Performance and Limitations

The FST engine runs in linear time, processing hundreds of characters per second. Rules are controllable and explainable, with lower compute requirements than neural end-to-end models. Weight and priority mechanisms select the best path among competing rules.

Limitations include:

• Rule coverage and maintenance: long-tail cases require manual rule fixes and familiarity with rule syntax.
• Context ambiguity: e.g., "三点五分" can mean 3:05 (time) or 3.5 points; context hints help but cannot fully disambiguate.
• Deployment friction: installation and compilation challenges remain on platforms like Windows; the C++ runtime requires building from source.
• Model updates: rules must track language evolution; quarterly rule-library updates are recommended.

For complex domains, WeTextProcessing can be combined with deep learning models in hybrid systems to balance controllability and robustness.

Summary and Outlook

WeTextProcessing delivers bidirectional, multilingual, extensible text standardization with full-stack deployment from Python to C++ and Rust. Future work focuses on expanding rule coverage with real-world corner cases, additional languages (e.g., Korean), and easier deployment (pre-built binaries, better Windows support). As text normalization grows more critical across NLP applications, WeTextProcessing is positioned as a leading open-source choice for multilingual text preprocessing.