T5 (Text-To-Text Transfer Transformer) Model

What is T5?

T5 is a revolutionary pre-trained language model proposed by Google that solves all NLP tasks through a unified text-to-text framework.

Full name: Text-To-Text Transfer Transformer

Core Features

1. Encoder-Decoder Architecture

Combines an encoder and a decoder to enable powerful sequence-to-sequence learning.

2. The Unification Idea

Frames all NLP tasks as text-to-text conversion problems.

3. Pre-training + Fine-tuning

Acquires general language understanding through large-scale pre-training.

Why is T5 Important?

Unified Framework

Classification, translation, summarization, and other tasks are unified into the same format.

Strong Performance

Achieves SOTA results on multiple NLP benchmarks.

Simplified Development

A unified interface greatly simplifies model deployment and application.

The Unification Idea

T5's core innovation: unifying all NLP tasks into the form "text input → text output"

Task Examples

| Task Type | Input Example | Output Example | |---------|---------|---------| | Text classification | classify sentiment: This product is really great! | positive | | Machine translation | translate English to Chinese: How are you? | (Chinese translation) | | Summarization | summarize: [long text...] | [summary] | | Question answering | question: What is T5's full name? context: [...] | Text-To-Text Transfer Transformer | | Text generation | generate: Write a poem about spring | (generated poem) |

Task Prefix System

Using unified prefixes, T5 can understand and execute many different NLP tasks:

• translate English to French: - English-French translation
• summarize: - text summarization
• cola sentence: - grammatical acceptability
• stsb sentence1: - semantic similarity
• mnli premise: - natural language inference
• question: - question answering

Advantages of the Unification Idea

• Unified interface - All tasks share the same input/output format, simplifying model design
• Parameter sharing - Different tasks share model parameters, improving training efficiency
• Transfer learning - Pre-trained knowledge transfers easily to downstream tasks
• Multi-task learning - Training multiple tasks simultaneously improves generalization

Encoder-Decoder Architecture

T5 uses the classic Encoder-Decoder structure.

Encoder

• Role: Understands the input semantics
• Mechanism: Bidirectional attention for comprehensive understanding
• Advantages:
• Parallel processing, high training efficiency
• Strong feature extraction

Decoder

• Role: Generates the target text
• Mechanism: Autoregressive generation with Cross-Attention
• Advantages:
• Ensures coherent output
• Effectively uses input information
• Suits a wide range of generation tasks

Data Flow

1. Tokenization - Input text is tokenized 2. Encoder processing - Self-Attention understands the semantics 3. Encoder-Decoder interaction - The Decoder attends to Encoder outputs via Cross-Attention 4. Decoder generation - Target tokens are generated one by one 5. Output - The complete translation/generation result

Pre-training Task: MLM

T5 uses Masked Language Modeling (MLM) for pre-training.

How MLM Works

1. Initialize a character vocabulary - Start from basic characters 2. Random masking - Randomly mask about 15% of tokens 3. Prediction - The model predicts masked tokens using surrounding context 4. Iterative merging - Count character-pair frequencies and merge the most common pairs

MLM Advantages

• Bidirectional learning - Predicting from both left and right context learns deeper semantics
• Unsupervised - No manual labels needed; can train on large corpora
• Lossless and reversible - Token sequences can be fully restored to the original text
• Strong generalization - Handles out-of-vocabulary (OOV) words well

--- Source: Easy AI Learning Platform | This tutorial was created for AI knowledge popularization.