T5 (Text-To-Text Transfer Transformer) Model

What is T5?

T5 is a revolutionary pre-trained language model proposed by Google that solves all NLP tasks through a unified text-to-text framework.

Full name: Text-To-Text Transfer Transformer

Core Features

1. Encoder-Decoder Architecture

Combines an encoder and a decoder to enable powerful sequence-to-sequence learning.

2. The "Unification" Philosophy

Converts all NLP tasks into text-to-text transformation problems.

3. Pre-training + Fine-tuning

Acquires general language understanding through large-scale pre-training.

Why is T5 Important?

• Unified framework — classification, translation, summarization, and more all share the same format.
• Strong performance — achieves SOTA results on multiple NLP benchmarks.
• Simplified development — a unified interface greatly simplifies deployment and application.

The Grand Unification Philosophy

T5's core innovation: all NLP tasks are unified as "text input → text output"

Task Examples

| Task Type | Input Example | Output Example | |-----------|--------------|----------------| | Text classification | classify sentiment: This product is really great! | positive | | Machine translation | translate English to Chinese: How are you? | (Chinese translation) | | Summarization | summarize: [long text...] | [summary] | | Question answering | question: What is T5's full name? context: [...] | Text-To-Text Transfer Transformer | | Text generation | generate: Write a poem about spring | (a spring poem) |

Task Prefix System

Through unified prefixes, T5 can understand and execute many different NLP tasks:

• translate English to French: — English-to-French translation
• summarize: — text summarization
• cola sentence: — grammatical acceptability
• stsb sentence1: — semantic similarity
• mnli premise: — natural language inference
• question: — question answering

Advantages of Unification

• Unified interface — the same input/output format for all tasks simplifies model design
• Parameter sharing — tasks share model parameters, improving training efficiency
• Transfer learning — pre-trained knowledge transfers easily to downstream tasks
• Multi-task learning — training multiple tasks together improves generalization

Encoder-Decoder Architecture

Encoder

• Role: understands input semantics
• Mechanism: bidirectional self-attention for comprehensive understanding
• Advantages:
• Parallel processing, efficient training
• Powerful feature extraction

Decoder

• Role: generates target text
• Mechanism: autoregressive generation with cross-attention
• Advantages:
• Coherent output
• Effectively uses input information
• Suits a wide range of generation tasks

Data Flow

1. Tokenization — input text is tokenized 2. Encoder processing — semantics understood via self-attention 3. Encoder-Decoder interaction — the decoder attends to encoder outputs via cross-attention 4. Decoder generation — target tokens generated one by one 5. Output — the complete translation/generation result

Pre-training Task: MLM

T5 is pre-trained with masked language modeling (MLM).

How MLM Works

1. Initialize a character vocabulary — start from base characters 2. Random masking — randomly select ~15% of tokens to mask 3. Prediction — the model predicts masked tokens from surrounding context 4. Iterative merging — token pair frequencies are counted and the most frequent pairs merged

MLM Advantages

• Bidirectional learning — uses both left and right context to learn deep semantics
• Unsupervised — no manual labels needed; trainable on large corpora
• Lossless and invertible — token sequences can be fully restored to original text
• Strong generalization — handles out-of-vocabulary (OOV) words

--- Source: Easy AI learning platform | This tutorial was created for AI knowledge popularization.