Easy AI Tutorial: Learning Rate Explained

Forum topic · 小凯 · 2026-03-27

Summary

This tutorial from the Easy AI series explains the learning rate, one of the most important hyperparameters in machine learning. The learning rate controls how much model parameters are adjusted during each update, typically set between 0 and 1, and balances training speed against stability. The article offers three intuitive analogies: adjusting study methods after practice problems (large rates like 0.1 learn fast but may overshoot, while tiny rates like 0.0001 are stable but slow), steering a car (big turns react quickly but wobble; small turns are smooth but slow on sharp curves), and seasoning food while cooking. It compares the effects of learning rates that are too large versus too small across convergence speed, stability, and reaching the optimal solution, and outlines three adjustment strategies: fixed learning rate, learning rate decay over training, and adaptive methods such as Adam and AdamW. Typical ranges are also provided: SGD at 0.01-0.1, and Adam/AdamW at 0.0001-0.001.

Learning Rate (Easy AI Tutorial)

What Is a Learning Rate?

The learning rate is one of the most important hyperparameters in machine learning. It determines the magnitude of parameter adjustments at each update, typically set in the range (0, 1).

The learning rate tells the model how fast it should learn during training, balancing speed of progress against stability.

Core Roles

1. Controls parameter adjustment: determines how much parameters change at each update 2. Affects learning speed: balances progress speed with stability 3. Key hyperparameter: directly impacts training results

Intuitive Analogies

Studying and Reviewing

The learning rate is like how much you adjust your study methods after working through problems:

Large learning rate (0.1): big adjustments each time; fast progress but you may go off track
Small learning rate (0.0001): only tiny adjustments; stable but slow progress

Driving and Steering

Large steering: the car changes direction quickly, but may sway
Fine adjustment: smooth driving, but slow to react on sharp turns

Cooking and Seasoning

Adding a lot: flavor changes noticeably, but it's easy to over-salt and hard to fix
Adding a little: steady improvement, but requires many attempts

Learning Rate: Too Large vs. Too Small

| Dimension | Too Large | Too Small | |-----------|-----------|-----------| | Convergence speed | Fast (early on) | Slow | | Stability | Unstable, may oscillate | Stable | | Optimal solution | May miss the optimum | Can reach it, but slowly |

Learning Rate Adjustment Strategies

1. Fixed learning rate: use the same learning rate throughout training 2. Learning rate decay: gradually reduce it as training progresses 3. Adaptive learning rate: optimizers like Adam and AdamW adjust automatically based on gradients

Common Learning Rate Settings

| Optimizer | Typical Learning Rate Range | |-----------|------------------------------| | SGD | 0.01 - 0.1 | | Adam | 0.0001 - 0.001 | | AdamW | 0.0001 - 0.001 |

--- Source: Easy AI Tutorial