Easy AI Tutorial: What Is Learning Rate in Machine Learning?

Forum topic · 小凯 · 2026-03-27

Summary

Learning rate is one of the most important hyperparameters in machine learning, controlling how much model parameters are adjusted during each update. It is typically chosen in the range (0, 1) and balances training speed against stability. This Easy AI tutorial explains the concept with intuitive analogies: a large learning rate (e.g., 0.1) makes fast progress but risks overshooting or oscillating, like steering a car too aggressively or over-salting a dish; a small learning rate (e.g., 0.0001) is stable but slow to converge. The article compares the effects of learning rates that are too large versus too small across convergence speed, stability, and ability to reach the optimum. It also covers adjustment strategies—fixed learning rate, learning rate decay, and adaptive methods such as Adam and AdamW—and lists typical learning-rate ranges: SGD at 0.01-0.1, Adam at 0.0001-0.001, and AdamW at 0.0001-0.001. A beginner-friendly reference for understanding and tuning learning rates.

Learning Rate (Easy AI Tutorial)

What Is Learning Rate?

The learning rate is one of the most important hyperparameters in machine learning. It determines the magnitude of parameter adjustments during each update and is usually set in the range (0, 1).

The learning rate tells the model how fast it should learn during training, balancing the speed of progress against stability.

Core Roles

1. Controls parameter adjustment: determines the step size of each update 2. Affects learning speed: balances progress speed and stability 3. Key hyperparameter: directly influences training effectiveness

Intuitive Analogies

Studying and Reviewing

The learning rate is like how much you adjust your study method after solving problems:

Large learning rate (0.1): big adjustments each time—fast progress, but you may go off track
Small learning rate (0.0001): tiny adjustments each time—stable, but slow progress

Driving and Steering

Large turns: the car changes direction quickly, but may sway
Fine adjustments: smooth driving, but slow to react on sharp curves

Seasoning While Cooking

A lot of salt: the flavor changes noticeably, but it can easily become too salty and hard to fix
A little salt: steady improvement, but requires many tries

Too Large vs. Too Small

| Dimension | Learning Rate Too Large | Learning Rate Too Small | |-----------|------------------------|--------------------------| | Convergence speed | Fast (initially) | Slow | | Stability | Unstable, may oscillate | Stable | | Optimal solution | May miss the optimum | Can reach it, but very slowly |

Adjustment Strategies

1. Fixed learning rate: use the same learning rate throughout training 2. Learning rate decay: gradually reduce it as training progresses 3. Adaptive learning rate: optimizers like Adam and AdamW adjust automatically based on gradients

Typical Learning Rate Settings

| Optimizer | Typical Learning Rate Range | |-----------|------------------------------| | SGD | 0.01 - 0.1 | | Adam | 0.0001 - 0.001 | | AdamW | 0.0001 - 0.001 |

--- Source: Easy AI Tutorial