Easy AI Tutorial: Understanding Loss Functions
*Part of the Easy AI tutorial series on zhichai.net. Loss is the core optimization objective in machine learning model training.*
What Is a Loss Function?
- Loss is a single number that scores "how wrong" the model is during fine-tuning.
- It is a differentiable mathematical function that quantifies the error between model predictions and ground-truth labels.
- It outputs a scalar value; larger values mean bigger mistakes.
- Serves as the optimization target
- Guides the direction of parameter updates
- Evaluates training progress and decides when to stop training
- Learning rate too large: may overshoot the optimum.
- Learning rate too small: convergence is slow.
- A suitable learning rate: stable, fast convergence — it needs tuning per problem.
- MSE: (prediction - truth)²
- MAE: |prediction - truth|
- Huber Loss: a smooth approximation of absolute error
- Loss quantifies prediction error as a differentiable scalar.
- Gradient descent reduces loss by iteratively updating parameters against the gradient, scaled by the learning rate.
- Compare training vs. validation loss to detect overfitting.
- Continuous iteration toward minimizing error is the ultimate goal of model training.
How Is Loss Calculated?
1. The model makes predictions on the data. 2. Predictions are compared against the correct labels. 3. A mathematical formula computes the difference — common formulas include MSE (Mean Squared Error), MAE (Mean Absolute Error), Huber Loss, and Cross-Entropy.
Why Does It Matter for Fine-Tuning?
The core goal of fine-tuning is to make the model make fewer mistakes, i.e., drive the loss lower:
How Is Loss Reduced?
Via the gradient descent algorithm:
1. Compute the gradient of the loss with respect to the parameters. 2. Update parameters in the opposite direction of the gradient. 3. Use a learning rate to control the update step size. 4. Repeat until convergence.
Gradient Descent Animation
The tutorial includes an interactive animation using a simple quadratic loss function: (x - 0.3)^2 + 0.1. It visualizes how the algorithm iteratively finds the minimum, with playback controls, adjustable learning rate, and live status (iteration count, current position, current gradient).
Key insights:
Interactive Loss Function Demo
An interactive component lets users adjust the prediction value and true value and observe how the loss changes in real time, comparing:
Training Loss Curve Visualization
A simulated training-curve component demonstrates several scenarios:
| Scenario | Behavior | |---|---| | Normal training | Loss decreases steadily — training is working well | | Overfitting | Training loss falls but validation loss rises | | Unstable training | Loss fluctuates sharply — learning rate may be too large | | Learning stagnation | Loss drops then flattens out (convergence) |
How to read the chart: the x-axis is training epochs; downward curves mean the model is improving; flat regions indicate convergence. Ideally, loss decreases steadily, without sharp oscillation, and eventually plateaus.
Common Misconceptions
1. A pretty loss curve ≠ a good model. Stable decline only means training proceeded normally, not that the model performs well on real data. Training loss small but validation loss large is overfitting. 2. Don't fixate on absolute loss values. Loss ranges differ greatly across tasks (classification loss is often 0–10; regression can be 0–1000). Watch trends, not absolute numbers. 3. Fluctuation is not failure. Moderate loss fluctuation is normal, especially early in training, due to batch randomness, learning-rate adjustments, or data augmentation. The overall trend matters. 4. Smaller loss is not always better. Over-pursuing minimal loss can cause overfitting — overly long training and excessive model complexity lead to memorizing noise. Use early stopping, regularization, and balance training fit against generalization.