Summary
Deep learning training can fail silently: hardware faults, compiler bugs, or silent data corruption can produce corrupted models without any crash or error message. TrainCheck, presented at OSDI 2025, addresses this by automatically inferring invariants from DL training code and actively checking them during training. The core idea is that certain quantities in training should maintain specific statistical properties—for example, the variance of gradient norms should not change abruptly, and the output distribution of a layer should match historical patterns. TrainCheck learns these invariants automatically from the training code and validates them at every training iteration. In evaluation, TrainCheck detected 18 out of 20 real-world silent errors that were manually reproduced, within a single training iteration, and it uncovered 6 previously unknown bugs in popular training libraries. The key insight is that the inherent mathematical structure of deep learning training itself provides a rich set of 'health check' signals—researchers just need to monitor them. This makes TrainCheck a practical tool for improving the reliability and trustworthiness of large-scale model training pipelines.
Deep learning training can silently produce wrong results due to hardware faults, compiler bugs, or silent data corruption — no crash, no error message, the training keeps running, but the resulting model is broken. These are called silent errors.
TrainCheck (OSDI 2025) automatically infers invariants of DL training and actively checks them during the training process.
Core idea: certain quantities during training "should" maintain specific statistical properties — for example:
- The variance of gradient norms should not change abruptly.
- The output distribution of a layer should follow its historical pattern.
TrainCheck learns these invariants automatically from the training code and checks them at every training iteration.
Results:
- On 20 manually reproduced real-world silent errors, TrainCheck detected 18 within a single training iteration.
- It also discovered 6 previously unknown bugs in training libraries.
Key insight: the intrinsic mathematical structure of DL training itself provides a set of "health check" signals — you just have to pay attention to them.
Source: [Training with Confidence / OSDI 2025]
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619976