Optical flow is a foundational problem in computer vision: estimating the displacement of pixels or features across an image sequence to capture scene motion. Since the classic Lucas-Kanade and Horn-Schunck methods, optical flow algorithms have evolved significantly in accuracy, robustness, and scope. This article systematically reviews that evolution and benchmarks leading algorithms on MPI Sintel, KITTI, and Spring.
Classic Optical Flow Methods
The core of optical flow is solving for pixel displacement under the brightness constancy assumption. Since this yields one equation with two unknowns per pixel, the problem is ill-posed and requires additional constraints.
- Lucas-Kanade (LK): A local method assuming all pixels in a small window share the same flow, solved via least squares. It requires sufficient texture (gradients) in the window—the aperture problem—and only handles small displacements unless combined with pyramids. It is simple, fast, but sensitive to weak texture and illumination changes.
- Horn-Schunck (HS): A global method minimizing an energy function combining a brightness-constancy data term and a smoothness term, producing dense flow fields. It mitigates the aperture problem but blurs motion boundaries and requires iteratively solving large linear systems.
- Multi-scale / coarse-to-fine: Image pyramids estimate large displacements at coarse scales, then warp and refine at finer scales. This strategy is now a standard component of both classical and deep-learning approaches.
- FlowNet / FlowNet 2.0: The first end-to-end CNN approaches, using an encoder-decoder architecture with a correlation layer for matching cost. FlowNet 2.0 stacked modules and improved training to approach or exceed traditional methods.
- PWC-Net: Combined classical ideas with deep learning: a pyramidal coarse-to-fine scheme, warping the second frame with current flow, computing a feature cost volume, and regressing flow residuals via CNN.
- RAFT (Recurrent All-Pairs Field Transforms, ECCV 2020): A milestone architecture featuring:
- All-pairs correlations: a full 4D correlation volume rather than local-window matching.
- GRU-based iterative updates of the flow field, learned analogues of classical optimization.
- Multi-scale context during iteration.
- Video stabilization and motion compensation: dense pixel-level motion estimation for smoothing camera shake and improving video compression.
- Object tracking and motion detection: trajectory estimation, crowd behavior analysis, and motion-region detection for surveillance.
- 3D reconstruction and SfM: flow as a matching cue, especially in texture-poor scenes; joint depth-and-motion recovery from monocular video.
- Video segmentation and action recognition: propagating segmentation masks over time; flow as complementary motion features alongside RGB frames.
- Autonomous driving and robotics: moving-object detection, scene flow for 3D motion understanding, visual odometry, and SLAM support.
Deep Learning Evolution: FlowNet to RAFT
RAFT achieved state-of-the-art EPE on MPI Sintel and KITTI and spawned many follow-ups (GMA, SKFlow, FlowFormer with Transformers, FlowDiffuser with diffusion models).
Benchmark Performance
MPI Sintel
Derived from the open movie *Sintel*, with Clean and Final versions featuring large motions, illumination changes, motion blur, and occlusions. As of 2025, the top ten methods are nearly all deep learning models. On the Final benchmark: FreeFlow-L leads with 1.480 average EPE, followed by ViCo_VideoFlow_MOF (1.618) and CFFlow (1.647). Most top methods build on RAFT-style architectures. Remaining challenges include textureless regions and severe illumination changes.KITTI
Real driving scenes with LiDAR-derived ground truth covering only foreground objects; evaluated by Fl-all (percentage of pixels with error >3px or 5%). As of 2025, SEA-Flow3D + Monster leads at 2.53% Fl-all, with MS-RAFT-3D+ and ARFlow at 2.85%. Multi-frame methods (e.g., VideoFlow, MEMFOF) and scene-flow approaches (e.g., RAFT-3D) that jointly estimate flow, depth, and camera motion perform best. Future work may integrate stereo and SLAM information.Spring
A high-resolution benchmark targeting 8K input. DPFlow uses a dual-pyramid framework to adapt models trained on low-resolution data to high-resolution inputs, achieving state-of-the-art results on Spring as well as Sintel and KITTI. FlowSeek leverages deep foundation models for strong performance with minimal data resources.Typical Applications
Outlook
Optical flow has leapt from classical theory to deep learning, with RAFT-family models setting unprecedented accuracy on major benchmarks. Future research will focus on accuracy, robustness, and efficiency: stability under extreme illumination, occlusion, and motion; cross-task joint learning (flow + depth + segmentation); few-shot/unsupervised learning; and lightweight real-time models for mobile and embedded deployment. As a foundational technology for understanding dynamic scenes, optical flow will continue to drive advances in video understanding, autonomous driving, and robotics.