Reproducing BBR: A Stanford Reproduction of Google's Congestion-Based TCP Algorithm

Background: The Congestion Control Problem

Traditional TCP congestion control algorithms rely on packet loss as the primary signal of network congestion. When packets are dropped, these algorithms assume the bottleneck is overflowing and sharply reduce the sending rate. CUBIC, the default Linux congestion control, exemplifies this loss-based approach: efficient in low-loss environments, but collapsing quickly when loss rates rise due to wireless interference, shallow buffers, or cellular variability.

In 2016, Google engineers introduced BBR (Bottleneck Bandwidth and Round-trip propagation time), a fundamentally different, congestion-based algorithm. Rather than waiting for loss, BBR continuously estimates the bottleneck bandwidth (BtlBw) and minimum round-trip propagation time (RTprop), aiming for the theoretically optimal operating point proven by Leonard Kleinrock in 1979: maximal throughput with minimal delay and loss. Google reported throughput improvements of 2x to 25x on its B4 wide-area network after deployment.

The Reproduction Effort: Why It Matters

Reproducibility is a cornerstone of credible research. Stanford's CS244 course has long encouraged students to reproduce seminal networking papers on a public blog. Luke Hsiao and Jervis Muindi chose to reproduce the original BBR paper's Figure 8, the most compelling demonstration that BBR achieves high utilization even under realistic packet loss. Their goals were to:

• Validate the original loss-vs-throughput claims.
• Examine the influence of experimental parameters (bandwidth, RTT, buffer size).
• Test against real-world cellular traces.
• Publish all code and instructions for community verification.

Experimental Setup

• OS/Kernel: Ubuntu 16.04 with Linux kernel 4.11.1 (native BBR support)
• Emulator: Mahimahi, for precise control over bottleneck bandwidth, RTT, and random loss
• Traffic: Custom Python TCP client/server
• Cloud: Google Compute Engine standard images
• TCP buffers: Expanded to 6.25 MB to support high bandwidth-delay product (BDP) flows
• Flow duration: 30–120 seconds depending on scenario

Key Findings

Reproducing Figure 8 (100 Mbps, 100 ms RTT)

• At extremely low loss rates, CUBIC slightly outperforms BBR, likely because BBR's initial implementation was not fully optimized (confirmed by original author Neal Cardwell).
• As loss increases, BBR stays close to the ideal throughput line (1 - loss_rate) * link_rate, maintaining high utilization until roughly 45% loss.
• CUBIC degrades far earlier, collapsing under even modest loss.
• Slight divergence from the original (BBR tolerating higher loss than the paper's 20%) is attributed to differences between Mahimahi's loss model and Linux netem.
• Expanding the TCP window was critical; without it, BBR was capped at ~80% of available bandwidth.

Multi-Algorithm Comparison

Adding BIC, RENO, VEGAS, and WESTWOOD to the mix, BBR maintained clear dominance under non-negligible loss. BIC (designed for high-bandwidth, high-latency paths) and WESTWOOD (designed for wireless loss) were the closest competitors, but still well behind BBR. Delay-based VEGAS was overly conservative; RENO was outdated.

Bandwidth Variation (0.01 – 100 Mbps)

• At very low bandwidths, loss-based algorithms tolerate higher loss rates because the tolerable loss rate scales inversely with the square of the BDP.
• At 10 Kbps, BBR and CUBIC performed similarly; across most realistic broadband scenarios, BBR's advantage was substantial.
• Implication: BBR's benefit is most pronounced on modern broadband, while narrowing on low-rate IoT links.

RTT Variation (2 – 500 ms)

• At 100 Mbps, lower RTT (smaller BDP) gives CUBIC slightly more tolerance for loss, though it still loses decisively to BBR.
• BBR maintained high utilization across all tested RTTs, with longer flows expected to converge even closer to the ideal curve.

Real-World LTE Validation

Using real Verizon LTE cellular traces in place of simulated loss/delay, BBR continued to outperform CUBIC by a wide margin, confirming that the laboratory result generalizes to noisy, variable cellular networks.

Why This Matters

This reproduction confirms that BBR's gains in lossy networks are real, reproducible, and robust across bandwidth, RTT, and loss regimes. Beyond validation, the project also surfaced useful implementation details: the importance of enlarged TCP buffers, the behavior of ProbeRTT state, and the subtle difference between loss models. By open-sourcing their scripts, Hsiao and Muindi lowered the barrier for future researchers to study congestion control empirically.

References

1. Cardwell, N., Cheng, Y., Gunn, C. S., Yeganeh, S. H., & Jacobson, V. (2017). *BBR: Congestion-Based Congestion Control*. Queue, 14(5), 20-53. 2. Hsiao, L., & Muindi, J. (2017). *ReBBR: Reproducing BBR Performance in Lossy Networks*. Stanford CS244 Reproducing Network Research Blog. 3. Reproducing Network Research Blog, Stanford University. https://reproducingnetworkresearch.wordpress.com/ 4. Nettles, J., et al. *Mahimahi Network Emulator*. http://mahimahi.mit.edu/ 5. Hsiao, L., & Muindi, J. (2017). *rebbr GitHub Repository*. https://github.com/jervisfm/rebbr