QUIC’s BBR congestion control algorithm can achieve significantly higher throughput than Cubic on high-bandwidth, high-latency networks, but it can also lead to increased bufferbloat and latency under certain conditions.
Let’s see BBR in action. Imagine you’re transferring a large file over a network with a good amount of bandwidth but also a noticeable delay.
# Simulate a file transfer with QUIC (using a hypothetical tool)
quic_transfer --server server.example.com --file large_dataset.zip --congestion_control bbr
The transfer starts. BBR, unlike Cubic, doesn’t aggressively probe for bandwidth by filling up buffers. Instead, it tries to estimate the actual available bandwidth and the round-trip time (RTT) of the connection. It maintains a "bandwidth product" (BDP), which is essentially the amount of data that can be in flight at any given time. BBR aims to keep the buffer occupancy roughly equal to this BDP.
Here’s how it works internally:
- Bandwidth Estimation: BBR constantly probes for the maximum bandwidth it can sustain. It does this by observing how much data is acknowledged within a given time window.
- RTT Estimation: It also tracks the minimum RTT observed over a period, assuming this represents the "idle" RTT of the connection.
- Pacing: Based on these estimates, BBR paces outgoing packets. It tries to send packets at a rate that precisely matches the estimated bandwidth, avoiding unnecessary buffer filling.
- Operating Modes: BBR cycles through different modes:
- Startup: Aggressively probes for bandwidth to quickly reach the estimated BDP.
- Drain: Once the estimated BDP is reached, it reduces sending rate to match the estimated bandwidth, aiming to drain the buffer down to the BDP.
- Cruise: Maintains the estimated bandwidth and RTT, sending data at a controlled pace.
- Reordering: Handles packet reordering.
Cubic, on the other hand, uses a cubic function to increase its congestion window after packet loss. It’s designed to be more aggressive in bandwidth probing and less sensitive to buffer occupancy, which can be good on lossy networks but less efficient when buffers are large and latency is high.
The key levers you control are primarily at the operating system level, influencing BBR’s behavior:
-
net.core.default_qdisc: This sets the default queueing discipline. For BBR to be effective, you generally wantfq(fair queueing) orfq_codelas the underlying qdisc.sudo sysctl -w net.core.default_qdisc=fqThis ensures that packets are fairly distributed among flows and helps manage buffer occupancy.
-
net.ipv4.tcp_congestion_control: This is where you explicitly enable BBR.sudo sysctl -w net.ipv4.tcp_congestion_control=bbrThis tells the kernel to use the BBR algorithm for TCP and, by extension, for QUIC connections that leverage the kernel’s congestion control stack.
-
BBR specific tuning (less common, but powerful):
-
net.ipv4.tcp_bbr_min_rtt: Sets a minimum RTT threshold.sudo sysctl -w net.ipv4.tcp_bbr_min_rtt=10000 # Example: 10ms in microsecondsThis prevents BBR from using an RTT estimate that’s unrealistically low, which could happen due to short-lived connections or network anomalies.
-
net.ipv4.tcp_bbr_pacing_gain: Controls how aggressively BBR paces packets. A value of2(the default) means it tries to pace packets to fill the BDP. Higher values can increase throughput but also buffer occupancy.sudo sysctl -w net.ipv4.tcp_bbr_pacing_gain=2This parameter is crucial for BBR’s ability to smooth out traffic and avoid sudden bursts that can overwhelm buffers.
-
net.ipv4.tcp_bbr_cwnd_gain: Controls how aggressively BBR probes for bandwidth. A value of2(the default) allows the congestion window to grow up to twice the BDP during the startup phase.sudo sysctl -w net.ipv4.tcp_bbr_cwnd_gain=2This parameter balances the need for rapid bandwidth discovery against the risk of overshooting the actual available capacity.
-
When BBR is enabled and configured correctly, you’ll observe significantly higher throughput on connections with high bandwidth-delay products. Instead of the aggressive, sawtooth-like window growth of Cubic, BBR’s window will appear to grow more smoothly, often staying consistently higher.
A common pitfall is enabling BBR without also ensuring the underlying queueing discipline is set to fq or fq_codel. Without fair queueing, BBR’s smooth pacing can be undermined by other flows or misbehaving applications, leading to unpredictable latency.
The next logical step after optimizing BBR is to understand how its packet pacing interacts with network devices that have their own rate limiting or shaping mechanisms.