How to Diagnose and Fix a CUBIC Congestion Window Stuck Bug in QUIC

By ✦ min read

Introduction

When implementing congestion control for QUIC, you might encounter a bizarre bug where the CUBIC algorithm's congestion window (cwnd) gets permanently pinned at its minimum value after a congestion collapse, never recovering. This guide walks through diagnosing and fixing that exact issue, which appeared in Cloudflare's open-source QUIC implementation (quiche) after porting a Linux kernel optimization. By following these steps, you'll understand the root cause—a subtle interaction with app-limited exclusion logic—and apply an elegant one-line fix to break the cycle.

What You Need

• CUBIC knowledge: Basic understanding of RFC 9438 and how CUBIC adjusts cwnd based on loss and available bandwidth.
• Source code access: Either the Linux kernel's CUBIC implementation or your own QUIC stack (e.g., Cloudflare quiche).
• Test infrastructure: A simulation that introduces heavy packet loss early in a connection.
• Debugging tools: Logging to trace cwnd changes over time.
• Familiarity with congestion control states: Know terms like app-limited, congestion avoidance, and recovery.

Step 1: Understand CUBIC's Core Logic

CUBIC, defined in RFC 9438, is a loss-based congestion control algorithm. It increases the congestion window (cwnd) aggressively when no loss is detected and cuts it drastically on packet loss. The key assumption: no loss means bandwidth is available; loss means the network is saturated.

Inside CUBIC, the cwnd is the number of bytes the sender can keep in flight. A larger cwnd increases throughput; a smaller one throttles it. After a loss event, cwnd is reduced to a minimum value, and CUBIC begins probing for more bandwidth. But the algorithm also includes an app-limited exclusion: if the sender has no data to send (app-limited), it should not count that idle time as available bandwidth. This optimization prevents unnecessarily growing cwnd during pauses.

Step 2: Identify the Symptom — Test Failure Rate

The bug first surfaced as a failing integration test. In a scenario with heavy early loss, CUBIC's cwnd dropped to its minimum and never recovered. The test failed 61% of the time, showing the bug was reproducible but not deterministic. If your tests show erratic recovery after congestion collapse, suspect this bug.

Step 3: Reproduce the Bug in a Controlled Environment

Set up a QUIC connection simulation with CUBIC as the congestion controller. Induce early severe packet loss (e.g., 50% drop rate) for the first few round trips. After the loss subsides, monitor cwnd. In a correct implementation, cwnd should gradually increase. Here, you'll see it stuck at its minimum (typically 2-4 packets).

Use detailed logging to capture cwnd every RTT. Compare with a working scenario (e.g., without early loss) to confirm the anomaly.

Step 4: Investigate the Root Cause

The bug originated from a Linux kernel change that strictified the app-limited exclusion in CUBIC (RFC 9438 §4.2-12). When ported to quiche, the logic inadvertently kept the connection marked as app-limited even after data became available again. This prevented CUBIC from ever increasing cwnd after recovery.

Trace the code flow:

• After loss, cwnd drops to minimum.
• The app-limited check sees that the sender was idle (no data pending) during the loss event.
• It sets a flag that persists across states.
• Even when new data arrives, that flag blocks cwnd growth because CUBIC assumes the sender is still app-limited.

In Linux TCP, the same optimization works because the stack clears the flag appropriately; in QUIC's event-driven model, the flag lingered.

Step 5: Apply the One-Line Fix

The fix is elegantly simple: after congestion recovery, reset the app-limited flag. In quiche, this meant adding a line in the on_recovery_end callback:

1. Locate the function where CUBIC handles exiting recovery.
2. Insert code to clear the app-limited state (e.g., self.is_app_limited = false;).
3. Ensure this occurs before any cwnd growth calculations.

This breaks the cycle: now after a congestion collapse, once recovery finishes, CUBIC treats the connection as active and starts probing for available bandwidth.

Step 6: Verify the Fix

Re-run the same test with early heavy loss. The cwnd should now recover normally. Run a suite of congestion control tests—steady-state, growth, and edge cases—to ensure no regression. In Cloudflare's case, the fix reduced the test failure rate from 61% to 0%.

Tips

• Test edge cases: Minimum cwnd scenarios are often neglected. Include them in your test plan.
• Understand RFC 9438: The app-limited exclusion is beneficial for TCP but needs careful adaptation for QUIC's connection model.
• Use structured logging: Track cwnd, recovery state, and app-limited flags per event to spot anomalies.
• Collaborate with kernel developers: Similar bugs may appear when syncing TCP and QUIC congestion control logic.
• One-line fixes are powerful: Simple state resets can fix subtle bugs without major refactoring.

By following these steps, you can identify, reproduce, and fix the CUBIC congestion window stuck bug in your own QUIC implementation—ensuring your connections recover robustly from even the worst network events.