diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index 4be13b7..8c1b1a3 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -155,3 +155,76 @@ FortiGate would not accept — a short tunnel outage until `aes256` was restored config enum this way, restore the known-good value after **every** 200, not at the end of the loop. Post-change verification: the UDM object was diffed field-by-field against its pre-change snapshot and is **byte-identical**. + +--- + +## FOLLOW-UP (2026-08-23): what the per-stream limit actually is + +The correction above called the constraint "per-SA/per-flow serialisation". +That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of +~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning +problem, not loss, not window size. + +### The evidence: pin the send buffer and sweep it + +Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight: + +| in-flight cap | throughput | RTT in flight | minRTT | retrans | +|---|---|---|---|---| +| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 | +| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 | +| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 | +| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 | + +**Throughput is flat across a 13× range of in-flight data while RTT scales with +it.** That is the signature of a fixed service rate with a standing queue: the +window controls only how much queue you build, never how fast you go. Had this +been window-limited, throughput would have risen with the buffer. Had it been +congestion, there would be retransmits — there are essentially none +(`retrans:0`, 0% ping loss). + +So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all +**red herrings here**. Do not go there. + +### Bufferbloat: one bulk stream wrecks latency for everything else + +Measured on the same tunnel, ping to ana-docker: + +- idle: **6.9 ms** avg +- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss + +**15× latency inflation from one transfer.** This is the operationally +important finding — any interactive traffic sharing the Anaheim link (ssh, +RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it +takes only one stream to do it. Parallelising transfers makes throughput +better and this *worse*. If it starts biting, the fix is an AQM/shaper on the +tunnel (or rate-limiting bulk jobs), not more buffer. + +### Where the cap lives — strong inference, not proof + +Three paths, and the FortiGate is the only variable: + +| path | single-stream | +|---|---| +| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s | +| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s | +| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s | + +Present in both slow paths, absent from the fast one. Aggregate over the same +SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole — +many flows spread out fine, one flow does not. + +The mechanism that fits is **FortiGate NPU IPsec offload being per-session**: +each firewall session is bound to one crypto engine, so a single TCP flow is +capped at one engine's rate while many sessions spread across engines. **This +is inference from the throughput shape, not something confirmed on the box** — +`diagnose sys session list` was not captured for a TCP flow (the filter caught +only traceroute UDP probes). A single-stream control through ana-gw *without* +IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is +window-limited and does not discriminate. **If this matters, the clean test is +a non-IPsec single stream between two Anaheim VLANs at low RTT.** + +**Relevant to the FortiGate cutover decision:** if the per-flow cap is the +FortiGate's IPsec path, replacing the box plausibly lifts single-stream +throughput toward the WireGuard figure. That is a point in favour of the +cutover, and it is cheap to verify afterwards by re-running the sweep.