ops(ana-gw): pin down what limits a single stream across the tunnel

The earlier note described the constraint as per-flow serialisation without evidence. Pinning SO_SNDBUF and sweeping it shows what it actually is: throughput holds flat between 224 and 247 Mbit/s across a thirteenfold range of in-flight data while round-trip time scales with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions are absent throughout. That is a fixed service rate with a standing queue ahead of it, so socket buffer, window scaling and congestion control are not the lever and should not be pursued. The same measurements surface something with wider consequences than throughput: a single bulk stream lifts tunnel latency from 6.9 ms to 102 ms average, 136 ms peak, with no loss. Interactive traffic sharing the Anaheim link degrades sharply whenever anything moves bulk data, and parallelising transfers makes that worse rather than better. Localisation rests on the FortiGate appearing in both slow paths and in neither fast one, with aggregate throughput over the same security association reaching 692 Mbit/s. Per-session NPU offload fits the shape but was not confirmed on the device; the note records the test that would settle it and flags the bearing this has on the pending FortiGate replacement.
2026-08-23 14:42:17 -07:00
parent e41d19f1cb
commit cf0cb2cbb3
1 changed files with 73 additions and 0 deletions
@@ -155,3 +155,76 @@ FortiGate would not accept — a short tunnel outage until `aes256` was restored
 config enum this way, restore the known-good value after **every** 200, not at
 the end of the loop. Post-change verification: the UDM object was diffed
 field-by-field against its pre-change snapshot and is **byte-identical**.
+
+---
+
+## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
+
+The correction above called the constraint "per-SA/per-flow serialisation".
+That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
+~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
+problem, not loss, not window size.
+
+### The evidence: pin the send buffer and sweep it
+
+Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
+
+| in-flight cap | throughput | RTT in flight | minRTT | retrans |
+|---|---|---|---|---|
+| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
+| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
+| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
+| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
+
+**Throughput is flat across a 13× range of in-flight data while RTT scales with
+it.** That is the signature of a fixed service rate with a standing queue: the
+window controls only how much queue you build, never how fast you go. Had this
+been window-limited, throughput would have risen with the buffer. Had it been
+congestion, there would be retransmits — there are essentially none
+(`retrans:0`, 0% ping loss).
+
+So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
+**red herrings here**. Do not go there.
+
+### Bufferbloat: one bulk stream wrecks latency for everything else
+
+Measured on the same tunnel, ping to ana-docker:
+
+- idle: **6.9 ms** avg
+- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
+
+**15× latency inflation from one transfer.** This is the operationally
+important finding — any interactive traffic sharing the Anaheim link (ssh,
+RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
+takes only one stream to do it. Parallelising transfers makes throughput
+better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
+tunnel (or rate-limiting bulk jobs), not more buffer.
+
+### Where the cap lives — strong inference, not proof
+
+Three paths, and the FortiGate is the only variable:
+
+| path | single-stream |
+|---|---|
+| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
+| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
+| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
+
+Present in both slow paths, absent from the fast one. Aggregate over the same
+SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
+many flows spread out fine, one flow does not.
+
+The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
+each firewall session is bound to one crypto engine, so a single TCP flow is
+capped at one engine's rate while many sessions spread across engines. **This
+is inference from the throughput shape, not something confirmed on the box** —
+`diagnose sys session list` was not captured for a TCP flow (the filter caught
+only traceroute UDP probes). A single-stream control through ana-gw *without*
+IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
+window-limited and does not discriminate. **If this matters, the clean test is
+a non-IPsec single stream between two Anaheim VLANs at low RTT.**
+
+**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
+FortiGate's IPsec path, replacing the box plausibly lifts single-stream
+throughput toward the WireGuard figure. That is a point in favour of the
+cutover, and it is cheap to verify afterwards by re-running the sweep.