diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index e137c08..9faf8aa 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -293,3 +293,81 @@ single-stream. If it also caps ~250, the FortiGate is exonerated and replacing it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM WireGuard achieves, the 80F is the limiter. **Bears directly on the pending FortiGate cutover** — worth running before that decision, not after. + +--- + +## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream + +Operator asked for a WireGuard test from `ana-wg` to NH3 over the public +internet. It is the test that separates the *path* from the *crypto*, and the +answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the +transport is not worth it" — that conclusion compared 8-stream numbers and was +wrong for single-stream workloads.** + +### Setup (fully torn down afterwards) + +`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an +internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by +FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337, +policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A +temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on +wg0, measured, then removed — ana-wg is back to its original 3 peers and the +keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in +place** (benign, and wanted if this becomes permanent). + +In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM +only NAT/forward UDP, and Linux does WireGuard at both ends. + +### The full comparison + +| path | crypto performed by | 1 stream | 8 streams | +|---|---|---|---| +| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s | +| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s | +| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s | +| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s | +| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s | + +**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is +no per-flow penalty at all, and a single flow already saturates the path. So +the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3 +uplink and not the physical path — all of which sustain 767 on one flow. + +Per-flow penalty ranks by implementation: + +- **Linux WireGuard — none** (767 → 763, flat) +- **UDM WireGuard — mild**, ~1.4× (557 → 795) +- **IPsec on this pair — severe**, ~2.8× (245 → 692) + +### Latency under load — the same story + +| path | idle | during ONE bulk stream | +|---|---|---| +| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max | +| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max | + +WireGuard carries **3.1× the single-stream throughput with 8× less latency +inflation** on the same wire. + +### Attribution — still not fully separated, and it no longer matters much + +Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still +does not isolate which one imposes the 2.8× penalty. Closing that would need +Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical +decision no longer depends on the answer**, because the fix is the same either +way and it is already demonstrated. + +### Recommendation (supersedes FOLLOW-UP 1) + +A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is +worth real consideration: 3.1× single-stream, flat scaling, far better latency +under load, and it reuses infrastructure that already exists and is already +internet-reachable. It is also the architecture already proven for NH3↔ESH. +Open questions before committing: routing/failover if ana-wg (an LXC) is down, +whether it replaces or parallels the IPsec tunnel, and firewall policy for the +new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it +has headroom. + +**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the +same lever (turn one flow into many) and brokkr-smithy-dev has given standing +approval to apply it once the FortiGate work settles, with no need to ask again.