From 6edebe4864635f5c8b3d82a077339e7ee7d482e8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 23 Aug 2026 15:25:05 -0700 Subject: [PATCH] ops: forwarding through a downstream WireGuard terminator is free FortiOS has no WireGuard, so any WireGuard site-to-site has to terminate behind the edge, and ana-wg already fills that role. The earlier 767 Mbit/s figure was taken with traffic terminating on that box, which left the forwarding case unmeasured and overstated what a real deployment would see. Measured properly, transit from NH3 through the tunnel and onward to ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight, so the forwarding hop costs nothing and the full threefold gain over IPsec survives. The container uses around a fifth of four cores at that rate, against the UniFi gateway spending a third of its own four to move a third as much. The note records what still needs deciding rather than presenting this as ready to build: Anaheim hosts have to route to the terminator directly or pay a hairpin through the edge, which was not measured; the terminator introduces a failure mode the edge did not have, with a fallback route as the mitigation; and the NH3 end needs its own terminator, where a Linux host matches the measured figure while the existing UniFi WireGuard server would likely land lower. The test tunnel, its peer and the temporary route were all removed. --- ...2026-08-23-anaheim-ipsec-tunnel-ceiling.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index 514a472..b0a580a 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -444,3 +444,60 @@ UDM network object verified **byte-identical** to its pre-test snapshot left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer offers only aes256 so the extra entries are inert, but **narrowing it back to `aes256-sha1` is one line** if the looser list is unwanted. + +--- + +## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through + +Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must +terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on +pfi-pve, 10.250.50.252) already is that box. + +**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with +traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other +Anaheim hosts, which was never measured. Now it is: + +| topology | 1 stream | 8 streams | +|---|---|---| +| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s | +| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s | +| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** | + +**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM +architecture delivers the full 3.1× single-stream for real transit traffic, not +just for traffic landing on the tunnel box. + +ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle), +so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's* +load, not the container's — do not read it as ana-wg's own. For contrast the +UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom. + +### Design consequences of terminating downstream — the parts that need decisions + +1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8 + figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252` + route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default + gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a + LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT + measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated + transit VLAN for ana-wg, or accept the hairpin. +2. **New single point of failure.** Today site-to-site dies only when the edge + dies, which is total anyway. A downstream terminator fails independently. + Mitigation: keep the IPsec tunnel configured as a higher-metric fallback + route so it takes over when ana-wg is down. +3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the + hypervisor shared with the rest of the Anaheim VMs. +4. **The NH3 end needs a terminator too**, and there are two shapes: + - **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what + was measured: **764 Mbit/s**. + - **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on + UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and + UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on + Site Magic), so expect ~557 rather than 764. Still 2.3× today. + +### Standing recommendation + +Worth doing, but it is **a project, not a config tweak** — routing, failover and +policy all need deciding. The cheap wins remain available meanwhile and are +independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending +the FortiGate work settling) and AES-128 for ~20%.