ops: forwarding through a downstream WireGuard terminator is free

FortiOS has no WireGuard, so any WireGuard site-to-site has to
terminate behind the edge, and ana-wg already fills that role. The
earlier 767 Mbit/s figure was taken with traffic terminating on that
box, which left the forwarding case unmeasured and overstated what a
real deployment would see.

Measured properly, transit from NH3 through the tunnel and onward to
ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight,
so the forwarding hop costs nothing and the full threefold gain over
IPsec survives. The container uses around a fifth of four cores at
that rate, against the UniFi gateway spending a third of its own four
to move a third as much.

The note records what still needs deciding rather than presenting this
as ready to build: Anaheim hosts have to route to the terminator
directly or pay a hairpin through the edge, which was not measured;
the terminator introduces a failure mode the edge did not have, with a
fallback route as the mitigation; and the NH3 end needs its own
terminator, where a Linux host matches the measured figure while the
existing UniFi WireGuard server would likely land lower.

The test tunnel, its peer and the temporary route were all removed.

persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md

@@ -444,3 +444,60 @@ UDM network object verified **byte-identical** to its pre-test snapshot
 left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
 offers only aes256 so the extra entries are inert, but **narrowing it back to
 `aes256-sha1` is one line** if the looser list is unwanted.
+
+---
+
+## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
+
+Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
+terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
+pfi-pve, 10.250.50.252) already is that box.
+
+**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
+traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
+Anaheim hosts, which was never measured. Now it is:
+
+| topology | 1 stream | 8 streams |
+|---|---|---|
+| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
+| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
+| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
+
+**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
+architecture delivers the full 3.1× single-stream for real transit traffic, not
+just for traffic landing on the tunnel box.
+
+ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
+so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
+load, not the container's — do not read it as ana-wg's own. For contrast the
+UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
+
+### Design consequences of terminating downstream — the parts that need decisions
+
+1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
+   figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
+   route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
+   gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
+   LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
+   measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
+   transit VLAN for ana-wg, or accept the hairpin.
+2. **New single point of failure.** Today site-to-site dies only when the edge
+   dies, which is total anyway. A downstream terminator fails independently.
+   Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
+   route so it takes over when ana-wg is down.
+3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
+   hypervisor shared with the rest of the Anaheim VMs.
+4. **The NH3 end needs a terminator too**, and there are two shapes:
+   - **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
+     was measured: **764 Mbit/s**.
+   - **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
+     UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
+     UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
+     Site Magic), so expect ~557 rather than 764. Still 2.3× today.
+
+### Standing recommendation
+
+Worth doing, but it is **a project, not a config tweak** — routing, failover and
+policy all need deciding. The cheap wins remain available meanwhile and are
+independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
+the FortiGate work settling) and AES-128 for ~20%.