diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index a198a5d..4be13b7 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -1,5 +1,14 @@ # [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit +> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at +> the bottom before acting on anything here.** The headline is wrong (the +> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the +> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the +> original stopped measuring at 4), and the proposed remedy is **impossible**: +> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The +> per-stream observation and the parallelise-your-transfers mitigation are the +> parts that survive. + The operator noticed site-to-site transfers were slow for a datacenter fiber handoff and asked whether WireGuard was the limit. It is not WireGuard, and the circuit is fine. @@ -69,3 +78,80 @@ on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list` prints live ESP session keys** — never paste its output into althing, a booth, or a commit. + +--- + +## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist + +The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done, +and the measurements taken while trying show there is very little left to win. + +### AES-GCM is unavailable on the far end — not a naming problem + +The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf +`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the +crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`, +`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128` +— all `HTTP 400 api.err.InvalidPayload`, nothing applied. + +**The control that makes this conclusive:** the *identical* request body with +`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the +enum rejecting the value, not a malformed body. Corroborating: **zero +case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.** + +Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`, +`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on +UniFi far ends, so this blocks the ESH tunnel too. + +The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now +reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is +functionally identical while the peer only offers CBC, and reverting it would +cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched; +IKE protects the control channel only and has no bearing on data throughput. + +### The numbers that retire this as a problem + +Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on +the same UDM and the same uplink, with the same stdlib TCP probe: + +| streams | IPsec NH3→ANA | WireGuard NH3→ESH | +|---|---|---| +| 1 | 245 Mbit/s | 557 Mbit/s | +| 4 | 471 Mbit/s | 767 Mbit/s | +| 8 | **692 Mbit/s** | **795 Mbit/s** | + +**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) — +that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel. +So the tunnel does **~69% of the achievable uplink** at 8 streams, and the +IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**. + +Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since +FortiOS has no WireGuard) would chase that last 15%. Not worth it. + +### What the constraint actually is + +A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both +endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03` +(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The +shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from +a higher floor (557 → 795 is only 1.43× scaling). + +### Actionable consequence + +Anything moving bulk data across this link should **parallelise** — 245 → 692 +Mbit/s, a 2.8× win with no config change. For single-stream workloads that +cannot be parallelised at the application layer, **NFS `nconnect=N` is the +lever**: it opens N TCP connections per mount, converting a single-stream +workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at +24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live +example — remounting with `nconnect=8` is the obvious test. + +### Foot-gun recorded + +Probing the enum by PUTting candidate values **applies the accepted ones**. A +probe loop here timed out with `3des` briefly live on the NH3 side, which the +FortiGate would not accept — a short tunnel outage until `aes256` was restored +(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi +config enum this way, restore the known-good value after **every** 200, not at +the end of the loop. Post-change verification: the UDM object was diffed +field-by-field against its pre-change snapshot and is **byte-identical**. diff --git a/persistent-memory.md b/persistent-memory.md index 0d3b8b5..bef76cc 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -108,11 +108,11 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-08-23 — a long multi-party ops session; open with the operator are the Anaheim tunnel cipher and how long WAN admin on ana-gw stays open._ +_As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._ - **🔓 ana-gw WAN admin is OPEN again as of 2026-08-23 — deliberately, and it should be re-closed after the FortiGate cutover.** Operator asked for it as the cutover contingency ("so I don't have to drive down there"). `https://ana-fw.phasefinal.com/` (wan1's own IP 38.120.12.42) serves the admin GUI on a browser-trusted LE cert; **https only**, and only the `infra-ops` account is WAN-reachable (`admin` remains 10.0.0.0/8-locked). Verified end-to-end with a real login from two sites. This partially reverses the 2026-08-12 hardening on a box still running the **EOL, actively-exploited FortiOS 7.2 branch** — so the exposure is CVE-shaped, and time-boxing it is the mitigation. Re-close = `config system interface / edit wan1 / set allowaccess ping / next / end`. Config backups pre+post at `/var/tmp/ana-gw-config-20260823T20*.conf` on nh3-dev. **Two follow-ons:** (a) ACME renewal for that cert needs **port 80** on wan1 — currently absent, next attempt ≈2026-09-27, expiry 2026-10-27; (b) ~5 SYN/s from `179.51.184.0/21` now draws SYN-ACKs, CPU impact nil, unmitigated by choice. Detail + the FortiOS trusthost gotchas: auto-memory `reference_fortigate_ana_gw_access`. -- **⏳ THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit.** Circuit measured at **2,153 Mbit/s**; the tunnel ceilings ~550 Mbit/s aggregate, ~230 single-stream. FortiGate CPU **idle**, IPsec **NPU-offloaded**, interface error-free — so it is not crypto exhaustion. Both tunnels negotiate **`aes256-sha1`**; **AES-GCM is the proposed change** and the operator has signalled he will authorize it. Not executed: production edge, needs a matching change at NH3 + ESH, each tunnel drops during renegotiation. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` +- **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB).