ops(ana-gw): close out the Anaheim tunnel cipher question

The AES-GCM cutover was authorised and attempted, NH3 side first. It cannot be completed: UniFi's manual site-to-site IPsec implements no AEAD cipher. Eight GCM spellings were rejected with api.err.InvalidPayload while an otherwise identical body carrying aes256 returned rc:ok, which isolates the enum rather than the request as the cause. The accepted set is aes128/aes192/aes256/3des. Both Anaheim tunnels terminate on UniFi gateways, so this blocks the ESH tunnel on the same grounds. Measuring while testing also retires the premise. NH3's uplink is a 1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather than the ~550 previously recorded from a 4-stream run. Compared with WireGuard over the same gateway and uplink, the deficit narrows from 2.3x at one stream to 15% at eight, so moving this link onto a different transport is not worth doing. The constraint is per-stream, around 245 Mbit/s, with both endpoints idle under load. Parallelising bulk transfers remains the mitigation and is worth 2.8x at no cost; NFS nconnect is the equivalent lever for single-stream mounts such as /mnt/smithy on ana-ml2. FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list, which is inert while the peer offers only CBC and avoids a further renegotiation. The UDM network object was diffed field-by-field against its pre-change snapshot and is unchanged.
2026-08-23 14:29:17 -07:00
parent 5af362e9d0
commit e41d19f1cb
2 changed files with 88 additions and 2 deletions
@@ -1,5 +1,14 @@
 # [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
 > **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
 > the bottom before acting on anything here.** The headline is wrong (the
 > relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
 > aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
 > original stopped measuring at 4), and the proposed remedy is **impossible**:
 > UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
 > per-stream observation and the parallelise-your-transfers mitigation are the
 > parts that survive.
 The operator noticed site-to-site transfers were slow for a datacenter fiber
 handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
 circuit is fine.
@@ -69,3 +78,80 @@ on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
 vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
 prints live ESP session keys** — never paste its output into althing, a
 booth, or a commit.
 ---
 ## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
 The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
 and the measurements taken while trying show there is very little left to win.
 ### AES-GCM is unavailable on the far end — not a naming problem
 The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
 `_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
 crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
 `aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
 — all `HTTP 400 api.err.InvalidPayload`, nothing applied.
 **The control that makes this conclusive:** the *identical* request body with
 `ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
 enum rejecting the value, not a malformed body. Corroborating: **zero
 case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
 Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
 `chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
 UniFi far ends, so this blocks the ESH tunnel too.
 The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
 reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
 functionally identical while the peer only offers CBC, and reverting it would
 cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
 IKE protects the control channel only and has no bearing on data throughput.
 ### The numbers that retire this as a problem
 Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
 the same UDM and the same uplink, with the same stdlib TCP probe:
 | streams | IPsec NH3→ANA | WireGuard NH3→ESH |
 |---|---|---|
 | 1 | 245 Mbit/s | 557 Mbit/s |
 | 4 | 471 Mbit/s | 767 Mbit/s |
 | 8 | **692 Mbit/s** | **795 Mbit/s** |
 **NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
 that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
 So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
 IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
 Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
 FortiOS has no WireGuard) would chase that last 15%. Not worth it.
 ### What the constraint actually is
 A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
 endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
 (offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
 shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
 a higher floor (557 → 795 is only 1.43× scaling).
 ### Actionable consequence
 Anything moving bulk data across this link should **parallelise** — 245 → 692
 Mbit/s, a 2.8× win with no config change. For single-stream workloads that
 cannot be parallelised at the application layer, **NFS `nconnect=N` is the
 lever**: it opens N TCP connections per mount, converting a single-stream
 workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
 24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
 example — remounting with `nconnect=8` is the obvious test.
 ### Foot-gun recorded
 Probing the enum by PUTting candidate values **applies the accepted ones**. A
 probe loop here timed out with `3des` briefly live on the NH3 side, which the
 FortiGate would not accept — a short tunnel outage until `aes256` was restored
 (~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
 config enum this way, restore the known-good value after **every** 200, not at
 the end of the loop. Post-change verification: the UDM object was diffed
 field-by-field against its pre-change snapshot and is **byte-identical**.
@@ -108,11 +108,11 @@ no longer deployed sidecars here. See Recent decisions.)
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
-_As of 2026-08-23 — a long multi-party ops session; open with the operator are the Anaheim tunnel cipher and how long WAN admin on ana-gw stays open._
+_As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._
 - **🔓 ana-gw WAN admin is OPEN again as of 2026-08-23 — deliberately, and it should be re-closed after the FortiGate cutover.** Operator asked for it as the cutover contingency ("so I don't have to drive down there"). `https://ana-fw.phasefinal.com/` (wan1's own IP 38.120.12.42) serves the admin GUI on a browser-trusted LE cert; **https only**, and only the `infra-ops` account is WAN-reachable (`admin` remains 10.0.0.0/8-locked). Verified end-to-end with a real login from two sites. This partially reverses the 2026-08-12 hardening on a box still running the **EOL, actively-exploited FortiOS 7.2 branch** — so the exposure is CVE-shaped, and time-boxing it is the mitigation. Re-close = `config system interface / edit wan1 / set allowaccess ping / next / end`. Config backups pre+post at `/var/tmp/ana-gw-config-20260823T20*.conf` on nh3-dev. **Two follow-ons:** (a) ACME renewal for that cert needs **port 80** on wan1 — currently absent, next attempt ≈2026-09-27, expiry 2026-10-27; (b) ~5 SYN/s from `179.51.184.0/21` now draws SYN-ACKs, CPU impact nil, unmitigated by choice. Detail + the FortiOS trusthost gotchas: auto-memory `reference_fortigate_ana_gw_access`.
- **⏳ THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit.** Circuit measured at **2,153 Mbit/s**; the tunnel ceilings ~550 Mbit/s aggregate, ~230 single-stream. FortiGate CPU **idle**, IPsec **NPU-offloaded**, interface error-free — so it is not crypto exhaustion. Both tunnels negotiate **`aes256-sha1`**; **AES-GCM is the proposed change** and the operator has signalled he will authorize it. Not executed: production edge, needs a matching change at NH3 + ESH, each tunnel drops during renegotiation. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
+- **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
 - **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB).