diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index b0a580a..c0f9254 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -501,3 +501,68 @@ Worth doing, but it is **a project, not a config tweak** — routing, failover a policy all need deciding. The cheap wins remain available meanwhile and are independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending the FortiGate work settling) and AES-128 for ~20%. + +--- + +## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed + +Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break, +then close the FortiGate's WAN and SSH admin surfaces. All done and verified. + +**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is +being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the +edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is +therefore moot — do not scope it. This also **un-parks the OPNsense migration**, +which auto-memory recorded as PARKED pending "hardware acquisition"; the R420 +is that trigger. + +### What changed + +Make-before-break on the FortiGate first, so neither tunnel dropped waiting on +a far end: + +| phase2 | proposal now | +|---|---| +| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` | +| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` | + +Then each UDM flipped to `ipsec_esp_encryption: aes128`: + +| tunnel | UDM object | before | after | +|---|---|---|---| +| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** | +| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** | + +Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B — +the difference is live-link variance, not a different result. Both UDM objects +were diffed field-by-field against pre-change snapshots: **the only field that +moved on either is `ipsec_esp_encryption`.** + +The FortiGate proposal lists were deliberately **left permissive** (still +accepting aes256). The peers offer only aes128 so the extra entries are inert, +and keeping them means a UDM reverting does not strand the tunnel. Narrowing to +`aes128-sha1` alone is a one-liner if the looser list is unwanted. + +### Admin surfaces closed + +`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost +→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed +from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works. +**Sequencing that matters: the close was executed over the TUNNEL path, not over +WAN** — removing `ssh` from allowaccess while connected over WAN kills the +session mid-command. + +**Consequence to hold in mind: ana-gw now has no out-of-band management path.** +If both tunnels drop it is console-only until someone is on site. + +### Gotcha: the two UDM vault items have DIFFERENT shapes + +- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key. +- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a + `key:` line. `secret get` piped straight into a header yields a 1396-byte + value and the UDM answers **`400 Bad Request` from nginx**. Extract with + `grep '^key:' | awk '{print $2}'`. + +**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded +it as read-verified only): a control PUT of the unchanged object returned +`rc:ok`, then the real change did too. That key has a full read+write admin role. diff --git a/persistent-memory.md b/persistent-memory.md index bef76cc..dc11e56 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._ -- **🔓 ana-gw WAN admin is OPEN again as of 2026-08-23 — deliberately, and it should be re-closed after the FortiGate cutover.** Operator asked for it as the cutover contingency ("so I don't have to drive down there"). `https://ana-fw.phasefinal.com/` (wan1's own IP 38.120.12.42) serves the admin GUI on a browser-trusted LE cert; **https only**, and only the `infra-ops` account is WAN-reachable (`admin` remains 10.0.0.0/8-locked). Verified end-to-end with a real login from two sites. This partially reverses the 2026-08-12 hardening on a box still running the **EOL, actively-exploited FortiOS 7.2 branch** — so the exposure is CVE-shaped, and time-boxing it is the mitigation. Re-close = `config system interface / edit wan1 / set allowaccess ping / next / end`. Config backups pre+post at `/var/tmp/ana-gw-config-20260823T20*.conf` on nh3-dev. **Two follow-ons:** (a) ACME renewal for that cert needs **port 80** on wan1 — currently absent, next attempt ≈2026-09-27, expiry 2026-10-27; (b) ~5 SYN/s from `179.51.184.0/21` now draws SYN-ACKs, CPU impact nil, unmitigated by choice. Detail + the FortiOS trusthost gotchas: auto-memory `reference_fortigate_ana_gw_access`. +- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ Port 80 on 38.120.12.42 answers a bare `403` from **outside** — that is an ISP transparent HTTP proxy on the CLIENT side (NH3 and ESH are both behind CGNAT), **not** the FortiGate: inbound :80 SYNs from scanner ranges reach the box and get **no SYN-ACK**, and our own :80 packets never appear in a capture at the box at all. - **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`