ops: adopt AES-128 on both Anaheim tunnels and close public admin

Both tunnels now negotiate AES-128 for ESP, applied make-before-break so neither dropped waiting on its far end: the FortiGate was widened to accept the new cipher alongside the old one first, then each UniFi gateway was flipped. Single-stream throughput moves from 245 to 270 on the NH3 tunnel and from 268 to 304 on the ESH tunnel. Both network objects were diffed against pre-change snapshots and the only field that moved on either is the ESP cipher. The proposal lists are left accepting AES-256 as well. The peers offer only AES-128 so the extra entries are inert, and retaining them means a gateway reverting cannot strand a tunnel. With that up, the WAN administrative surfaces are closed. The interface is back to permitting only ping, and the infra-ops account is again restricted to RFC1918 space. Ports 443 and 22 were confirmed closed from two separate sites and management over the tunnel still works. The close was issued over the tunnel rather than over the WAN, since withdrawing SSH from the interface while connected through it would sever the session mid-command. The box now has no out-of-band path, which the memory records explicitly. Also captured: the two UniFi vault items have different shapes, one a bare key and one a documentation note requiring extraction, which produces an opaque nginx rejection if missed, and the ESH key's first confirmed write.
2026-08-23 15:44:32 -07:00
parent 6edebe4864
commit 7bc9754e40
2 changed files with 66 additions and 1 deletions
@@ -501,3 +501,68 @@ Worth doing, but it is **a project, not a config tweak** — routing, failover a
 policy all need deciding. The cheap wins remain available meanwhile and are
 independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
 the FortiGate work settling) and AES-128 for ~20%.
 ---
 ## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
 Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
 then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
 **Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
 being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
 edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
 therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
 which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
 is that trigger.
 ### What changed
 Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
 a far end:
 | phase2 | proposal now |
 |---|---|
 | `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
 | `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
 Then each UDM flipped to `ipsec_esp_encryption: aes128`:
 | tunnel | UDM object | before | after |
 |---|---|---|---|
 | NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
 | ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
 Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
 the difference is live-link variance, not a different result. Both UDM objects
 were diffed field-by-field against pre-change snapshots: **the only field that
 moved on either is `ipsec_esp_encryption`.**
 The FortiGate proposal lists were deliberately **left permissive** (still
 accepting aes256). The peers offer only aes128 so the extra entries are inert,
 and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
 `aes128-sha1` alone is a one-liner if the looser list is unwanted.
 ### Admin surfaces closed
 `wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
 → **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
 from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
 **Sequencing that matters: the close was executed over the TUNNEL path, not over
 WAN** — removing `ssh` from allowaccess while connected over WAN kills the
 session mid-command.
 **Consequence to hold in mind: ana-gw now has no out-of-band management path.**
 If both tunnels drop it is console-only until someone is on site.
 ### Gotcha: the two UDM vault items have DIFFERENT shapes
 - `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
 - `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
  `key:` line. `secret get` piped straight into a header yields a 1396-byte
  value and the UDM answers **`400 Bad Request` from nginx**. Extract with
  `grep '^key:' | awk '{print $2}'`.
 **The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
 it as read-verified only): a control PUT of the unchanged object returned
 `rc:ok`, then the real change did too. That key has a full read+write admin role.
@@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.)
 _As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._
- **🔓 ana-gw WAN admin is OPEN again as of 2026-08-23 — deliberately, and it should be re-closed after the FortiGate cutover.** Operator asked for it as the cutover contingency ("so I don't have to drive down there"). `https://ana-fw.phasefinal.com/` (wan1's own IP 38.120.12.42) serves the admin GUI on a browser-trusted LE cert; **https only**, and only the `infra-ops` account is WAN-reachable (`admin` remains 10.0.0.0/8-locked). Verified end-to-end with a real login from two sites. This partially reverses the 2026-08-12 hardening on a box still running the **EOL, actively-exploited FortiOS 7.2 branch** — so the exposure is CVE-shaped, and time-boxing it is the mitigation. Re-close = `config system interface / edit wan1 / set allowaccess ping / next / end`. Config backups pre+post at `/var/tmp/ana-gw-config-20260823T20*.conf` on nh3-dev. **Two follow-ons:** (a) ACME renewal for that cert needs **port 80** on wan1 — currently absent, next attempt ≈2026-09-27, expiry 2026-10-27; (b) ~5 SYN/s from `179.51.184.0/21` now draws SYN-ACKs, CPU impact nil, unmitigated by choice. Detail + the FortiOS trusthost gotchas: auto-memory `reference_fortigate_ana_gw_access`.
+- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ Port 80 on 38.120.12.42 answers a bare `403` from **outside** — that is an ISP transparent HTTP proxy on the CLIENT side (NH3 and ESH are both behind CGNAT), **not** the FortiGate: inbound :80 SYNs from scanner ranges reach the box and get **no SYN-ACK**, and our own :80 packets never appear in a capture at the box at all.
 - **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`