diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md index c0f9254..6e8f7a9 100644 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -566,3 +566,51 @@ If both tunnels drop it is console-only until someone is on site. **The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded it as read-verified only): a control PUT of the unchanged object returned `rc:ok`, then the real change did too. That key has a full read+write admin role. + +--- + +## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener + +The claim in the previous section that `.42:80` was an **ISP transparent proxy** +was **WRONG**, and so was the earlier warning that ACME renewal would fail with +port 80 absent from `allowaccess`. Operator pushed back asking where the port-80 +map terminated. It terminates **on the FortiGate itself**. + +**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system +acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to +answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1 +allowaccess` reads `ping` only and the port is still open. Every non-challenge +request returns a fixed 403 whose body is literally: + +``` +ACME Access OnlyACME Access Only +``` + +**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` — +`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252). +Worth noting separately: **four VIPs are all-port static NAT** (no `portforward`), +so they map *every* port of their external IP — `Rustdesk` (.44 → 10.250.50.70), +`https-to-tacticalrmm` (.46 → .57), `web-to-webhost` (.45 → .52), +`web-to-sfcontainer` (.43 → 10.250.150.100). + +### The methodology error that produced the wrong answer — worth not repeating + +The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host` +matches only inbound packets**, so outbound SYN-ACKs were excluded *by +construction*; concluding "the box sends no SYN-ACK" from that capture was +unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it +immediately shows `wan1 out 38.120.12.42.80 -> : syn ack`. + +**Rule: when testing whether a box *answers*, the sniffer filter must be +bidirectional. `dst host` silently answers a different question.** + +### Consequences + +- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http` + back or the cert expires" warning is retracted — FortiOS opens the challenge + port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27. +- **It is not an admin surface** — static 403, no auth, no GUI. +- Its practical value is now low: WAN admin is closed, so the cert only serves + the internal GUI at 10.250.0.1, where the name would not match anyway. Killing + it (`config system acme` → unset interface) would close the last WAN listener + at the cost of cert renewal. Operator's call; **not done**. diff --git a/persistent-memory.md b/persistent-memory.md index dc11e56..e1be7e1 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._ -- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ Port 80 on 38.120.12.42 answers a bare `403` from **outside** — that is an ISP transparent HTTP proxy on the CLIENT side (NH3 and ESH are both behind CGNAT), **not** the FortiGate: inbound :80 SYNs from scanner ranges reach the box and get **no SYN-ACK**, and our own :80 packets never appear in a capture at the box at all. +- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ **Port 80 on 38.120.12.42 is still open and it IS the FortiGate** — the FortiOS **ACME HTTP-01 challenge listener**, opened by `config system acme / set interface "wan1"`, which **bypasses `allowaccess` by design**. Body is literally `ACME Access Only` (403, 101 bytes) for every path; confirmed by sniffer showing `wan1 out 38.120.12.42.80 -> ...: syn ack`. Not a DNAT (only two VIPs exist on .42: Kokoro `:8880`, wg `:31337`) and not an admin surface — it serves only challenge tokens. Closing it = `config system acme` unset interface, which also ends cert renewal. - **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`