ops(ana-gw): port 80 on the WAN address is the FortiOS ACME listener

The previous note attributed the open port to an ISP transparent proxy. That was wrong. It terminates on the FortiGate: system acme is bound to wan1, and FortiOS opens port 80 there to answer HTTP-01 challenges regardless of what allowaccess permits, which is why the port stays open with the interface set to ping only. Every non-challenge request returns a fixed 403 whose body reads ACME Access Only. No DNAT is involved; of the fourteen virtual IPs only two land on that address, neither on port 80. The wrong conclusion came from a sniffer filtered on dst host, which matches inbound packets alone and so excluded the replies being looked for. Filtered bidirectionally the box is plainly seen emitting SYN-ACK on port 80. The note records the rule. Two consequences follow. The earlier warning that certificate renewal would fail without http in allowaccess is retracted, since FortiOS opens the challenge port itself. And the listener is not an administrative surface, though its value is now marginal with WAN administration closed, so the note records how to remove it and leaves that decision open. Also captured: four virtual IPs are all-port static NAT and map every port of their external address.
2026-08-23 15:53:55 -07:00
parent 7bc9754e40
commit f39b66d2e1
2 changed files with 49 additions and 1 deletions
@@ -566,3 +566,51 @@ If both tunnels drop it is console-only until someone is on site.
 **The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
 it as read-verified only): a control PUT of the unchanged object returned
 `rc:ok`, then the real change did too. That key has a full read+write admin role.
+
+---
+
+## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
+
+The claim in the previous section that `.42:80` was an **ISP transparent proxy**
+was **WRONG**, and so was the earlier warning that ACME renewal would fail with
+port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
+map terminated. It terminates **on the FortiGate itself**.
+
+**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
+acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
+answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
+allowaccess` reads `ping` only and the port is still open. Every non-challenge
+request returns a fixed 403 whose body is literally:
+
+```
+<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
+```
+
+**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
+`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
+Worth noting separately: **four VIPs are all-port static NAT** (no `portforward`),
+so they map *every* port of their external IP — `Rustdesk` (.44 → 10.250.50.70),
+`https-to-tacticalrmm` (.46 → .57), `web-to-webhost` (.45 → .52),
+`web-to-sfcontainer` (.43 → 10.250.150.100).
+
+### The methodology error that produced the wrong answer — worth not repeating
+
+The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
+matches only inbound packets**, so outbound SYN-ACKs were excluded *by
+construction*; concluding "the box sends no SYN-ACK" from that capture was
+unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
+immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
+
+**Rule: when testing whether a box *answers*, the sniffer filter must be
+bidirectional. `dst host` silently answers a different question.**
+
+### Consequences
+
+- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
+  back or the cert expires" warning is retracted — FortiOS opens the challenge
+  port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
+- **It is not an admin surface** — static 403, no auth, no GUI.
+- Its practical value is now low: WAN admin is closed, so the cert only serves
+  the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
+  it (`config system acme` → unset interface) would close the last WAN listener
+  at the cost of cert renewal. Operator's call; **not done**.
@@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.)

 _As of 2026-08-23 — a long multi-party ops session. The Anaheim tunnel cipher is now settled (closed: the remedy does not exist). The one thing still open with the operator is how long WAN admin on ana-gw stays open._

- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ Port 80 on 38.120.12.42 answers a bare `403` from **outside** — that is an ISP transparent HTTP proxy on the CLIENT side (NH3 and ESH are both behind CGNAT), **not** the FortiGate: inbound :80 SYNs from scanner ranges reach the box and get **no SYN-ACK**, and our own :80 packets never appear in a capture at the box at all.
+- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ **Port 80 on 38.120.12.42 is still open and it IS the FortiGate** — the FortiOS **ACME HTTP-01 challenge listener**, opened by `config system acme / set interface "wan1"`, which **bypasses `allowaccess` by design**. Body is literally `ACME Access Only` (403, 101 bytes) for every path; confirmed by sniffer showing `wan1 out 38.120.12.42.80 -> ...: syn ack`. Not a DNAT (only two VIPs exist on .42: Kokoro `:8880`, wg `:31337`) and not an admin surface — it serves only challenge tokens. Closing it = `config system acme` unset interface, which also ends cert renewal.

 - **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`