ops(pfi-tacticalrmm): *.phasefinal.com renews via dns-cloudflare (was manual; 16 days from expiry)

persistent-memory.md

@@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 
 ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT)
 
-- **Waiting on Prime (2026-10-05 0355):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; parked pfi-pteradactyl, pfi-tacticalrmm (certbot), pfi-gx10 (dgx-release); CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause).
+- **Waiting on Prime (2026-10-05 0355):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; parked pfi-pteradactyl, pfi-tacticalrmm (cert now fixed, so it can be patched), pfi-gx10 (dgx-release); CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause).
 - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%.
 - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
 - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
@@ -302,6 +302,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 ## Recent decisions
 
 - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
+- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
 - `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md`
 - `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result
 - `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`

servers/pfi-tacticalrmm/README.md

@@ -18,6 +18,26 @@ Tactical RMM (remote monitoring + management) server at the Anaheim colo.
 [TacticalRMM](https://tacticalrmm.com/) — open-source RMM platform.
 Monitors and manages endpoints, pushes patches, runs scripts, etc.
 
+## TLS certificate — `*.phasefinal.com`, auto-renewing since 2026-10-05
+
+- nginx serves the Let's Encrypt wildcard from `/etc/letsencrypt/live/phasefinal.com` for rmm, rmm-api and rmm-mesh
+  (`:443`). It is nginx's only consumer: NATS runs without TLS behind nginx. MeshCentral's AMT/CIRA listener on
+  `:4433` has its own self-signed certificate (valid to 2049) and does not depend on this one.
+- **It used to be issued by hand** (`authenticator = manual`), which can never auto-renew. Found 2026-10-05 with 16
+  days left (it expired 2026-10-21). Now `certbot.timer` renews it with **dns-cloudflare**:
+  - credentials: `/etc/letsencrypt/cloudflare-phasefinal.ini` (root 0600)
+  - token: Cloudflare account token `pfi-tacticalrmm-certbot-dns01` (id `9f83ca60…`), Zone Read + DNS Write on
+    **phasefinal.com only**, minted with infra-ops's token-creator and vaulted as
+    `pfi-tacticalrmm/cloudflare-certbot-dns01-token`
+  - deploy hook: `/etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh`
+  - reissued 2026-10-05, expires 2027-01-03. `certbot renew --dry-run` passes. Pre-change renewal conf:
+    `/root/phasefinal.com.conf.pre-dns-cloudflare-20261005`.
+- The stale lineage `phasefinal.com-0001` (apex only, expired 2025-07-06) was deleted. Its renewal failure was half of
+  what made `certbot.service` fail daily.
+- ⚠ `/etc/nginx/sites-enabled/meshcentral.conf.bak-20261002-upload` is a backup file sitting in `sites-enabled`, so
+  nginx loads it (`conflicting server name "rmm-mesh.phasefinal.com" on :80, ignored`). Harmless today; move it out
+  when convenient.
+
 ## MeshCentral (bundled with TacticalRMM) — facts checked 2026-10-02
 
 - Runs natively (`meshcentral.service`, user `tactical`, `/meshcentral`), Node 18.20.8, MeshCentral 1.2.0,