diff --git a/persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md b/persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md new file mode 100644 index 0000000..a5fc636 --- /dev/null +++ b/persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md @@ -0,0 +1,58 @@ +# irv-ml1 still points at the retired wg0 lifeline `10.100.79.3` (2026-09-15) + +Found while chasing a single stale Homepage href that tts-dev flagged after the +Parakeet bench. It is not one card. + +## Scope + +`10.100.79.3` — the wg0 tunnel lifeline retired at the **2026-09-06 headscale +cutover** — appears **96 times** under `/opt/docker` on irv-ml1. The address is on +**no interface on that host**: it is `10.6.110.50` (Irvine LAN) and `100.64.0.6` +(mesh). A request to it gets no route (`curl` → `000`), not a refusal. + + 32 homepage.href labels + 64 other (mostly README / .env.example / .bak — but not all) + +**Eight RUNNING containers carry a dead `homepage.href`:** `breeze-tts`, +`tts-gateway`, `arbo`, `dockge`, `waterland-studio`, `comfyui`, `parakeet`, +`kokoro`. + +## ⚠ One LIVE breakage, not just dead links + +- **`voice-studio` cannot reach `studio-gate`.** Its running container carries + `STUDIO_GATE_URL=http://10.100.79.3:8217`. `studio-gate` is up (4 weeks) and + answers on 8217 at `127.0.0.1`, `10.6.110.50` and `100.64.0.6`. The two are on + **separate docker networks** (`voice-studio_default` / `studio-gate_default`), so + voice-studio must reach it by a host address — and it is using a dead one. Its + gate calls have been failing since 2026-09-06 and nothing alerted. + `voice-studio/app.py` also hardcodes the same dead address at `:8208` and `:8212`. + One-line unblock: `STUDIO_GATE_URL` → `http://10.6.110.50:8217`. +- **`waterland-studio`'s `homepage.siteMonitor`** points at + `http://10.100.79.3:8410/api/health`, so Homepage reports it down while it runs fine. + +## ✅ What is NOT affected — checked explicitly + +**`tts-gateway` / `ext-tts` is fine.** Its live `.env` uses +`irv-ml1.nh3.internal:8204`; only its `.bak` files and `.env.example` carry the dead +IP. The fleet TTS path is unaffected — verified by actually generating audio +through `ext-tts` during the Parakeet work. + +## Why it was not fixed on the spot + +Eight containers to recreate, three load-bearing (`arbo`, `tts-gateway`, `comfyui`), +on a host outside the night's scope, and the voice-studio repair touches `app.py` +rather than config — somebody else's code. Broken nine days already; it wants a +scheduled pass, not a 02:00 improvisation. Surfaced to the operator with this +evidence. + +## The pattern this belongs to + +Third instance of the same shape. The 2026-09-13 ana-ml2→fv-ml1 renumber left 16 +live Homepage entries on a dead IP; the sweep allowlist was built from files that +mention the HOST, and an `href` mentions only an IP, so every label-only stack fell +outside it **by construction**. Same failure here, different cutover. + +⭐ **A retired address needs a repo-wide grep by ADDRESS, not by hostname, and it +needs to cover running container labels — which live in no file the sweep reads +unless the container is recreated.** Labels apply at creation, so a fixed compose +with a stale container still serves the stale label. diff --git a/persistent-memory.md b/persistent-memory.md index 277b328..04a44e7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -181,7 +181,9 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). - `[2026-09-15]` ⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned. -- `[2026-09-15]` ⚠ **Two Homepage ASR cards point at the dead wg0 lifeline `10.100.79.3`** (retired at the headscale cutover): the old irv-ml1 `parakeet` :8765 and `Speaches ASR` :8204. The irv-ml1 parakeet is **genuinely still running** (healthz 200 on `100.64.0.6` and `10.6.110.50`) — only its href is stale. So the fleet now has **two** Parakeets. Retiring the irv-ml1 one is the operator's call; not torn down. +- `[2026-09-15]` ⚠⚠ **irv-ml1 still points at the retired wg0 lifeline `10.100.79.3` in 96 places — and one is a LIVE breakage, not a dead link.** `voice-studio` cannot reach `studio-gate` (both up, separate docker networks, gate URL is the dead IP) and has been failing since the 2026-09-06 cutover with nothing alerting. 8 running containers carry dead `homepage.href` labels; `waterland-studio`'s siteMonitor too. ✅ `tts-gateway`/`ext-tts` verified UNAFFECTED. Not fixed — wants a scheduled pass, not a 02:00 improvisation. ⭐ Third instance of the same shape: **a retired address needs a grep by ADDRESS, not by hostname, and labels live in no file until the container is recreated.** → `persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md` + +- `[2026-09-15]` **Parakeet bench settled by tts-dev — FV wins at both clip lengths and beats the incumbent Whisper; IRV seat is now retirable.** FV 155 ms / 391 ms on 1.84 s / 6.24 s clips vs IRV 354 / 1010 vs whisper-large-v3 457 / 690 — IRV is *slower than Whisper* at 6.24 s. Length sweep (n=9/cell, first 3 discarded) fits ~58 ms fixed + 56 ms per audio-second, asymptote ~17.8x realtime, which independently reproduces our 17x on a different harness. Gateway hop measured **below harness resolution** (±30 ms), so `ext-stt` is the right consumer path. ⚠ tts-dev retracted their own plan's 60-120 ms projection: **published RTFx is BATCHED THROUGHPUT, not single-stream latency — the two differ by ~200x.** ⚠ Their between-run variance is ±20% because GPU 0 is the live chat path; our 0.50 s median was taken on an idle GPU 3 and is a best case. - `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`