From 66c860d6c1dcb99ef1944c78534029a1103a518b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 08:41:44 -0700 Subject: [PATCH] fix(sweep): retire the dead 10.100.79.3 address across the fleet Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is on no interface anywhere, so anything pointing at it gets no route at all. Homepage went from 9 dead cards to 0 of 112. The load-bearing part is that there is no single right target: it depends on who resolves it. The operator's browser and the Homepage and open-webui containers on esh-docker-vm all resolve nh3.internal, so those get the name and survive the next renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those get the IP. litellm on ana-docker looked like a counterexample and is not: it resolves the name only through its own extra_hosts entry, while asset-engine on the same host fails on it. Test from the container you are about to change, never from a neighbour. Before committing to the name I confirmed the Homepage container actually fetches ytvc's healthz through it in production rather than assuming resolution implies reach. On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history. Seven running containers recreated so the labels took. Seven dormant ones carried stale labels because editing a compose file does not touch an existing container object - fixed with compose create --force-recreate, which rebuilds the container without starting it, the right tool for a deliberately dormant stack. The sweep's real find was off irv-ml1 entirely: four live values on two other hosts, silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS, asset-engine's inference host, and two skaldsong TTS URLs. Both running services were recreated and verified reaching their targets afterwards rather than merely carrying the new string. One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I caught it only because I had taken a baseline before touching it. A label-only edit still costs a full model reload on a GPU container. --- .../2026-09-15-irv-ml1-address-sweep-done.md | 81 +++++++++++++++++++ stacks/asset-engine/.env.example | 2 +- stacks/open-webui/.env.example | 2 +- stacks/skaldsong/.env.example | 2 +- 4 files changed, 84 insertions(+), 3 deletions(-) create mode 100644 persistent-memory.d/2026-09-15-irv-ml1-address-sweep-done.md diff --git a/persistent-memory.d/2026-09-15-irv-ml1-address-sweep-done.md b/persistent-memory.d/2026-09-15-irv-ml1-address-sweep-done.md new file mode 100644 index 0000000..c861016 --- /dev/null +++ b/persistent-memory.d/2026-09-15-irv-ml1-address-sweep-done.md @@ -0,0 +1,81 @@ +# irv-ml1 dead-address sweep — EXECUTED 2026-09-15 (operator-directed) + +Closes `2026-09-15-irv-ml1-dead-wg0-address.md`. `10.100.79.3` (the wg0 lifeline +retired at the 2026-09-06 headscale cutover) is on **no interface anywhere**; a +request to it gets *no route*, not a refusal. + +**Result: 0 of 112 Homepage cards point at it. Was 9.** + +## Which address, and why it differs per host — this is the load-bearing part + +⚠ **There is no single right answer.** The correct target depends on **who resolves +it**, and getting this wrong swaps a dead address for an unresolvable one — which is +what tts-dev nearly did on voice-studio. + +| resolver | can it resolve `*.nh3.internal`? | use | +|---|---|---| +| operator's browser (`homepage.href`) | yes | **name** | +| Homepage container @ esh-docker-vm (`siteMonitor`) | **yes** (measured) | **name** | +| open-webui container @ esh-docker-vm | **yes** (measured) | **name** | +| containers on **irv-ml1** | **NO** | IP | +| containers on **ana-docker** | **NO** | IP | + +⚠⚠ **litellm on ana-docker resolves the name ONLY because of its own `extra_hosts` +entry.** Testing from litellm and generalising would have been wrong: `asset-engine` +on the same host, with no `extra_hosts`, gets `Name or service not known`. **Test from +the container you are about to change, not a neighbour.** + +Positive control used before committing to the name: the Homepage container actually +*fetched* ytvc's `/healthz` through `irv-ml1.nh3.internal` and got `{"ok":true,…}` — +the pattern already worked in production. + +## What was changed + +**irv-ml1 — 24 files swept**, 14 comment-only hits deliberately left as +port-allocation history. + +- **7 running** containers recreated so labels took: `arbo`, `breeze-tts`, `comfyui`, + `dockge`, `kokoro`, `tts-gateway`, `waterland-studio` (+ its `siteMonitor`). + `dockge`'s href is built from `DOCKGE_HOST_IP` in `.env`, so one var fixed it. +- **7 dormant** containers in `Created`/`Exited` state carried stale labels because a + compose edit alone does not touch an existing container object. Fixed with + **`docker compose create --force-recreate`** — rebuilds the container with new + labels **without starting it**, which is exactly right for a deliberately dormant + stack: `chatterbox-fast`, `dots-tts`, `speaches`, `cosyvoice`, `fish-s2`, + `index-tts`, `omnivoice`. `omnivoice-ref` (a different stack, Up 4 weeks) untouched. +- 17 more dormant compose files swept; inert until someone starts them. + +## ⚠⚠ FOUR MORE LIVE BREAKAGES on OTHER hosts — the sweep's real find + +The renumber left a trail off irv-ml1 that nobody grepped for. All silently dead nine +days, none alerting: + +| host | service | was | now | +|---|---|---|---| +| esh-docker-vm | **open-webui** read-aloud TTS | `AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1` | `irv-ml1.nh3.internal` | +| ana-docker | **asset-engine** (Up, healthy) | `INFERENCE_HOST=10.100.79.3` | `10.6.110.50` | +| ana-docker | skaldsong TTS (Exited 2 mo) | `SKALDSONG_TTS_BASE_URL=…8193` | `10.6.110.50` | +| ana-docker | skaldsong vibevoice | `…8194` | `10.6.110.50` | + +Both running services recreated and **verified reaching their targets after the +change**, not just carrying the new string. Repo `.env.example` templates updated to +match per-host (a template hands the next person a dead address otherwise). + +## ⚠ My own error, worth not repeating + +I recreated **`breeze-tts` for a cosmetic label change** and took the fleet TTS path +down for its ~90 s CUDA-graph warm-up. `ext-tts` returned **500** until it finished. +Caught it only because I had taken a **baseline before touching it** (200 / 35,180 +bytes) — without that, a 500 would have looked like a pre-existing fault. + +⭐ **A label-only edit still costs a full model reload on a GPU container.** Batch +those into a window or accept a stale label; do not bounce a warming model for a link. + +## Still open + +- `speaches`'s label says `:8204`, which is **breeze-tts's live port**. Latent conflict + if anyone starts it. Not fixed — it is a port-register question, not an address one. +- ⭐ The durable fix for ana-docker/irv-ml1 containers is **`extra_hosts` + the name** + (what `litellm` and `talk` already do), so the IP lives in one place a single `.env` + line moves. Left as an improvement rather than done, because `asset-engine` is + CI-deployed and its compose is not mine to edit from here. diff --git a/stacks/asset-engine/.env.example b/stacks/asset-engine/.env.example index 970e920..ee9f4fb 100644 --- a/stacks/asset-engine/.env.example +++ b/stacks/asset-engine/.env.example @@ -31,7 +31,7 @@ ASSET_ENGINE_SSH_DIR=/opt/docker/conf/asset-engine/ssh # Inference target. Default is irv-ml1 over WG. Override if the fleet's # inference host moves. -INFERENCE_HOST=10.100.79.3 +INFERENCE_HOST=10.6.110.50 # OIDC seam — empty in v1 (auth is no-op). Populate when v2 forward-auth # lands. Pre-allocated here so the surface is visible in the config file diff --git a/stacks/open-webui/.env.example b/stacks/open-webui/.env.example index 8e14747..a052abb 100644 --- a/stacks/open-webui/.env.example +++ b/stacks/open-webui/.env.example @@ -40,7 +40,7 @@ RAG_OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1 # --- TTS (G4 — tts-dev tunes model/voice). Direct at the dots gateway, no LiteLLM. --- AUDIO_TTS_ENGINE=openai -AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1 +AUDIO_TTS_OPENAI_API_BASE_URL=http://irv-ml1.nh3.internal:8198/v1 AUDIO_TTS_OPENAI_API_KEY=sk-no-auth-lan-seat AUDIO_TTS_MODEL=tts-1 AUDIO_TTS_VOICE=nova diff --git a/stacks/skaldsong/.env.example b/stacks/skaldsong/.env.example index aa6492b..0d5a839 100644 --- a/stacks/skaldsong/.env.example +++ b/stacks/skaldsong/.env.example @@ -61,4 +61,4 @@ SKALDSONG_HOST_CORS_ORIGINS=["http://10.250.50.70:8300"] # TTS — Kokoro on irv-ml1, reached over WireGuard via ana-wg. Same URL # works fleet-wide; the WG tunnel handles the cross-site routing. SKALDSONG_TTS_ENGINE=kokoro -SKALDSONG_TTS_BASE_URL=http://10.100.79.3:8193 +SKALDSONG_TTS_BASE_URL=http://10.6.110.50:8193