vh
66c860d6c1
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is on no interface anywhere, so anything pointing at it gets no route at all. Homepage went from 9 dead cards to 0 of 112. The load-bearing part is that there is no single right target: it depends on who resolves it. The operator's browser and the Homepage and open-webui containers on esh-docker-vm all resolve nh3.internal, so those get the name and survive the next renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those get the IP. litellm on ana-docker looked like a counterexample and is not: it resolves the name only through its own extra_hosts entry, while asset-engine on the same host fails on it. Test from the container you are about to change, never from a neighbour. Before committing to the name I confirmed the Homepage container actually fetches ytvc's healthz through it in production rather than assuming resolution implies reach. On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history. Seven running containers recreated so the labels took. Seven dormant ones carried stale labels because editing a compose file does not touch an existing container object - fixed with compose create --force-recreate, which rebuilds the container without starting it, the right tool for a deliberately dormant stack. The sweep's real find was off irv-ml1 entirely: four live values on two other hosts, silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS, asset-engine's inference host, and two skaldsong TTS URLs. Both running services were recreated and verified reaching their targets afterwards rather than merely carrying the new string. One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I caught it only because I had taken a baseline before touching it. A label-only edit still costs a full model reload on a GPU container.
4.3 KiB
irv-ml1 dead-address sweep — EXECUTED 2026-09-15 (operator-directed)
Closes 2026-09-15-irv-ml1-dead-wg0-address.md. 10.100.79.3 (the wg0 lifeline
retired at the 2026-09-06 headscale cutover) is on no interface anywhere; a
request to it gets no route, not a refusal.
Result: 0 of 112 Homepage cards point at it. Was 9.
Which address, and why it differs per host — this is the load-bearing part
⚠ There is no single right answer. The correct target depends on who resolves it, and getting this wrong swaps a dead address for an unresolvable one — which is what tts-dev nearly did on voice-studio.
| resolver | can it resolve *.nh3.internal? |
use |
|---|---|---|
operator's browser (homepage.href) |
yes | name |
Homepage container @ esh-docker-vm (siteMonitor) |
yes (measured) | name |
| open-webui container @ esh-docker-vm | yes (measured) | name |
| containers on irv-ml1 | NO | IP |
| containers on ana-docker | NO | IP |
⚠⚠ litellm on ana-docker resolves the name ONLY because of its own extra_hosts
entry. Testing from litellm and generalising would have been wrong: asset-engine
on the same host, with no extra_hosts, gets Name or service not known. Test from
the container you are about to change, not a neighbour.
Positive control used before committing to the name: the Homepage container actually
fetched ytvc's /healthz through irv-ml1.nh3.internal and got {"ok":true,…} —
the pattern already worked in production.
What was changed
irv-ml1 — 24 files swept, 14 comment-only hits deliberately left as port-allocation history.
- 7 running containers recreated so labels took:
arbo,breeze-tts,comfyui,dockge,kokoro,tts-gateway,waterland-studio(+ itssiteMonitor).dockge's href is built fromDOCKGE_HOST_IPin.env, so one var fixed it. - 7 dormant containers in
Created/Exitedstate carried stale labels because a compose edit alone does not touch an existing container object. Fixed withdocker compose create --force-recreate— rebuilds the container with new labels without starting it, which is exactly right for a deliberately dormant stack:chatterbox-fast,dots-tts,speaches,cosyvoice,fish-s2,index-tts,omnivoice.omnivoice-ref(a different stack, Up 4 weeks) untouched. - 17 more dormant compose files swept; inert until someone starts them.
⚠⚠ FOUR MORE LIVE BREAKAGES on OTHER hosts — the sweep's real find
The renumber left a trail off irv-ml1 that nobody grepped for. All silently dead nine days, none alerting:
| host | service | was | now |
|---|---|---|---|
| esh-docker-vm | open-webui read-aloud TTS | AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1 |
irv-ml1.nh3.internal |
| ana-docker | asset-engine (Up, healthy) | INFERENCE_HOST=10.100.79.3 |
10.6.110.50 |
| ana-docker | skaldsong TTS (Exited 2 mo) | SKALDSONG_TTS_BASE_URL=…8193 |
10.6.110.50 |
| ana-docker | skaldsong vibevoice | …8194 |
10.6.110.50 |
Both running services recreated and verified reaching their targets after the
change, not just carrying the new string. Repo .env.example templates updated to
match per-host (a template hands the next person a dead address otherwise).
⚠ My own error, worth not repeating
I recreated breeze-tts for a cosmetic label change and took the fleet TTS path
down for its ~90 s CUDA-graph warm-up. ext-tts returned 500 until it finished.
Caught it only because I had taken a baseline before touching it (200 / 35,180
bytes) — without that, a 500 would have looked like a pre-existing fault.
⭐ A label-only edit still costs a full model reload on a GPU container. Batch those into a window or accept a stale label; do not bounce a warming model for a link.
Still open
speaches's label says:8204, which is breeze-tts's live port. Latent conflict if anyone starts it. Not fixed — it is a port-register question, not an address one.- ⭐ The durable fix for ana-docker/irv-ml1 containers is
extra_hosts+ the name (whatlitellmandtalkalready do), so the IP lives in one place a single.envline moves. Left as an improvement rather than done, becauseasset-engineis CI-deployed and its compose is not mine to edit from here.