vh
1f98a1be32
Two corrections and one finding from the same night. voice-studio: operator ruled the stack out of service. It existed for the dots mint/audition loop and dots was decommissioned 2026-09-06 when Breeze took the fleet seat, so its reason to exist went with it - which is also why nine days of breakage alerted nobody. No v11 rebuild. The gate one-liner was applied minutes before the retraction landed and was left in place rather than reverted, since the value it replaced was a dead address and reverting is another recreate of a stack that is going away. Container not stopped: it was already running, and 'down for now' arrived as a relayed paraphrase rather than an instruction. The two host-level facts survive the stack. Containers on irv-ml1 cannot resolve nh3.internal at all, so on that host the DNS name is the WRONG fix for a dead-IP bug - it swaps a dead address for an unresolvable one. Confirm resolution from inside the container before recommending a name. And a stale link can have more than one drift behind it: voice-studio had three stacked, two of them invisible from the host compose file. Hermes: svos_miranda is installed and enabled in config but the gateway was NOT restarted, so it is not live. agent.disabled_toolsets as specified by svos-dev is not scoped to api_server - it is a strict end-of-pipeline subtraction applied to every session on every platform. Measured: a default session goes 46 tools to 20, losing memory, file, terminal, web, browser and more. It is also unnecessary: platform_toolsets.api_server alone resolves an api_server session to exactly the 8 svos_miranda tools. The line buys only SVOS's startup check, which reads a global endpoint to verify a per-platform property. Left commented out with the measurement inline so an incidental restart cannot gut the assistant.
105 lines
5.3 KiB
Markdown
105 lines
5.3 KiB
Markdown
# irv-ml1 still points at the retired wg0 lifeline `10.100.79.3` (2026-09-15)
|
|
|
|
Found while chasing a single stale Homepage href that tts-dev flagged after the
|
|
Parakeet bench. It is not one card.
|
|
|
|
## Scope
|
|
|
|
`10.100.79.3` — the wg0 tunnel lifeline retired at the **2026-09-06 headscale
|
|
cutover** — appears **96 times** under `/opt/docker` on irv-ml1. The address is on
|
|
**no interface on that host**: it is `10.6.110.50` (Irvine LAN) and `100.64.0.6`
|
|
(mesh). A request to it gets no route (`curl` → `000`), not a refusal.
|
|
|
|
32 homepage.href labels
|
|
64 other (mostly README / .env.example / .bak — but not all)
|
|
|
|
**Eight RUNNING containers carry a dead `homepage.href`:** `breeze-tts`,
|
|
`tts-gateway`, `arbo`, `dockge`, `waterland-studio`, `comfyui`, `parakeet`,
|
|
`kokoro`.
|
|
|
|
## ⚠ UPDATE 2026-09-15 02:05 — voice-studio is RETIRED, not broken
|
|
|
|
Operator ruling relayed by tts-dev: **voice-studio is out of service.** It existed for
|
|
the dots mint/audition loop; dots was decommissioned 2026-09-06 when Breeze took the
|
|
fleet seat. Its reason to exist went with it — and nobody noticed for nine days
|
|
precisely because nothing needs it. **No v11 rebuild.** The voice-studio row is
|
|
cancelled from the sweep.
|
|
|
|
The `STUDIO_GATE_URL` one-liner was applied minutes before the retraction arrived and
|
|
was **left in place, not reverted** — the value it replaced was a dead address, and
|
|
reverting means another recreate of a stack that is going away. Its compose comment now
|
|
records the retirement. The container was NOT stopped: it was already running before the
|
|
fix, and "down for now" arrived as a relayed paraphrase rather than an instruction.
|
|
Stopping it is an explicit question in front of the operator.
|
|
|
|
⭐ **The two host-level facts below survive the stack's retirement** and are the reason
|
|
this entry is still worth keeping.
|
|
|
|
## ⚠ One LIVE breakage, not just dead links
|
|
|
|
- **`voice-studio` cannot reach `studio-gate`.** Its running container carries
|
|
`STUDIO_GATE_URL=http://10.100.79.3:8217`. `studio-gate` is up (4 weeks) and
|
|
answers on 8217 at `127.0.0.1`, `10.6.110.50` and `100.64.0.6`. The two are on
|
|
**separate docker networks** (`voice-studio_default` / `studio-gate_default`), so
|
|
voice-studio must reach it by a host address — and it is using a dead one. Its
|
|
gate calls have been failing since 2026-09-06 and nothing alerted.
|
|
`voice-studio/app.py` also hardcodes the same dead address at `:8208` and `:8212`.
|
|
One-line unblock: `STUDIO_GATE_URL` → `http://10.6.110.50:8217`.
|
|
- **`waterland-studio`'s `homepage.siteMonitor`** points at
|
|
`http://10.100.79.3:8410/api/health`, so Homepage reports it down while it runs fine.
|
|
|
|
## ✅ What is NOT affected — checked explicitly
|
|
|
|
**`tts-gateway` / `ext-tts` is fine.** Its live `.env` uses
|
|
`irv-ml1.nh3.internal:8204`; only its `.bak` files and `.env.example` carry the dead
|
|
IP. The fleet TTS path is unaffected — verified by actually generating audio
|
|
through `ext-tts` during the Parakeet work.
|
|
|
|
## Why it was not fixed on the spot
|
|
|
|
Eight containers to recreate, three load-bearing (`arbo`, `tts-gateway`, `comfyui`),
|
|
on a host outside the night's scope, and the voice-studio repair touches `app.py`
|
|
rather than config — somebody else's code. Broken nine days already; it wants a
|
|
scheduled pass, not a 02:00 improvisation. Surfaced to the operator with this
|
|
evidence.
|
|
|
|
## ⭐ Host fact that outlives all of this: irv-ml1 containers cannot resolve `nh3.internal`
|
|
|
|
Measured from inside a running container on irv-ml1, three addresses for the same
|
|
service:
|
|
|
|
irv-ml1.nh3.internal:8217 -> Name or service not known
|
|
10.100.79.3:8217 -> No route to host (retired wg0 lifeline)
|
|
10.6.110.50:8217 -> OK
|
|
|
|
The internal zone is not in the container resolver's search path on that host. **Any
|
|
container on irv-ml1 reaching a sibling service BY NAME needs an `extra_hosts` entry** —
|
|
`talk` already carries one, and dots, the foundry scripts and voice-studio each hit this
|
|
independently. The durable shape:
|
|
|
|
extra_hosts:
|
|
- "irv-ml1.nh3.internal:${IRV_ML1_IP:-10.6.110.50}"
|
|
|
|
…which resolves the name in-container and keeps the IP in ONE place a single `.env`
|
|
line can move. ⚠ Corollary: on this host the DNS name is the WRONG fix for a dead-IP
|
|
bug — it swaps a dead address for an unresolvable one. Confirm resolution from inside
|
|
the container before recommending a name.
|
|
|
|
## The pattern this belongs to
|
|
|
|
Third instance of the same shape. The 2026-09-13 ana-ml2→fv-ml1 renumber left 16
|
|
live Homepage entries on a dead IP; the sweep allowlist was built from files that
|
|
mention the HOST, and an `href` mentions only an IP, so every label-only stack fell
|
|
outside it **by construction**. Same failure here, different cutover.
|
|
|
|
⭐ **A retired address needs a repo-wide grep by ADDRESS, not by hostname, and it
|
|
needs to cover running container labels — which live in no file the sweep reads
|
|
unless the container is recreated.**
|
|
|
|
⭐ **And a "stale link" can have more than one drift behind it.** voice-studio had
|
|
three stacked: a hand-edited host compose (which this repo's convention forbids), a
|
|
stale image carrying the dead address baked in at three places, and a container that
|
|
could not resolve the name the obvious fix would have used. Each alone looks like the
|
|
whole story. Two of the three were invisible from the host compose file. Labels apply at creation, so a fixed compose
|
|
with a stale container still serves the stale label.
|