Files
esh-pfi-infrastructure/persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md
T
vh 1f98a1be32 docs(memory): voice-studio is retired not broken; agent.disabled_toolsets is global
Two corrections and one finding from the same night.

voice-studio: operator ruled the stack out of service. It existed for the dots
mint/audition loop and dots was decommissioned 2026-09-06 when Breeze took the
fleet seat, so its reason to exist went with it - which is also why nine days of
breakage alerted nobody. No v11 rebuild. The gate one-liner was applied minutes
before the retraction landed and was left in place rather than reverted, since
the value it replaced was a dead address and reverting is another recreate of a
stack that is going away. Container not stopped: it was already running, and
'down for now' arrived as a relayed paraphrase rather than an instruction.

The two host-level facts survive the stack. Containers on irv-ml1 cannot resolve
nh3.internal at all, so on that host the DNS name is the WRONG fix for a dead-IP
bug - it swaps a dead address for an unresolvable one. Confirm resolution from
inside the container before recommending a name. And a stale link can have more
than one drift behind it: voice-studio had three stacked, two of them invisible
from the host compose file.

Hermes: svos_miranda is installed and enabled in config but the gateway was NOT
restarted, so it is not live. agent.disabled_toolsets as specified by svos-dev is
not scoped to api_server - it is a strict end-of-pipeline subtraction applied to
every session on every platform. Measured: a default session goes 46 tools to 20,
losing memory, file, terminal, web, browser and more. It is also unnecessary:
platform_toolsets.api_server alone resolves an api_server session to exactly the
8 svos_miranda tools. The line buys only SVOS's startup check, which reads a
global endpoint to verify a per-platform property. Left commented out with the
measurement inline so an incidental restart cannot gut the assistant.
2026-09-15 02:08:00 -07:00

5.3 KiB

irv-ml1 still points at the retired wg0 lifeline 10.100.79.3 (2026-09-15)

Found while chasing a single stale Homepage href that tts-dev flagged after the Parakeet bench. It is not one card.

Scope

10.100.79.3 — the wg0 tunnel lifeline retired at the 2026-09-06 headscale cutover — appears 96 times under /opt/docker on irv-ml1. The address is on no interface on that host: it is 10.6.110.50 (Irvine LAN) and 100.64.0.6 (mesh). A request to it gets no route (curl → 000), not a refusal.

32  homepage.href labels
64  other (mostly README / .env.example / .bak — but not all)

Eight RUNNING containers carry a dead homepage.href: breeze-tts, tts-gateway, arbo, dockge, waterland-studio, comfyui, parakeet, kokoro.

⚠ UPDATE 2026-09-15 02:05 — voice-studio is RETIRED, not broken

Operator ruling relayed by tts-dev: voice-studio is out of service. It existed for the dots mint/audition loop; dots was decommissioned 2026-09-06 when Breeze took the fleet seat. Its reason to exist went with it — and nobody noticed for nine days precisely because nothing needs it. No v11 rebuild. The voice-studio row is cancelled from the sweep.

The STUDIO_GATE_URL one-liner was applied minutes before the retraction arrived and was left in place, not reverted — the value it replaced was a dead address, and reverting means another recreate of a stack that is going away. Its compose comment now records the retirement. The container was NOT stopped: it was already running before the fix, and "down for now" arrived as a relayed paraphrase rather than an instruction. Stopping it is an explicit question in front of the operator.

⭐ The two host-level facts below survive the stack's retirement and are the reason this entry is still worth keeping.

  • voice-studio cannot reach studio-gate. Its running container carries STUDIO_GATE_URL=http://10.100.79.3:8217. studio-gate is up (4 weeks) and answers on 8217 at 127.0.0.1, 10.6.110.50 and 100.64.0.6. The two are on separate docker networks (voice-studio_default / studio-gate_default), so voice-studio must reach it by a host address — and it is using a dead one. Its gate calls have been failing since 2026-09-06 and nothing alerted. voice-studio/app.py also hardcodes the same dead address at :8208 and :8212. One-line unblock: STUDIO_GATE_URL → http://10.6.110.50:8217.
  • waterland-studio's homepage.siteMonitor points at http://10.100.79.3:8410/api/health, so Homepage reports it down while it runs fine.

✅ What is NOT affected — checked explicitly

tts-gateway / ext-tts is fine. Its live .env uses irv-ml1.nh3.internal:8204; only its .bak files and .env.example carry the dead IP. The fleet TTS path is unaffected — verified by actually generating audio through ext-tts during the Parakeet work.

Why it was not fixed on the spot

Eight containers to recreate, three load-bearing (arbo, tts-gateway, comfyui), on a host outside the night's scope, and the voice-studio repair touches app.py rather than config — somebody else's code. Broken nine days already; it wants a scheduled pass, not a 02:00 improvisation. Surfaced to the operator with this evidence.

⭐ Host fact that outlives all of this: irv-ml1 containers cannot resolve nh3.internal

Measured from inside a running container on irv-ml1, three addresses for the same service:

irv-ml1.nh3.internal:8217   -> Name or service not known
10.100.79.3:8217            -> No route to host   (retired wg0 lifeline)
10.6.110.50:8217            -> OK

The internal zone is not in the container resolver's search path on that host. Any container on irv-ml1 reaching a sibling service BY NAME needs an extra_hosts entry — talk already carries one, and dots, the foundry scripts and voice-studio each hit this independently. The durable shape:

extra_hosts:
  - "irv-ml1.nh3.internal:${IRV_ML1_IP:-10.6.110.50}"

…which resolves the name in-container and keeps the IP in ONE place a single .env line can move. ⚠ Corollary: on this host the DNS name is the WRONG fix for a dead-IP bug — it swaps a dead address for an unresolvable one. Confirm resolution from inside the container before recommending a name.

The pattern this belongs to

Third instance of the same shape. The 2026-09-13 ana-ml2→fv-ml1 renumber left 16 live Homepage entries on a dead IP; the sweep allowlist was built from files that mention the HOST, and an href mentions only an IP, so every label-only stack fell outside it by construction. Same failure here, different cutover.

⭐ A retired address needs a repo-wide grep by ADDRESS, not by hostname, and it needs to cover running container labels — which live in no file the sweep reads unless the container is recreated.

⭐ And a "stale link" can have more than one drift behind it. voice-studio had three stacked: a hand-edited host compose (which this repo's convention forbids), a stale image carrying the dead address baked in at three places, and a container that could not resolve the name the obvious fix would have used. Each alone looks like the whole story. Two of the three were invisible from the host compose file. Labels apply at creation, so a fixed compose with a stale container still serves the stale label.