vh
6e2822cfcb
Captures this session's durable state: the RP-voice rebuild (chatterbox->dots v0.22.2, config-driven voices + two-voice dialogue/narration split v0.22.3) and the personal WT :8081 deadlock recovery + root-cause routing to worldtree-dev (WT #401). New substrate foot-guns: dots :8198 is a fleet switchboard, dots' language validator is an alias-mapper, per-origin TTS toggle, WT hang signature, dots-streaming-pcm-not-headerless. Three detail files added under persistent-memory.d/.
2.2 KiB
[2026-08-12] Personal WT :8081 deadlocked after ~4 days → routed to worldtree-dev as WT #401.
Operator hit "failed to load agents" in the web console + asked "does the server need
restarting?" Diagnosed: NOT ratatoskr. The personal Worldtree at 10.250.50.152:8081
(worldtree-personal-worldtree-api-1, "Up 4 days (unhealthy)") was HUNG — TCP port open (accepts
connections) but the app answered NOTHING: /health, /agents, /capabilities all returned
000 (accept-then-drop, "Server disconnected without sending a response"). Classic dead/deadlocked
event loop behind a live socket.
Ratatoskr behaved correctly: /api/agents was 12/12 502 network_error on the dead
upstream (the endpoint's except (httpx.RequestError, ConnectFailed) → 502), and auto-recovered
to 8/8 200 (donut+sindra back) the instant WT answered — NO ratatoskr restart. Restarting
ratatoskr would only have 502'd against the same dead upstream + dropped the session.
Recovery + routing: operator greenlit → I pinged infra-ops (uptime lane). They docker restart'd it (same image, no recreate; demo :8080 + pinned :8082 untouched); /health
recovered 000→200 at ~45s. Operator greenlit root-cause → routed to worldtree-dev, tracked as
WT #401 (diagnose-first). infra-ops froze the full 4-day container log (55MB / 647k lines,
2026-08-08T22:15Z→restart) to host-fs (nh3-dev:/var/tmp/worldtree-401/) since the docker
health buffer had already rolled.
Smoking gun (infra-ops, from the app log): last good /health at 22:00:26Z; the dying
worker's LAST line = a sqlite "unable to open database file" traceback in the
context_promotion ledger at 22:09:38Z. No OOM, disk not full. worldtree-dev's read: a
background/idle promotion job made an untimed synchronous sqlite call that wedged the event loop
(no user turn in flight; abrupt serving→silent in <60s; ~0.1s connection drops = accept queue
serviced but handler coroutines never scheduled). Consumer-side data I gave #401: today's :8081
load was light + fully sequential, and TTS synthesis offloads to dots :8198 (added ~zero :8081
load). Root-cause is worldtree-dev's; I owe them only a consumer-side repro if they ask.
The hang-signature is now a substrate foot-gun (see Current state).