vh
6e2822cfcb
Captures this session's durable state: the RP-voice rebuild (chatterbox->dots v0.22.2, config-driven voices + two-voice dialogue/narration split v0.22.3) and the personal WT :8081 deadlock recovery + root-cause routing to worldtree-dev (WT #401). New substrate foot-guns: dots :8198 is a fleet switchboard, dots' language validator is an alias-mapper, per-origin TTS toggle, WT hang signature, dots-streaming-pcm-not-headerless. Three detail files added under persistent-memory.d/.
31 lines
2.2 KiB
Markdown
31 lines
2.2 KiB
Markdown
`[2026-08-12]` **Personal WT :8081 deadlocked after ~4 days → routed to worldtree-dev as WT #401.**
|
|
|
|
Operator hit "failed to load agents" in the web console + asked "does the server need
|
|
restarting?" Diagnosed: NOT ratatoskr. The personal Worldtree at `10.250.50.152:8081`
|
|
(`worldtree-personal-worldtree-api-1`, "Up 4 days (unhealthy)") was HUNG — TCP port open (accepts
|
|
connections) but the app answered NOTHING: `/health`, `/agents`, `/capabilities` all returned
|
|
`000` (accept-then-drop, "Server disconnected without sending a response"). Classic dead/deadlocked
|
|
event loop behind a live socket.
|
|
|
|
**Ratatoskr behaved correctly:** `/api/agents` was 12/12 502 `network_error` on the dead
|
|
upstream (the endpoint's `except (httpx.RequestError, ConnectFailed) → 502`), and auto-recovered
|
|
to 8/8 200 (donut+sindra back) the instant WT answered — NO ratatoskr restart. Restarting
|
|
ratatoskr would only have 502'd against the same dead upstream + dropped the session.
|
|
|
|
**Recovery + routing:** operator greenlit → I pinged infra-ops (uptime lane). They `docker
|
|
restart`'d it (same image, no recreate; demo `:8080` + pinned `:8082` untouched); `/health`
|
|
recovered 000→200 at ~45s. Operator greenlit root-cause → routed to worldtree-dev, tracked as
|
|
**WT #401** (diagnose-first). infra-ops froze the full 4-day container log (55MB / 647k lines,
|
|
`2026-08-08T22:15Z`→restart) to host-fs (`nh3-dev:/var/tmp/worldtree-401/`) since the docker
|
|
health buffer had already rolled.
|
|
|
|
**Smoking gun (infra-ops, from the app log):** last good `/health` at `22:00:26Z`; the dying
|
|
worker's LAST line = a sqlite **"unable to open database file"** traceback in the
|
|
**`context_promotion` ledger** at `22:09:38Z`. No OOM, disk not full. worldtree-dev's read: a
|
|
background/idle promotion job made an untimed synchronous sqlite call that wedged the event loop
|
|
(no user turn in flight; abrupt serving→silent in <60s; ~0.1s connection drops = accept queue
|
|
serviced but handler coroutines never scheduled). Consumer-side data I gave #401: today's :8081
|
|
load was light + fully sequential, and TTS synthesis offloads to dots `:8198` (added ~zero :8081
|
|
load). **Root-cause is worldtree-dev's; I owe them only a consumer-side repro if they ask.**
|
|
The hang-signature is now a substrate foot-gun (see Current state).
|