ratatoskr/persistent-memory.md

Persistent memory — ratatoskr

Last updated: 2026-05-22

This file captures durable intent and supporting evidence (goals, decisions, foot-gun warnings, in-flight state) across context resets. Read it at session start; treat it as one input alongside CLAUDE.md and the auto-memory system, not as the single source of truth.

When durable state shifts enough to warrant capture, run /snapshot and commit alongside the next commit per the persistent-memory commit-along rule in CLAUDE.md.

---

Repo purpose

Ratatoskr is a dev-grade debug-observability TUI for Worldtree's Conversation API. The product IS the observability surface; chat is the input mechanism. Devs run Ratatoskr against a local Worldtree to watch a turn flow through every layer of the system, side-by-side, in one terminal: agent SSE stream, persona/Vili affect dispatch, tool calls, Bifrost handshake state, admin lifecycle events, optional raw server log.

Named after the squirrel that runs up and down Yggdrasil carrying messages between layers. On-the-nose Worldtree resonance (Yggdrasil = the World Tree).

Origin: althing ask from worldtree-dev (thread 01KS3R34XD3N6HMK91VXESHGW7, 2026-05-20) for the shape of a TUI Conversation API consumer. brokkr-smithy ran the shape pass; operator's reframe routed it as a new repo with a separate dev team rather than an in-tree Worldtree tool.

Current state / in-flight

As of 2026-05-21:

Status: v0 milestone + issue #7 (mid-stream robustness) landed. Four core issues complete end-to-end (sse_client #1, sessions #2, cli #3, tui #4) + one robustness fix on top: MalformedSseData exception

• empty-data skip in _iter_events (issue #7, fixes mid-stream JSONDecodeError crash discovered during long TUI conversations). 172/172 tests GREEN (46 sse_client + 19 sessions + 60 cli + 44 tui + 2 boundary/metadata + 1 from #7 last-seen probe); ruff clean; all five issue contracts (#1, #3, #4, #5, #7) drift-check clean.

--send validated end-to-end against personal Worldtree (http://10.250.50.152:8081, mimir on qwen3.6-35-a3b). Smoke key in gitignored env.sh (delivered by infra-ops 2026-05-21; see personal-worldtree-smoke-target in auto-memory). Long-conversation smoke 2026-05-22 (3193-token completion, 374980-token context, 50s streaming) confirmed empty-data frames are silently skipped — the original 2026-05-22 crash unreproducible after fix.

In-flight: issue #5 (--end-user-id flag for per-user agents like lofn). Contract drafted + parked (untracked at docs/contracts/issues/5.contract.md); Volva paraphrase + TDD + smoke pending. Issue #6 (TUI startup error visibility) also filed but unscaffolded. Branch: main (clean apart from #5 contract). Remote: origin → git@gitea.phasefinal.com:vh/ratatoskr.git.

Next natural moves:

1. Resume issue #5 cadence — --end-user-id flag for per-user agents (lofn). Contract drafted + parked; next steps: Volva paraphrase, TDD, smoke against lofn. Unblocks the Persona/Vili agent family.
2. Issue #6 (TUI startup error visibility) — filed, unscaffolded. Restructure: move session-create out of on_mount into run_tui pre-App.run() so errors print to real stderr (not the alt-screen that tears down before user can read them). Independent of #5.
3. TUI smoke (operator-side) — source env.sh && uv run ratatoskr --new --agent mimir from an interactive terminal. Validates Textual app lifecycle + post-Done markdown re-render end-to-end. Needs a TTY which CC sessions don't have.
4. Side-pane issues — design-brief §5 lists 5 side panes (Persona, Tools, AdminEvents, BifrostState, ServerLog). Persona is the natural first (file-tail of persona.log).
5. Recorded SSE snapshot fixtures from a running Worldtree. --send --new > fixture IS the recording probe.

Recent decisions

Chronological log of decisions with [YYYY-MM-DD] prefix. One line per decision. Captures rationale that won't be obvious from code alone.

• [2026-05-20] Project name Ratatoskr (squirrel on Yggdrasil — runs up and down carrying messages). Earlier candidate Andvari demoted on the cursed-ring association.
• [2026-05-20] Separate repo, separate dev team. Operator's call; the in-tree-at-Worldtree/tools/ alternative was considered and rejected to dogfood the API boundary.
• [2026-05-20] No Worldtree-source imports. Spec-only dependency. Triple version-skew mitigation: spec-pin in pyproject.toml + recorded-SSE snapshot tests + conformance smoke. Initial pin: 55101e909abcd2219833266b6f905c5bc956e0f0 (Worldtree v0.19.0). See docs/SPEC-PIN.md.
• [2026-05-20] Textual (not rich+prompt_toolkit). Driver: debug observability is the primary purpose, and a multi-pane dashboard with persistent side panes + independent scrollback is structurally application-shell-shaped. Volva consulted via cross-frontier second-opinion and converged on the same call.
• [2026-05-20] httpx-sse for SSE consumption. The server emits composite {turn_id}:{seq} id: lines (Worldtree INV-014) load-bearing for SSE-resume; hand-rolled data:-only parsing (the skaldsong pattern) silently drops these. Ratatoskr becomes the reference Python SSE-resume implementation.
• [2026-05-20] Persona-pane PII posture: label-don't-refuse. persona.log is process-wide; pane title flips between [Persona — PROCESS-WIDE] and [Persona — session <id>…] based on whether log lines carry session_id. Refuse-against-non-local was considered and rejected as paternalistic.
• [2026-05-20] Server-stdout pane: opt-in via --server-log <path>. No auto-detection of well-known paths.
• [2026-05-20] Two-stage Ctrl-C. First cancels in-flight turn server-side; second exits app. Ctrl-D bound to immediate exit.
• [2026-05-20] Single-session-per-launch + startup picker. No in-app /switch. CLI flags --session <id> and --new for scripted use. Session identity always visible in Textual footer.
• [2026-05-20] Markdown rendering default-on; --raw opt-out. Don't pre-design --no-stream-formatting (Volva: add only if streaming-markdown rendering is empirically ugly).
• [2026-05-20] Non-interactive --send mode. Single SSE consumer module, two presenters (TUI + stdout). Keeps Ratatoskr honest as an API consumer; useful for CI / scripted probes.
• [2026-05-20] First contract: ratatoskr.sse_client. Bundles stream_turn + reconnect_turn + cancel_turn + private _parse_sse_id into one module — the SSE-resume flow is coupled (cancel needs turn_id from the SSE wire id:, reconnect re-uses the same parsed SseId), so they share a contract. Hard invariant INV-002 makes the composite {turn_id}:{seq} id: parsing load-bearing — closes the foot-gun the design-brief §3 names (hand-rolled data:-only parsing silently drops the id:).
• [2026-05-21] Contract converted to issue-scoped (issue #1). Frontmatter shape switched from module-scoped (module:/purpose:) to issue-scoped (target_module:/scope:/prd:) per CONTRACT-FORMAT §2.1.I. prd: block pins to issue body hash. Known parser stale-ness: contract_parser.py --validate ERRORs on issue-scoped frontmatter — CONTRACT-FORMAT §2.1.L H10, a documented Brokkr-side follow-up. Parser is a canonical sync, so we do NOT patch it locally. Treat parser ERROR-on-issue-scoped as expected until canonical bumps.
• [2026-05-21] Default issue-tracker labels seeded (17 total). Sleipnir gating, triage, type, resolution, Ratatoskr-specific area labels (sse-client, tui, cli, observability).
• [2026-05-21] Volva paraphrase + code-review across all 4 issues — calibration consistent. Paraphrase rounds flag 3-5 contract ambiguities per issue; code-review rounds flag 3-8 code-vs-contract drifts after TDD-passing implementation. Hit rates: #1 paraphrase 3-of-5 amended / code-review 4 findings; #2 3-of-5 / 3 findings; #3 5-of-5 / 5 findings; #4 5-of-5 / 8 findings. The post-TDD code-review consistently catches three classes of gap the test-author's hypotheses don't cover: PRE-assertion boundary drift, exception-payload truncation / never-rendered-to-user observability misses, and "tested the state but not whether the user can see it" gaps (issue #4's primary finding: TUI footer state stored but never rendered to a visible widget — same-model TDD would systematically miss this).
• [2026-05-21] Manual smoke is load-bearing — found a real defect tests couldn't. First wire-level smoke against personal Worldtree (post-TDD, post-Volva-code-review on #4) revealed httpx's default 5s read timeout killed the SSE connection mid-stream during mimir's thinking phase (~30s LLM latency >> 5s read timeout). The unit/contract test infrastructure (respx-mocked SSE wire) doesn't model real LLM latency, so the gap was invisible at the test layer. Fix: caller-owned httpx.AsyncClient constructed with timeout=httpx.Timeout(connect=10.0, read=None, write=10.0, pool=10.0); defense in depth: sse_client.stream_turn ERROR_ROUTING catches httpx.ReadTimeout → SseConnectionDropped. Three contracts amended in-place to document the timeout policy. Lesson: keep manual-smoke step in the per-issue cadence; mock-only validation is insufficient for streaming-against-real-server code. Re-smoke succeeded: [done] turn_id=88 model=qwen3.6-35-a3b duration_ms=2351. Wire-compat envelope (personal v0.16.2 vs ratatoskr's v0.19.0 pin) confirmed end-to-end.
• [2026-05-22] Issues #5/#6/#7 filed: per-user-agent support + TUI-startup-visibility + mid-stream-robustness. Discovered during 2026-05-22 mimir TUI conversation: long completion (turn 93, 1077 events consumed) crashed with JSONDecodeError("Expecting value: line 1 column 1 (char 0)") from json.loads('') on an empty-data: SSE frame. Diagnosis surfaced #7 (the crash). Earlier same day, ratatoskr --new --agent lofn failed with 422 end_user_id_required — surfacing #5 (--end-user-id flag needed for per-user agents). #6 (TUI alt-screen masks the diagnostic before user can read it) was a corollary observation. All three filed; user reordered to #7 first (highest-impact for daily TUI use).
• [2026-05-22] Issue #7 (MalformedSseData + empty-skip) implemented via TDD + Volva-code-reviewed + smoked. Contract → Volva paraphrase (4 ambiguities, all amended; INV-001 wording tightened around exact sse.data == '' rule, ordering-before-id-parse made explicit, test-description bug fixed) → TDD (6 tests, full vertical-slice ordering) → Volva code-review (3 findings — F1 test-gap probing internal last_sse_id non-advancement via post-skip drop, F2 contract precision around log-vs-propagate responsibility, F3 cli test tightening for raw='X' shape + truncation coverage; all amended) → smoke (3193-token completion against personal Worldtree confirmed clean termination; original crash unreproducible). Calibration milestone: issue #7 is the first issue with zero drift findings from Volva code-review — TDD caught all runtime behavior cleanly. The 3 findings were assertion-precision and architectural-correctness-of-wording, not behavioral. Hypothesis: the tighter the contract spec + the smaller the code surface, the more Volva's role shifts from "catch behavioral drift" to "tighten observability + wording". Calibration table now: #1 (4 findings, 3 drift + 1 test-gap), #2 (3, 1+1+1 precision), #3 (5, 3+1+1), #4 (8, 5+2+1), #7 (3, 0 drift + 2 test-gap + 1 precision).

For per-issue TDD implementation notes, Volva findings, and contract amendments, see the git log (commits 9703eb2..61c3941 carry the full per-issue trail with structured commit messages).

Tried and abandoned

Log of approaches that were tried and rejected, with rationale. Future-self defense against re-attempting the same cul-de-sac.

• [2026-05-20] rich + prompt_toolkit framework choice. Considered first (during initial shape draft). Volva flagged that §1 and §5 pulled in opposite directions: a real side-panel observability surface would silently become a widget framework reimplementation. Operator's debug-observability reframe sealed the flip to Textual. Don't re-attempt rich+pt unless the scope shrinks to transcript-first REPL (which would also flip back §5 to inline-log-presenter).
• [2026-05-20] In-tree at Worldtree/tools/ratatoskr/. Earlier draft committed to in-tree-with-import-direction-smoke-test. Rejected at operator-routing — separate dev team forces separate repo.
• [2026-05-20] New /persona/log SSE endpoint on Worldtree. Considered as alternative to file-tailing persona.log. Rejected — contract amendment + Vor round + AFK dispatch loop is weeks of consumer-side spec work for a debug feature file-tail handles in a day. Documented follow-up trigger in docs/design-brief.md §5: if a Worldtree-on-server / TUI-on-laptop debug case appears, the contract cost becomes worth paying.
• [2026-05-20] Cross-process Last-Event-ID resume. Considered — would require persisting per-session Last-Event-ID to ~/.config/ratatoskr/. Deferred to v2 if/when it turns out to matter; v1 ships "reconnect, not resume-across-process."
• [2026-05-21] RichLog widget with markup=True. Default impulse, but Rich interprets [xxx] spans as style markup and silently strips them. Every labeled stderr-style line — [cancel_failed], [done], [error], [busy], [worker_phase] — would render as just the content after the bracketed label, breaking the user-visible observability surface. Fix: markup=False. The post-Done Markdown rendering still works because rich.markdown.Markdown is a Renderable that ignores widget-level markup setting. Don't flip back to markup=True without first renaming every labeled-line format away from [bracket] notation.
• [2026-05-21] Querying self.query_one("#transcript", RichLog) from inside a Textual run_worker coroutine. Failed initially with NoMatches because the worker fires before the test's pilot.pause() allows the Input.Submitted handler to fully dispatch (and thus the widget tree to settle). Initial reactive fix: widen worker signature to take log as a parameter (passed from the handler). Volva code-review flagged this as contract drift (signature didn't match spec). Reverted to single-param signature. The real fix was test-side: add await pilot.pause() between inp.action_submit() and the polling loop in _submit_and_wait so the handler finishes dispatching before the worker reads the widget tree. Don't widen worker signatures to dodge test timing.
• [2026-05-21] TUI session-identity rendering via self.sub_title + self.hint plain attributes. Stored state but never rendered to a visible widget. The contract's "session-identity-always-visible" invariant was satisfied at the state-attribute level but not the user-visible-widget level. Tests asserted the attributes (which passed); Volva code-review flagged the gap. Fix: dedicated Static(id="identity") + Static(id="hint") widgets in compose; _set_hint() helper mirrors state → widget. Calibration evidence for the "TDD catches state, code-review catches whether the user can see it" pattern.
• [2026-05-22] json.loads(sse.data) unguarded against empty data. _iter_events unconditionally called json.loads on every dispatched ServerSentEvent. When httpx_sse surfaced a frame with id: present but data: empty (a known library-vs-spec divergence — RFC says don't dispatch; httpx_sse is permissive), json.loads('') raised JSONDecodeError → propagated through Textual's worker → app crash. Crashed mimir conversation at turn 93/seq 1078 after 1077 successful events. Fix: if sse.data == '': continue BEFORE _parse_sse_id (empty-data event with a malformed id is still a keepalive — don't reorder). Non-empty malformed data raises new MalformedSseData(raw[:200]). Don't reintroduce unconditional json.loads(sse.data); always pre-check for the empty case.