ratatoskr/persistent-memory.md

Persistent memory — ratatoskr

This file captures durable intent and supporting evidence (goals, decisions, foot-gun warnings, in-flight state) across context resets. Read it at session start; treat it as one input alongside CLAUDE.md and the auto-memory system, not as the single source of truth.

When durable state shifts enough to warrant capture, run /snapshot and commit alongside the next commit per the persistent-memory commit-along rule in CLAUDE.md.

---

Repo purpose

Ratatoskr is a dev-grade debug-observability TUI for Worldtree's Conversation API. The product IS the observability surface; chat is the input mechanism. Devs run Ratatoskr against a local Worldtree to watch a turn flow through every layer of the system, side-by-side, in one terminal: agent SSE stream, persona/Vili affect dispatch, tool calls, Bifrost handshake state, admin lifecycle events, optional raw server log.

Named after the squirrel that runs up and down Yggdrasil carrying messages between layers. On-the-nose Worldtree resonance (Yggdrasil = the World Tree).

Origin: althing ask from worldtree-dev (thread 01KS3R34XD3N6HMK91VXESHGW7, 2026-05-20) for the shape of a TUI Conversation API consumer. brokkr-smithy ran the shape pass; operator's reframe routed it as a new repo with a separate dev team rather than an in-tree Worldtree tool.

Current state / in-flight

Status: ratatoskr.sse_client implemented via TDD against issue #1's contract. 38/38 tests GREEN; ruff clean; boundary smoke (tests/test_no_worldtree_imports.py) still passes.

What's in the repo:

• docs/design-brief.md — the locked design (copy from brokkr-smithy/docs/ratatoskr-design-brief.md).
• docs/SPEC-PIN.md — Worldtree spec pin documentation + bump procedure.
• docs/conversation-api-spec.md — vendored Worldtree spec at the pinned SHA.
• docs/conversation_api.contract.md — vendored Worldtree server-side contract at the pinned SHA.
• docs/contracts/issues/1.contract.md — issue-scoped contract for issue #1 (#1). v2.1, complexity=high. target_module: ratatoskr.sse_client. prd: block pins to issue body SHA abcbc49467e86f1d at 2026-05-21T03:57:37+00:00. Four FN blocks: stream_turn, reconnect_turn, cancel_turn, _parse_sse_id. Drift check (scripts/contract_drift_check.py) returns clean.
• pyproject.toml — Python 3.12, hatchling, uv-managed. Deps: httpx, httpx-sse, textual. Dev deps: pytest, pytest-asyncio, respx, ruff, mypy, textual-dev, pyyaml (consumed by docs/contracts/contract_parser.py + scripts/contract_drift_check.py).
• src/ratatoskr/__init__.py + cli.py — stubs.
• src/ratatoskr/sse_client.py — implemented 2026-05-21 per docs/contracts/issues/1.contract.md. Four public entry points + nine typed Event variants + ten custom exceptions. Shared SSE-iteration logic (INV-002 + INV-003 + terminal-break) lives in private _iter_events(event_source, *, expected_turn_id) helper consumed by both stream_turn and reconnect_turn — expected_turn_id=None triggers "establish from first event" semantics, expected_turn_id=N triggers "first event is already a flip-candidate" semantics (the two-entry-point distinction Volva surfaced).
• tests/test_sse_client.py — 37 tests covering all four FN blocks' TESTS: entries verbatim (13 + 10 + 8 + 6). Real HTTP wire via respx mocks; SSE wire format constructed by helper _sse_chunk. Connection-drop test uses custom httpx.AsyncByteStream subclass that yields chunks then raises RemoteProtocolError.
• tests/test_no_worldtree_imports.py — boundary smoke test (passes; verified 2026-05-20).
• tests/snapshots/README.md — recording/replay convention for SSE snapshot tests.

What's NOT in the repo yet:

• Gitea remote — operator provided git@gitea.phasefinal.com:vh/ratatoskr.git on 2026-05-20; about to be added + first push at the same commit as this update.
• CLAUDE.md customization — currently using the canonical template's generic CLAUDE.md. The dev team may want to add Ratatoskr-specific conventions on first substantive work.

Branch: main. Remote: origin → git@gitea.phasefinal.com:vh/ratatoskr.git (added 2026-05-20).

Next natural moves:

1. TDD-implement ratatoskr.sessions per docs/contracts/issues/2.contract.md. Two FNs (create_session, list_sessions) + two dataclasses (SessionInfo, SessionPage). complexity=low; ~150 LOC. Tracer order: create_session first (unblocks --send --new), then list_sessions (for the eventual TUI picker). Optional: /volva-contract-review docs/contracts/issues/2.contract.md before implementing.
2. Build the --send stdout presenter under ratatoskr.cli — composes create_session + stream_turn into the non-interactive mode (design-brief §8b).
3. Record real SSE snapshot fixtures from a running Worldtree. --send --new is itself a recording probe — capture its outputs to tests/snapshots/ for replay-based regression coverage.
4. Textual TUI app shell — second presenter; multi-pane observability dashboard per design-brief §5.

Recent decisions

Chronological log of decisions with [YYYY-MM-DD] prefix. One line per decision. Captures rationale that won't be obvious from code alone.

• [2026-05-20] Project name Ratatoskr (squirrel on Yggdrasil — runs up and down carrying messages). Earlier candidate Andvari demoted on the cursed-ring association.
• [2026-05-20] Separate repo, separate dev team. Operator's call; the in-tree-at-Worldtree/tools/ alternative was considered and rejected to dogfood the API boundary.
• [2026-05-20] No Worldtree-source imports. Spec-only dependency. Triple version-skew mitigation: spec-pin in pyproject.toml + recorded-SSE snapshot tests + conformance smoke. Initial pin: 55101e909abcd2219833266b6f905c5bc956e0f0 (Worldtree v0.19.0). See docs/SPEC-PIN.md.
• [2026-05-20] Textual (not rich+prompt_toolkit). Driver: debug observability is the primary purpose, and a multi-pane dashboard with persistent side panes + independent scrollback is structurally application-shell-shaped. Volva consulted via cross-frontier second-opinion and converged on the same call.
• [2026-05-20] httpx-sse for SSE consumption. The server emits composite {turn_id}:{seq} id: lines (Worldtree INV-014) load-bearing for SSE-resume; hand-rolled data:-only parsing (the skaldsong pattern) silently drops these. Ratatoskr becomes the reference Python SSE-resume implementation.
• [2026-05-20] Persona-pane PII posture: label-don't-refuse. persona.log is process-wide; pane title flips between [Persona — PROCESS-WIDE] and [Persona — session <id>…] based on whether log lines carry session_id. Refuse-against-non-local was considered and rejected as paternalistic.
• [2026-05-20] Server-stdout pane: opt-in via --server-log <path>. No auto-detection of well-known paths.
• [2026-05-20] Two-stage Ctrl-C. First cancels in-flight turn server-side; second exits app. Ctrl-D bound to immediate exit.
• [2026-05-20] Single-session-per-launch + startup picker. No in-app /switch. CLI flags --session <id> and --new for scripted use. Session identity always visible in Textual footer.
• [2026-05-20] Markdown rendering default-on; --raw opt-out. Don't pre-design --no-stream-formatting (Volva: add only if streaming-markdown rendering is empirically ugly).
• [2026-05-20] Non-interactive --send mode. Single SSE consumer module, two presenters (TUI + stdout). Keeps Ratatoskr honest as an API consumer; useful for CI / scripted probes.
• [2026-05-20] First contract: ratatoskr.sse_client. Bundles stream_turn + reconnect_turn + cancel_turn + private _parse_sse_id into one module — the SSE-resume flow is coupled (cancel needs turn_id from the SSE wire id:, reconnect re-uses the same parsed SseId), so they share a contract. Hard invariant INV-002 makes the composite {turn_id}:{seq} id: parsing load-bearing — closes the foot-gun the design-brief §3 names (hand-rolled data:-only parsing silently drops the id:). v2.1 test categories adversarial/scenario/trace used freely; parser warns but format spec §2.1.E permits them.
• [2026-05-21] Contract converted to issue-scoped (issue #1). Moved docs/contracts/sse_client.contract.md → docs/contracts/issues/1.contract.md. Frontmatter shape switched from module-scoped (module:/purpose:) to issue-scoped (target_module:/scope:/prd:) per CONTRACT-FORMAT §2.1.I. prd: block pins to issue #1's body hash (abcbc49467e86f1d). scripts/contract_drift_check.py returns clean. Known parser stale-ness: contract_parser.py --validate ERRORs on issue-scoped frontmatter (missing module:/purpose:) — this is CONTRACT-FORMAT §2.1.L H10, a documented Brokkr-side follow-up. Parser is a canonical sync, so we do NOT patch it locally (would drift from canonical). Treat parser ERROR-on-issue-scoped as expected until the canonical bumps.
• [2026-05-21] Default issue-tracker labels seeded (17 total). Sleipnir gating (ready-for-agent, blocked-needs-contract, blocked-needs-dependency), triage (needs-triage, needs-architect-decision, needs-info), type (bug, enhancement, task, documentation), resolution (duplicate, wontfix, invalid), Ratatoskr-specific area (sse-client, tui, cli, observability).
• [2026-05-21] ratatoskr.sse_client implemented via TDD against issue #1's contract. 37 contract-listed tests authored + GREEN per the tracer-bullet vertical-slice ordering (_parse_sse_id → stream_turn → reconnect_turn → cancel_turn). Refactor pass extracted _iter_events helper to dedupe INV-002 + INV-003 + terminal-break logic across stream_turn and reconnect_turn; expected_turn_id=None vs expected_turn_id=N distinguishes the two entry-point semantics Volva surfaced. Notable choices made during implementation: (a) regex ^-?\d+$ pre-check in _parse_sse_id to reject whitespace before int() (Python's int(" 3 ") would silently strip — this kept the strict-no-whitespace test honest); (b) _DropAfter AsyncByteStream subclass in tests to simulate mid-stream RemoteProtocolError; (c) ToolResult.result and ToolStart.arguments typed as Any (server JSON varies); (d) ruff line-length=100 (per pyproject) forced some test docstrings to be tighter than v0 draft.
• [2026-05-21] Issue #2 + contract: ratatoskr.sessions. Scope is narrow — create_session (POST /sessions) + list_sessions (GET /sessions, cursor-paginated) + shared SessionInfo and SessionPage frozen dataclasses. Bundles two endpoints in one contract because they share the response envelope shape; splitting would duplicate the dataclass. Bifrost binding (Worldtree issue #160), ephemeral sessions, GET /sessions/{id}, PATCH, DELETE, and GET /sessions/{id}/messages (history) are explicitly out of scope (codified in the contract's ## Out of scope H2 — first contract in this repo to carry that section, so future Volva consults resolve cleanly via the default path instead of needing --out-of-scope overrides). prd: pinned to issue #2 body SHA 01fbbd52b6d90eb0 at 2026-05-21T04:45:06+00:00; drift check clean. dependencies: lists issue #1 as a convention-dependency (no code import; same API-consumption posture).
• [2026-05-21] Volva paraphrase round on docs/contracts/issues/2.contract.md. Volva flagged 5 ambiguities; operator approved amendments to 3 of them. (1) tags/archived/name defaulting semantics now explicit: tags: list[str] (default []), archived: bool (default False), name: str | None (default None); INV-001/INV-002 + STEPS aligned. (2) include_archived_query test tightened: default → URL has NO include_archived param at all (was "no param OR explicit false" — softened the assertion against STEP 2's prescriptive behavior). (5) metadata populated-vs-defaulted slippage resolved: INV-001 + INV-002 now spell out the defensive body.get("metadata", {}) default for spec drift tolerance. Volva flags #3 (exception .body sensitivity) and #4 (assert for runtime validation) reviewed and kept as-is — both intentional and consistent with issue #1's precedent. Drift check still clean (amendments don't touch the pinned issue body).
• [2026-05-21] Volva code-vs-contract review round on ratatoskr.sse_client. Volva flagged 4 findings (3 drifts + 1 test-gap), all code-side "fix it" recommendations: (1) _iter_events fell off cleanly on EOF before terminal, violating INV-001 ("MUST NOT raise StopAsyncIteration before a terminal event arrives unless connection drops"); fix tracks terminal_seen flag and raises SseConnectionDropped on clean-EOF-without-terminal. (2) Both SseConnectFailed.body and CancelFailed.body stored full response bytes; ERROR_ROUTING specified truncation to [:1024]; fix truncates in __init__ before storing. (3) _parse_sse_id PRE-001 specified assert isinstance(raw, str), but code called .split(":") directly (incidental AttributeError on non-str); fix adds the assert. (4) Test-gap on cancel_turn's "other status → CancelFailed" branch; fix adds a 503 test with >1024-byte body that double-covers finding #2. Meta-note: Volva said TDD caught the main happy/adversarial shape; the misses were "negative space" cases (clean EOF, exception payload truncation, untested generic cancel branch) — calibration evidence that cross-model review pulls weight on the same-model author's blind spots. 43 tests GREEN post-fix (42 sse_client + 1 boundary), ruff clean.
• [2026-05-21] Volva paraphrase round on docs/contracts/issues/1.contract.md. Volva flagged 5 ambiguities; operator approved amendments to 3 of them. (1) reconnect_turn STEP 2 punt resolved: signature now carries content: str; STEP 2 body is json={"content": content} matching spec §Reconnect flow example verbatim. Spec line 732 makes the agent's tools+LLM run "exactly once regardless of disconnects/reconnects" — the content is a wire-schema requirement, not re-processed server-side. (2) _parse_sse_id tightened: turn_id ≥ 1 AND seq ≥ 1 (was ≥ 0); spec §SSE id format line 705 explicitly states seq starts at 1, and turn_id is SQLite autoincrement (≥1). Test happy_zero_seq flipped to zero_seq [adversarial]; new zero_turn_id + negative_seq adversarial tests added. (3) INV-003 clarified to spell out the two-entry-point semantics: stream_turn establishes turn_id from the first event (first event always yields); reconnect_turn parses the expected turn_id FROM last_event_id BEFORE the connection opens, so the first server event is already a flip-candidate and is NOT yielded on mismatch. Volva flags #3 (MalformedSseId-vs-ValueError split) and #5 (exactly-one-terminal as server-assumed) noted but kept as-is — deliberate distinctions. Drift check still clean against issue #1 (amending the contract doesn't touch the pinned issue body).

Tried and abandoned

Log of approaches that were tried and rejected, with rationale. Future-self defense against re-attempting the same cul-de-sac.

• [2026-05-20] rich + prompt_toolkit framework choice. Considered first (during initial shape draft). Volva flagged that §1 and §5 pulled in opposite directions: a real side-panel observability surface would silently become a widget framework reimplementation. Operator's debug-observability reframe sealed the flip to Textual. Don't re-attempt rich+pt unless the scope shrinks to transcript-first REPL (which would also flip back §5 to inline-log-presenter).
• [2026-05-20] In-tree at Worldtree/tools/ratatoskr/. Earlier draft committed to in-tree-with-import-direction-smoke-test. Rejected at operator-routing — separate dev team forces separate repo.
• [2026-05-20] New /persona/log SSE endpoint on Worldtree. Considered as alternative to file-tailing persona.log. Rejected — contract amendment + Vor round + AFK dispatch loop is weeks of consumer-side spec work for a debug feature file-tail handles in a day. Documented follow-up trigger in docs/design-brief.md §5: if a Worldtree-on-server / TUI-on-laptop debug case appears, the contract cost becomes worth paying.
• [2026-05-20] Cross-process Last-Event-ID resume. Considered — would require persisting per-session Last-Event-ID to ~/.config/ratatoskr/. Deferred to v2 if/when it turns out to matter; v1 ships "reconnect, not resume-across-process."