ratatoskr/persistent-memory.md

# Persistent memory — ratatoskr

This file captures durable intent and supporting evidence (goals, decisions,
foot-gun warnings, in-flight state) across context resets. Read it at session
start; treat it as one input alongside `CLAUDE.md` and the auto-memory system,
not as the single source of truth.

When durable state shifts enough to warrant capture, run `/snapshot` and
commit alongside the next commit per the persistent-memory commit-along rule
in `CLAUDE.md`.

---

## Repo purpose

Ratatoskr is a **dev-grade debug-observability TUI** for Worldtree's
Conversation API. The product IS the observability surface; chat is the
input mechanism. Devs run Ratatoskr against a local Worldtree to watch a
turn flow through every layer of the system, side-by-side, in one terminal:
agent SSE stream, persona/Vili affect dispatch, tool calls, Bifrost
handshake state, admin lifecycle events, optional raw server log.

Named after the squirrel that runs up and down Yggdrasil carrying messages
between layers. On-the-nose Worldtree resonance (Yggdrasil = the World Tree).

Origin: althing ask from worldtree-dev (thread `01KS3R34XD3N6HMK91VXESHGW7`,
2026-05-20) for the shape of a TUI Conversation API consumer. brokkr-smithy
ran the shape pass; operator's reframe routed it as a new repo with a
separate dev team rather than an in-tree Worldtree tool.

## Current state / in-flight

**Status: `ratatoskr.sse_client` implemented via TDD against issue #1's contract.** 38/38 tests GREEN; ruff clean; boundary smoke (`tests/test_no_worldtree_imports.py`) still passes.

What's in the repo:
- `docs/design-brief.md` — the locked design (copy from `brokkr-smithy/docs/ratatoskr-design-brief.md`).
- `docs/SPEC-PIN.md` — Worldtree spec pin documentation + bump procedure.
- `docs/conversation-api-spec.md` — vendored Worldtree spec at the pinned SHA.
- `docs/conversation_api.contract.md` — vendored Worldtree server-side contract at the pinned SHA.
- `docs/contracts/issues/1.contract.md` — **issue-scoped contract for issue #1** (https://gitea.phasefinal.com/vh/ratatoskr/issues/1). v2.1, complexity=high. `target_module: ratatoskr.sse_client`. `prd:` block pins to issue body SHA `abcbc49467e86f1d` at `2026-05-21T03:57:37+00:00`. Four FN blocks: `stream_turn`, `reconnect_turn`, `cancel_turn`, `_parse_sse_id`. Drift check (`scripts/contract_drift_check.py`) returns clean.
- `pyproject.toml` — Python 3.12, hatchling, uv-managed. Deps: httpx, httpx-sse, textual. Dev deps: pytest, pytest-asyncio, respx, ruff, mypy, textual-dev, pyyaml (consumed by `docs/contracts/contract_parser.py` + `scripts/contract_drift_check.py`).
- `src/ratatoskr/__init__.py` + `cli.py` — stubs.
- `src/ratatoskr/sse_client.py` — **implemented 2026-05-21** per `docs/contracts/issues/1.contract.md`. Four public entry points + nine typed Event variants + ten custom exceptions. Shared SSE-iteration logic (INV-002 + INV-003 + terminal-break) lives in private `_iter_events(event_source, *, expected_turn_id)` helper consumed by both `stream_turn` and `reconnect_turn` — `expected_turn_id=None` triggers "establish from first event" semantics, `expected_turn_id=N` triggers "first event is already a flip-candidate" semantics (the two-entry-point distinction Volva surfaced).
- `tests/test_sse_client.py` — 37 tests covering all four FN blocks' TESTS: entries verbatim (13 + 10 + 8 + 6). Real HTTP wire via respx mocks; SSE wire format constructed by helper `_sse_chunk`. Connection-drop test uses custom `httpx.AsyncByteStream` subclass that yields chunks then raises `RemoteProtocolError`.
- `tests/test_no_worldtree_imports.py` — boundary smoke test (passes; verified 2026-05-20).
- `tests/snapshots/README.md` — recording/replay convention for SSE snapshot tests.

What's NOT in the repo yet:
- Gitea remote — operator provided `git@gitea.phasefinal.com:vh/ratatoskr.git` on 2026-05-20; about to be added + first push at the same commit as this update.
- CLAUDE.md customization — currently using the canonical template's
  generic CLAUDE.md. The dev team may want to add Ratatoskr-specific
  conventions on first substantive work.

**Branch:** `main`. Remote: `origin → git@gitea.phasefinal.com:vh/ratatoskr.git` (added 2026-05-20).

**Next natural moves:**
1. TDD-implement `ratatoskr.sessions` per `docs/contracts/issues/2.contract.md`. Two FNs (`create_session`, `list_sessions`) + two dataclasses (`SessionInfo`, `SessionPage`). complexity=low; ~150 LOC. Tracer order: `create_session` first (unblocks `--send --new`), then `list_sessions` (for the eventual TUI picker). Optional: `/volva-contract-review docs/contracts/issues/2.contract.md` before implementing.
2. Build the `--send` stdout presenter under `ratatoskr.cli` — composes `create_session` + `stream_turn` into the non-interactive mode (design-brief §8b).
3. Record real SSE snapshot fixtures from a running Worldtree. `--send --new` is itself a recording probe — capture its outputs to `tests/snapshots/` for replay-based regression coverage.
4. Textual TUI app shell — second presenter; multi-pane observability dashboard per design-brief §5.

## Recent decisions

Chronological log of decisions with `[YYYY-MM-DD]` prefix. One line per
decision. Captures rationale that won't be obvious from code alone.

- `[2026-05-20]` Project name **Ratatoskr** (squirrel on Yggdrasil — runs up and down carrying messages). Earlier candidate Andvari demoted on the cursed-ring association.
- `[2026-05-20]` **Separate repo, separate dev team.** Operator's call; the in-tree-at-Worldtree/tools/ alternative was considered and rejected to dogfood the API boundary.
- `[2026-05-20]` **No Worldtree-source imports.** Spec-only dependency. Triple version-skew mitigation: spec-pin in pyproject.toml + recorded-SSE snapshot tests + conformance smoke. Initial pin: `55101e909abcd2219833266b6f905c5bc956e0f0` (Worldtree v0.19.0). See `docs/SPEC-PIN.md`.
- `[2026-05-20]` **Textual** (not rich+prompt_toolkit). Driver: debug observability is the primary purpose, and a multi-pane dashboard with persistent side panes + independent scrollback is structurally application-shell-shaped. Volva consulted via cross-frontier second-opinion and converged on the same call.
- `[2026-05-20]` **`httpx-sse`** for SSE consumption. The server emits composite `{turn_id}:{seq}` `id:` lines (Worldtree INV-014) load-bearing for SSE-resume; hand-rolled `data:`-only parsing (the skaldsong pattern) silently drops these. Ratatoskr becomes the reference Python SSE-resume implementation.
- `[2026-05-20]` **Persona-pane PII posture: label-don't-refuse.** `persona.log` is process-wide; pane title flips between `[Persona — PROCESS-WIDE]` and `[Persona — session <id>…]` based on whether log lines carry session_id. Refuse-against-non-local was considered and rejected as paternalistic.
- `[2026-05-20]` **Server-stdout pane: opt-in via `--server-log <path>`.** No auto-detection of well-known paths.
- `[2026-05-20]` **Two-stage Ctrl-C.** First cancels in-flight turn server-side; second exits app. Ctrl-D bound to immediate exit.
- `[2026-05-20]` **Single-session-per-launch + startup picker.** No in-app `/switch`. CLI flags `--session <id>` and `--new` for scripted use. Session identity always visible in Textual footer.
- `[2026-05-20]` **Markdown rendering default-on; `--raw` opt-out.** Don't pre-design `--no-stream-formatting` (Volva: add only if streaming-markdown rendering is empirically ugly).
- `[2026-05-20]` **Non-interactive `--send` mode.** Single SSE consumer module, two presenters (TUI + stdout). Keeps Ratatoskr honest as an API consumer; useful for CI / scripted probes.
- `[2026-05-20]` **First contract: `ratatoskr.sse_client`.** Bundles `stream_turn` + `reconnect_turn` + `cancel_turn` + private `_parse_sse_id` into one module — the SSE-resume flow is coupled (cancel needs `turn_id` from the SSE wire `id:`, reconnect re-uses the same parsed `SseId`), so they share a contract. Hard invariant INV-002 makes the composite `{turn_id}:{seq}` `id:` parsing load-bearing — closes the foot-gun the design-brief §3 names (hand-rolled `data:`-only parsing silently drops the `id:`). v2.1 test categories `adversarial`/`scenario`/`trace` used freely; parser warns but format spec §2.1.E permits them.
- `[2026-05-21]` **Contract converted to issue-scoped (issue #1).** Moved `docs/contracts/sse_client.contract.md` → `docs/contracts/issues/1.contract.md`. Frontmatter shape switched from module-scoped (`module:`/`purpose:`) to issue-scoped (`target_module:`/`scope:`/`prd:`) per CONTRACT-FORMAT §2.1.I. `prd:` block pins to issue #1's body hash (`abcbc49467e86f1d`). `scripts/contract_drift_check.py` returns clean. **Known parser stale-ness**: `contract_parser.py --validate` ERRORs on issue-scoped frontmatter (missing `module:`/`purpose:`) — this is CONTRACT-FORMAT §2.1.L H10, a documented Brokkr-side follow-up. Parser is a canonical sync, so we do NOT patch it locally (would drift from canonical). Treat parser ERROR-on-issue-scoped as expected until the canonical bumps.
- `[2026-05-21]` **Default issue-tracker labels seeded** (17 total). Sleipnir gating (`ready-for-agent`, `blocked-needs-contract`, `blocked-needs-dependency`), triage (`needs-triage`, `needs-architect-decision`, `needs-info`), type (`bug`, `enhancement`, `task`, `documentation`), resolution (`duplicate`, `wontfix`, `invalid`), Ratatoskr-specific area (`sse-client`, `tui`, `cli`, `observability`).
- `[2026-05-21]` **`ratatoskr.sse_client` implemented via TDD against issue #1's contract.** 37 contract-listed tests authored + GREEN per the tracer-bullet vertical-slice ordering (`_parse_sse_id` → `stream_turn` → `reconnect_turn` → `cancel_turn`). Refactor pass extracted `_iter_events` helper to dedupe INV-002 + INV-003 + terminal-break logic across `stream_turn` and `reconnect_turn`; `expected_turn_id=None` vs `expected_turn_id=N` distinguishes the two entry-point semantics Volva surfaced. Notable choices made during implementation: (a) regex `^-?\d+$` pre-check in `_parse_sse_id` to reject whitespace before `int()` (Python's `int(" 3 ")` would silently strip — this kept the strict-no-whitespace test honest); (b) `_DropAfter` AsyncByteStream subclass in tests to simulate mid-stream `RemoteProtocolError`; (c) ToolResult.result and ToolStart.arguments typed as `Any` (server JSON varies); (d) ruff line-length=100 (per pyproject) forced some test docstrings to be tighter than v0 draft.
- `[2026-05-21]` **Issue #2 + contract: `ratatoskr.sessions`.** Scope is narrow — `create_session` (POST /sessions) + `list_sessions` (GET /sessions, cursor-paginated) + shared `SessionInfo` and `SessionPage` frozen dataclasses. Bundles two endpoints in one contract because they share the response envelope shape; splitting would duplicate the dataclass. Bifrost binding (Worldtree issue #160), ephemeral sessions, `GET /sessions/{id}`, `PATCH`, `DELETE`, and `GET /sessions/{id}/messages` (history) are explicitly out of scope (codified in the contract's `## Out of scope` H2 — first contract in this repo to carry that section, so future Volva consults resolve cleanly via the default path instead of needing `--out-of-scope` overrides). `prd:` pinned to issue #2 body SHA `01fbbd52b6d90eb0` at `2026-05-21T04:45:06+00:00`; drift check clean. `dependencies:` lists issue #1 as a convention-dependency (no code import; same API-consumption posture).
- `[2026-05-21]` **Volva paraphrase round on `docs/contracts/issues/2.contract.md`.** Volva flagged 5 ambiguities; operator approved amendments to 3 of them. (1) `tags`/`archived`/`name` defaulting semantics now explicit: `tags: list[str]` (default `[]`), `archived: bool` (default `False`), `name: str | None` (default `None`); INV-001/INV-002 + STEPS aligned. (2) `include_archived_query` test tightened: default → URL has NO `include_archived` param at all (was "no param OR explicit false" — softened the assertion against STEP 2's prescriptive behavior). (5) `metadata` populated-vs-defaulted slippage resolved: INV-001 + INV-002 now spell out the defensive `body.get("metadata", {})` default for spec drift tolerance. Volva flags #3 (exception `.body` sensitivity) and #4 (`assert` for runtime validation) reviewed and kept as-is — both intentional and consistent with issue #1's precedent. Drift check still clean (amendments don't touch the pinned issue body).
- `[2026-05-21]` **Volva code-vs-contract review round on `ratatoskr.sse_client`.** Volva flagged 4 findings (3 drifts + 1 test-gap), all code-side "fix it" recommendations: (1) `_iter_events` fell off cleanly on EOF before terminal, violating INV-001 ("MUST NOT raise StopAsyncIteration before a terminal event arrives unless connection drops"); fix tracks `terminal_seen` flag and raises `SseConnectionDropped` on clean-EOF-without-terminal. (2) Both `SseConnectFailed.body` and `CancelFailed.body` stored full response bytes; ERROR_ROUTING specified truncation to `[:1024]`; fix truncates in `__init__` before storing. (3) `_parse_sse_id` PRE-001 specified `assert isinstance(raw, str)`, but code called `.split(":")` directly (incidental `AttributeError` on non-str); fix adds the assert. (4) Test-gap on cancel_turn's "other status → CancelFailed" branch; fix adds a 503 test with >1024-byte body that double-covers finding #2. Meta-note: Volva said TDD caught the main happy/adversarial shape; the misses were "negative space" cases (clean EOF, exception payload truncation, untested generic cancel branch) — calibration evidence that cross-model review pulls weight on the same-model author's blind spots. 43 tests GREEN post-fix (42 sse_client + 1 boundary), ruff clean.
- `[2026-05-21]` **Volva paraphrase round on `docs/contracts/issues/1.contract.md`.** Volva flagged 5 ambiguities; operator approved amendments to 3 of them. (1) `reconnect_turn` STEP 2 punt resolved: signature now carries `content: str`; STEP 2 body is `json={"content": content}` matching spec §Reconnect flow example verbatim. Spec line 732 makes the agent's tools+LLM run "exactly once regardless of disconnects/reconnects" — the `content` is a wire-schema requirement, not re-processed server-side. (2) `_parse_sse_id` tightened: `turn_id ≥ 1` AND `seq ≥ 1` (was `≥ 0`); spec §SSE id format line 705 explicitly states `seq` starts at 1, and `turn_id` is SQLite autoincrement (≥1). Test `happy_zero_seq` flipped to `zero_seq [adversarial]`; new `zero_turn_id` + `negative_seq` adversarial tests added. (3) INV-003 clarified to spell out the two-entry-point semantics: `stream_turn` establishes `turn_id` from the first event (first event always yields); `reconnect_turn` parses the expected `turn_id` FROM `last_event_id` BEFORE the connection opens, so the first server event is already a flip-candidate and is NOT yielded on mismatch. Volva flags #3 (MalformedSseId-vs-ValueError split) and #5 (exactly-one-terminal as server-assumed) noted but kept as-is — deliberate distinctions. Drift check still clean against issue #1 (amending the contract doesn't touch the pinned issue body).

## Tried and abandoned

Log of approaches that were tried and rejected, with rationale. Future-self
defense against re-attempting the same cul-de-sac.

- `[2026-05-20]` **rich + prompt_toolkit framework choice.** Considered first (during initial shape draft). Volva flagged that §1 and §5 pulled in opposite directions: a real side-panel observability surface would silently become a widget framework reimplementation. Operator's debug-observability reframe sealed the flip to Textual. Don't re-attempt rich+pt unless the scope shrinks to transcript-first REPL (which would also flip back §5 to inline-log-presenter).
- `[2026-05-20]` **In-tree at Worldtree/tools/ratatoskr/.** Earlier draft committed to in-tree-with-import-direction-smoke-test. Rejected at operator-routing — separate dev team forces separate repo.
- `[2026-05-20]` **New `/persona/log` SSE endpoint on Worldtree.** Considered as alternative to file-tailing `persona.log`. Rejected — contract amendment + Vor round + AFK dispatch loop is weeks of consumer-side spec work for a debug feature file-tail handles in a day. Documented follow-up trigger in `docs/design-brief.md` §5: if a Worldtree-on-server / TUI-on-laptop debug case appears, the contract cost becomes worth paying.
- `[2026-05-20]` **Cross-process Last-Event-ID resume.** Considered — would require persisting per-session Last-Event-ID to `~/.config/ratatoskr/`. Deferred to v2 if/when it turns out to matter; v1 ships "reconnect, not resume-across-process."