--- contract_version: "2.1" target_module: "ratatoskr.sse_client" scope: "Stop crashing mid-stream when an SSE event with empty `data:` is dispatched. Two-rule fix in `_iter_events`: (a) when `sse.data` is empty, SKIP the event silently (treat as keepalive per SSE spec — don't yield, don't update `last_sse_id`, don't crash); (b) when `sse.data` is non-empty but fails `json.loads`, raise a new `MalformedSseData(raw=sse.data[:200])` exception (mirrors the existing `MalformedSseId` shape — wire-level protocol error, surfaceable to presenters). Presenters (`cli._run_turn`, `tui._stream_turn_worker`) gain a `MalformedSseData` case in their ERROR_ROUTING. No new modules; in-place amendments to issues #1 (sse_client), #3 (cli), #4 (tui). Discovered by mid-stream crash 2026-05-22: turn 93 emitted 1077 events successfully, then event 1078 arrived with `id: 93:1078` but empty `data:`; `json.loads('')` raised `JSONDecodeError`, propagated through Textual's worker, crashed the app." depends_on: - "httpx" - "httpx-sse" used_by: [] language: "python" complexity: "low" estimated_loc: 25 confidence: 0.9 assumptions: - "`httpx_sse.ServerSentEvent` surfaces events with an `id:` line but no `data:` line (or empty `data:` content) as a `ServerSentEvent` instance with `data=''`. This is a known behavior-vs-spec divergence in `httpx_sse` — the SSE RFC says empty-data events SHOULD NOT be dispatched, but the library is permissive. We handle the library's actual behavior, not the RFC's prescription." - "Worldtree's spec declares every event has JSON-shaped `data:`. An empty-data event from Worldtree is either (a) a server bug, (b) a wire-level keepalive that should have been suppressed by `httpx_sse`, or (c) mid-stream truncation. Cause is upstream and out of scope; ratatoskr's job is to not crash." - "Non-empty data that fails `json.loads` is a true protocol error (server emitted malformed JSON). Surface as a named exception so presenters can render a clean error label, not propagate raw `JSONDecodeError`." - "Whitespace-only `sse.data` (e.g., `' '`, `'\\n'`) IS non-empty by Python truthiness and `json.loads` fails on it. Treat as malformed (raise) rather than skip. The skip path is only for the exact empty-string case (`sse.data == ''`), matching what `httpx_sse` produces for a truly data-less event." - "Truncation of `raw` to 200 chars in `MalformedSseData` mirrors `MalformedSseId.raw = raw[:64]` (sse_client.py:132) but allows more chars because JSON payloads are typically longer than wire IDs. Keep the exception body bounded so it stays loggable." open_questions: - "Telemetry on skipped empty-data events: should ratatoskr emit a stderr label like `[sse_keepalive]` so the operator can see they're happening? Draft: no for v1. Empty-data events are silently dropped; if frequency turns out to be high (or zero, suggesting the bug is actually elsewhere), revisit. The TUI's RichLog already gets all yielded events labeled, so a skip is genuinely invisible by design." - "Worldtree-side investigation: is mimir emitting empty-data events legitimately (keepalive) or is this a bug? Worth filing on worldtree-dev's tracker if frequency is non-trivial — but ratatoskr's robustness fix is independent of the upstream root cause." prd: issue: 7 issue_url: "https://gitea.phasefinal.com/vh/ratatoskr/issues/7" body_sha256_16: "155e21850365543f" lock_in_comment_id: null lock_in_sha256_16: null lock_in_at: null pinned_at: "2026-05-22T05:22:38+00:00" dependencies: - issue: 1 path: "src/ratatoskr/sse_client.py" reason: "In-place contract amendment: `_iter_events` STEPS gain the empty-skip + malformed-raise rules; new `MalformedSseData` exception added to the public surface; `stream_turn` ERROR_ROUTING lists `MalformedSseData` alongside `MalformedSseId`; new TESTS for the skip and the raise paths." - issue: 3 path: "src/ratatoskr/cli.py" reason: "In-place contract amendment: `_run_turn` ERROR_ROUTING gains a `MalformedSseData` case → stderr `[malformed_sse_data] raw=...` + exit code 22 (protocol-failure bucket, same as MalformedSseId/TurnIdFlip)." - issue: 4 path: "src/ratatoskr/tui.py" reason: "In-place contract amendment: `_stream_turn_worker` ERROR_ROUTING gains a `MalformedSseData` case → transcript `[malformed_sse_data] raw=...` line + state → idle per INV-008 (mid-session errors don't exit the app)." --- # Mid-stream robustness — empty SSE data is skipped, malformed is raised cleanly ## Context During a TUI conversation with `mimir` on 2026-05-22, ratatoskr consumed 1077 SSE events successfully then crashed on event 1078: ``` JSONDecodeError: Expecting value: line 1 column 1 (char 0) at src/ratatoskr/sse_client.py:301 in event = _envelope_for_type(json.loads(sse.data), sse_id=sse_id) sse = ServerSentEvent(event='message', id='93:1078') data = '' # ← EMPTY but id present sse_id = SseId(turn_id=93, seq=1078) ``` `_iter_events` unconditionally calls `json.loads(sse.data)` on every dispatched event. When `httpx_sse` surfaces an event with empty `data`, the JSON parser fails, the exception propagates through `stream_turn` → `_stream_turn_worker` → Textual's `Worker` → app abort → traceback dumps to the operator's terminal. The turn is lost. The SSE spec allows empty-data events (commonly keepalives). The Worldtree spec says every event carries JSON `data:`, so an empty-data frame from Worldtree is either a server bug, an `httpx_sse` permissive-parsing quirk, or mid-stream truncation. The cause is upstream; ratatoskr's responsibility is to not crash the operator's conversation over a single anomalous frame. This issue adds two robustness rules in `_iter_events`: empty data is silently skipped (keepalive semantics); non-empty malformed JSON raises a new named exception (`MalformedSseData`) that presenters surface as a clean error label, not a raw traceback. ## Data flow **Input change:** - `_iter_events` continues to consume `ServerSentEvent` objects from `httpx_sse.EventSource`. Now distinguishes three cases on `sse.data`: - **Empty (`sse.data == ''`)**: skip — don't yield, don't update `last_sse_id`, don't increment any counter. Continue to next. - **Non-empty + parses as JSON**: yield as today (typed Event per `_envelope_for_type`). - **Non-empty + JSONDecodeError**: raise `MalformedSseData(raw=sse.data[:200])`. **Output change:** - A stream containing K empty-data events interleaved with N regular events yields N events (the K empty are invisible to the caller). - A stream containing a malformed-data event raises `MalformedSseData` the same way `MalformedSseId` is raised today (consumer catches in ERROR_ROUTING). **No side effects:** purely an iterator-internal behavior change. No new disk I/O, no new HTTP calls, no new state. ## Invariants - **INV-001 [hard]**: `_iter_events` MUST silently skip events where `sse.data == ''` (exact empty string equality — NOT truthy-check, NOT a stripped-whitespace check, NOT a "looks like a keepalive" pattern). Whitespace-only data (`' '`, `'\n'`, `'\t'`) is NOT empty and MUST go through the malformed-raise path (INV-002). The "keepalive" framing is intent-of-the-skip, not a generalized rule — implementers MUST apply the literal `sse.data == ''` check. Skip means: don't yield, don't update `last_sse_id`, don't set `terminal_seen` to True, don't call `_parse_sse_id` on `sse.id`, don't raise. The iteration continues to the next event from `event_source.aiter_sse()`. (No other counters or accumulators exist in `_iter_events` — `last_sse_id` and `terminal_seen` are the only state to preserve. Future implementations must not silently grow the state set without re-evaluating this invariant.) - **INV-002 [hard]**: `_iter_events` MUST raise `MalformedSseData` (NOT `JSONDecodeError`, NOT `SseConnectionDropped`) when `sse.data` is non-empty but fails `json.loads(...)`. Truncation: `raw=sse.data[:200]` in the exception. The raw field is the only payload-derived data on the exception — keep it bounded. - **INV-003 [hard]**: Skip does NOT advance `last_sse_id`. Rationale: `last_sse_id` is what callers use for `reconnect_turn` on `SseConnectionDropped`. A skipped empty-data event was not user-visible content; the caller should resume from the last *real* event, not from the skipped one. (If `last_sse_id` advanced through the skip and a drop happened immediately after, the server would replay nothing of interest — the operator loses no data, but the precision is wrong.) - **INV-004 [hard]**: `MalformedSseData` is a sibling of `MalformedSseId` in the public exception surface. Both indicate wire-level protocol violations from the server; both abort the stream (not recoverable in process); both belong to exit-code bucket 22 (protocol failures) in cli's table. `MalformedSseData.raw` mirrors `MalformedSseId.raw` — both stored as truncated `str` attributes on the exception. - **INV-005 [hard]**: No `core.*` / `worldtree.*` imports (existing boundary; unchanged). ## Out of scope - **Diagnosing upstream root cause.** Whether Worldtree is emitting empty-data events legitimately (keepalive) or as a server bug is a worldtree-dev question worth filing separately if frequency is non-trivial. Ratatoskr's robustness fix is independent. - **Telemetry on skipped events.** No stderr-label or counter for `[sse_keepalive]`-style. v1 silent skip; revisit if frequency turns out to matter for observability. - **Auto-reconnect on `MalformedSseData`.** `reconnect_turn` exists for clean drops; whether to invoke it after a malformed frame is policy and currently the consumer's call. Presenters today don't auto- reconnect on any error class; that pattern stays unchanged here. - **Whitespace-only data handling beyond malformed-raise.** Treating `' '` or `'\\n'` as malformed (per the raise rule) is the v1 stance; if a real-world server emits whitespace keepalives, file a follow-up. - **`SseConnectionDropped` reclassification.** Existing wire-level exceptions (`ReadError`, `RemoteProtocolError`, `ReadTimeout`, clean-EOF-before-terminal) continue to map to `SseConnectionDropped`. `MalformedSseData` is a distinct class: payload-level malformation, not transport-level failure. ## Constraints - **[compatibility]** Spec pin unchanged. The empty-data handling is a client-side robustness change; nothing about the wire contract moves. - **[performance]** A skipped event costs one extra branch + continue in the loop. Negligible. - **[security]** `MalformedSseData.raw` truncates to 200 chars to bound the exception body — same hygiene as `MalformedSseId.raw[:64]` and `SseConnectFailed.body[:1024]`. - **[style]** Exception class follows existing pattern (frozen-ish: `__init__` stores `self.raw`, `super().__init__(message)`). Ruff line-length=100. --- ## In-place amendments (the work) This issue's contract is small because the real work is amendments to issues #1, #3, #4. The amendments are pinned here so reviewers see the whole change in one place; the actual contract files are amended in-place as part of this issue's commit. ### Issue #1 (`ratatoskr.sse_client`) amendments **New public exception:** ```python class MalformedSseData(Exception): """Raised when an SSE event's `data:` payload is non-empty but fails JSON parsing.""" def __init__(self, raw: str) -> None: super().__init__(f"malformed SSE data: {raw[:200]!r}") self.raw = raw[:200] ``` **`_iter_events` STEPS amended:** ``` FOR EACH sse in event_source.aiter_sse(): IF sse.data == "": # INV-001: empty-data event is treated as a keepalive; skip silently. # ORDERING NOTE: this branch fires BEFORE _parse_sse_id(sse.id). An # event with empty data AND a malformed id is therefore silently # skipped (MalformedSseId is NOT raised). This is intentional — a # keepalive with a bad id is still a keepalive, and the user-visible # behavior is "the frame had no content; ignore it". Don't reorder # the id-parse before the empty-skip check. # Do NOT yield, do NOT update last_sse_id, do NOT raise. CONTINUE CALL _parse_sse_id(sse.id) → SseId # existing; only runs for non-empty-data events ... existing turn_id checks ... TRY: body = json.loads(sse.data) CATCH json.JSONDecodeError as exc: # INV-002: malformed data is a protocol error; surface as named exception. RAISE MalformedSseData(raw=sse.data) event = _envelope_for_type(body, sse_id=sse_id) YIELD event ... existing terminal-event handling ... ``` **`stream_turn` ERROR_ROUTING amended:** ``` MalformedSseData: local_handling: log truncated raw `data:` payload; re-raise to caller flow_control: abort state_recovery: none (payload corruption; can't decode; consumer's call to reconnect) ``` (Mirrors the existing `MalformedSseId` entry shape.) **New TESTS entries on `_iter_events` / `stream_turn`:** - `empty_data_skipped [trace]`: stream yields four frames in order — `text(42:1)`, `empty-data(42:2 — id present, data empty)`, `text(42:3)`, `done(42:4)`. Consumer yields exactly three Events: `Text(42:1)`, `Text(42:3)`, `Done(42:4)` — the empty-data frame at `42:2` is invisible to the caller. Per INV-003, intermediate state after consuming the second `Text` (i.e., immediately after yielding the event with `sse_id=(42, 3)`) is `last_sse_id == (42, 3)` — proves the skip did NOT advance through `42:2`. Final state after consuming `Done` is `last_sse_id == (42, 4)` (the terminal event advances it as normal). - `malformed_data_raises [error]`: stream yields text(42:1) then event with `data: not-json` → consumer yields one Text then raises `MalformedSseData(raw="not-json")`. Iteration aborts. - `whitespace_data_raises [adversarial]`: stream yields one event with `data: ` (single space) → consumer raises `MalformedSseData` (whitespace is non-empty and JSON-parse-fails; the skip rule is empty-string only). - `malformed_data_truncation [security]`: stream yields one event with 5000-char malformed data → `MalformedSseData.raw` is exactly the first 200 chars; exception message includes only the truncated form. **`assumptions` block adds**: empty `httpx_sse.ServerSentEvent.data` behavior across the library's parsing modes. ### Issue #3 (`ratatoskr.cli`) amendments **`_run_turn` ERROR_ROUTING gains `MalformedSseData`:** ``` MalformedSseData: local_handling: write `[malformed_sse_data] raw={exc.raw!r}` to stderr flow_control: abort state_recovery: none (server-side wire bug) exit_code: 22 ``` **New TEST:** - `malformed_sse_data [error]`: mock yields one Text then an event with `data: not-json` → `_run_turn` returns 22; stderr `[malformed_sse_data]` is present; `raw=...` truncated. ### Issue #4 (`ratatoskr.tui`) amendments **`_stream_turn_worker` ERROR_ROUTING gains `MalformedSseData`:** ``` MalformedSseData: local_handling: log.write(f"[malformed_sse_data] raw={exc.raw!r}") to transcript flow_control: abort (the iteration aborts; finally-block restores state) state_recovery: state → idle; footer hint reset (INV-008: mid-session errors do NOT exit the app) ``` **New TEST:** - `malformed_sse_data_returns_to_idle [error]`: mock yields text + event with malformed data → transcript has `[malformed_sse_data]`; state → idle; app does NOT exit. --- ## Acceptance - All three amended contracts (#1, #3, #4) drift-check clean. - All existing tests + new `MalformedSseData` coverage GREEN under `uv run pytest tests/`. - `uv run ruff check src/ tests/` clean. - Boundary smoke `tests/test_no_worldtree_imports.py` still passes. - **Regression smoke**: re-run the long-conversation TUI smoke that previously crashed (mimic the 2026-05-22 crash conditions — a turn long enough that mimir emits ≥1000 events). Confirm no crash; the conversation completes cleanly to `[done]`. - The pre-existing happy-path smokes (mimir short responses) still work — empty-data skip is invisible when no empty-data events occur.