ratatoskr/docs/contracts/issues/7.contract.md

contract_version, target_module, scope, depends_on, used_by, language, complexity, estimated_loc, confidence, assumptions, open_questions, prd, dependencies

contract_version	target_module	scope	depends_on	used_by	language	complexity	estimated_loc	confidence	assumptions	open_questions	prd	dependencies
2.1	ratatoskr.sse_client	Stop crashing mid-stream when an SSE event with empty `data:` is dispatched. Two-rule fix in `_iter_events`: (a) when `sse.data` is empty, SKIP the event silently (treat as keepalive per SSE spec — don't yield, don't update `last_sse_id`, don't crash); (b) when `sse.data` is non-empty but fails `json.loads`, raise a new `MalformedSseData(raw=sse.data[:200])` exception (mirrors the existing `MalformedSseId` shape — wire-level protocol error, surfaceable to presenters). Presenters (`cli._run_turn`, `tui._stream_turn_worker`) gain a `MalformedSseData` case in their ERROR_ROUTING. No new modules; in-place amendments to issues #1 (sse_client), #3 (cli), #4 (tui). Discovered by mid-stream crash 2026-05-22: turn 93 emitted 1077 events successfully, then event 1078 arrived with `id: 93:1078` but empty `data:`; `json.loads('')` raised `JSONDecodeError`, propagated through Textual's worker, crashed the app.	httpx httpx-sse		python	low	25	0.9	`httpx_sse.ServerSentEvent` surfaces events with an `id:` line but no `data:` line (or empty `data:` content) as a `ServerSentEvent` instance with `data=''`. This is a known behavior-vs-spec divergence in `httpx_sse` — the SSE RFC says empty-data events SHOULD NOT be dispatched, but the library is permissive. We handle the library's actual behavior, not the RFC's prescription. Worldtree's spec declares every event has JSON-shaped `data:`. An empty-data event from Worldtree is either (a) a server bug, (b) a wire-level keepalive that should have been suppressed by `httpx_sse`, or (c) mid-stream truncation. Cause is upstream and out of scope; ratatoskr's job is to not crash. Non-empty data that fails `json.loads` is a true protocol error (server emitted malformed JSON). Surface as a named exception so presenters can render a clean error label, not propagate raw `JSONDecodeError`. Whitespace-only `sse.data` (e.g., `' '`, `'\n'`) IS non-empty by Python truthiness and `json.loads` fails on it. Treat as malformed (raise) rather than skip. The skip path is only for the exact empty-string case (`sse.data == ''`), matching what `httpx_sse` produces for a truly data-less event. Truncation of `raw` to 200 chars in `MalformedSseData` mirrors `MalformedSseId.raw = raw[:64]` (sse_client.py:132) but allows more chars because JSON payloads are typically longer than wire IDs. Keep the exception body bounded so it stays loggable.	Telemetry on skipped empty-data events: should ratatoskr emit a stderr label like `[sse_keepalive]` so the operator can see they're happening? Draft: no for v1. Empty-data events are silently dropped; if frequency turns out to be high (or zero, suggesting the bug is actually elsewhere), revisit. The TUI's RichLog already gets all yielded events labeled, so a skip is genuinely invisible by design. Worldtree-side investigation: is mimir emitting empty-data events legitimately (keepalive) or is this a bug? Worth filing on worldtree-dev's tracker if frequency is non-trivial — but ratatoskr's robustness fix is independent of the upstream root cause.	issue issue_url body_sha256_16 lock_in_comment_id lock_in_sha256_16 lock_in_at pinned_at 7 #7 155e21850365543f null null null 2026-05-22T05:22:38+00:00	issue path reason 1 src/ratatoskr/sse_client.py In-place contract amendment: `_iter_events` STEPS gain the empty-skip + malformed-raise rules; new `MalformedSseData` exception added to the public surface; `stream_turn` ERROR_ROUTING lists `MalformedSseData` alongside `MalformedSseId`; new TESTS for the skip and the raise paths. issue path reason 3 src/ratatoskr/cli.py In-place contract amendment: `_run_turn` ERROR_ROUTING gains a `MalformedSseData` case → stderr `[malformed_sse_data] raw=...` + exit code 22 (protocol-failure bucket, same as MalformedSseId/TurnIdFlip). issue path reason 4 src/ratatoskr/tui.py In-place contract amendment: `_stream_turn_worker` ERROR_ROUTING gains a `MalformedSseData` case → transcript `[malformed_sse_data] raw=...` line + state → idle per INV-008 (mid-session errors don't exit the app).

Mid-stream robustness — empty SSE data is skipped, malformed is raised cleanly

Context

During a TUI conversation with mimir on 2026-05-22, ratatoskr consumed 1077 SSE events successfully then crashed on event 1078:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  at src/ratatoskr/sse_client.py:301
  in event = _envelope_for_type(json.loads(sse.data), sse_id=sse_id)

  sse = ServerSentEvent(event='message', id='93:1078')
  data = ''                        # ← EMPTY but id present
  sse_id = SseId(turn_id=93, seq=1078)

_iter_events unconditionally calls json.loads(sse.data) on every dispatched event. When httpx_sse surfaces an event with empty data, the JSON parser fails, the exception propagates through stream_turn → _stream_turn_worker → Textual's Worker → app abort → traceback dumps to the operator's terminal. The turn is lost.

The SSE spec allows empty-data events (commonly keepalives). The Worldtree spec says every event carries JSON data:, so an empty-data frame from Worldtree is either a server bug, an httpx_sse permissive-parsing quirk, or mid-stream truncation. The cause is upstream; ratatoskr's responsibility is to not crash the operator's conversation over a single anomalous frame.

This issue adds two robustness rules in _iter_events: empty data is silently skipped (keepalive semantics); non-empty malformed JSON raises a new named exception (MalformedSseData) that presenters surface as a clean error label, not a raw traceback.

Data flow

Input change:

• _iter_events continues to consume ServerSentEvent objects from httpx_sse.EventSource. Now distinguishes three cases on sse.data:
  • Empty (sse.data == ''): skip — don't yield, don't update last_sse_id, don't increment any counter. Continue to next.
  • Non-empty + parses as JSON: yield as today (typed Event per _envelope_for_type).
  • Non-empty + JSONDecodeError: raise MalformedSseData(raw=sse.data[:200]).

Output change:

• A stream containing K empty-data events interleaved with N regular events yields N events (the K empty are invisible to the caller).
• A stream containing a malformed-data event raises MalformedSseData the same way MalformedSseId is raised today (consumer catches in ERROR_ROUTING).

No side effects: purely an iterator-internal behavior change. No new disk I/O, no new HTTP calls, no new state.

Invariants

• INV-001 [hard]: _iter_events MUST silently skip events where sse.data == '' (exact empty string equality — NOT truthy-check, NOT a stripped-whitespace check, NOT a "looks like a keepalive" pattern). Whitespace-only data (' ', '\n', '\t') is NOT empty and MUST go through the malformed-raise path (INV-002). The "keepalive" framing is intent-of-the-skip, not a generalized rule — implementers MUST apply the literal sse.data == '' check. Skip means: don't yield, don't update last_sse_id, don't set terminal_seen to True, don't call _parse_sse_id on sse.id, don't raise. The iteration continues to the next event from event_source.aiter_sse(). (No other counters or accumulators exist in _iter_events — last_sse_id and terminal_seen are the only state to preserve. Future implementations must not silently grow the state set without re-evaluating this invariant.)
• INV-002 [hard]: _iter_events MUST raise MalformedSseData (NOT JSONDecodeError, NOT SseConnectionDropped) when sse.data is non-empty but fails json.loads(...). Truncation: raw=sse.data[:200] in the exception. The raw field is the only payload-derived data on the exception — keep it bounded.
• INV-003 [hard]: Skip does NOT advance last_sse_id. Rationale: last_sse_id is what callers use for reconnect_turn on SseConnectionDropped. A skipped empty-data event was not user-visible content; the caller should resume from the last real event, not from the skipped one. (If last_sse_id advanced through the skip and a drop happened immediately after, the server would replay nothing of interest — the operator loses no data, but the precision is wrong.)
• INV-004 [hard]: MalformedSseData is a sibling of MalformedSseId in the public exception surface. Both indicate wire-level protocol violations from the server; both abort the stream (not recoverable in process); both belong to exit-code bucket 22 (protocol failures) in cli's table. MalformedSseData.raw mirrors MalformedSseId.raw — both stored as truncated str attributes on the exception.
• INV-005 [hard]: No core.* / worldtree.* imports (existing boundary; unchanged).

Out of scope

• Diagnosing upstream root cause. Whether Worldtree is emitting empty-data events legitimately (keepalive) or as a server bug is a worldtree-dev question worth filing separately if frequency is non-trivial. Ratatoskr's robustness fix is independent.
• Telemetry on skipped events. No stderr-label or counter for [sse_keepalive]-style. v1 silent skip; revisit if frequency turns out to matter for observability.
• Auto-reconnect on MalformedSseData. reconnect_turn exists for clean drops; whether to invoke it after a malformed frame is policy and currently the consumer's call. Presenters today don't auto- reconnect on any error class; that pattern stays unchanged here.
• Whitespace-only data handling beyond malformed-raise. Treating ' ' or '\\n' as malformed (per the raise rule) is the v1 stance; if a real-world server emits whitespace keepalives, file a follow-up.
• SseConnectionDropped reclassification. Existing wire-level exceptions (ReadError, RemoteProtocolError, ReadTimeout, clean-EOF-before-terminal) continue to map to SseConnectionDropped. MalformedSseData is a distinct class: payload-level malformation, not transport-level failure.

Constraints

• [compatibility] Spec pin unchanged. The empty-data handling is a client-side robustness change; nothing about the wire contract moves.
• [performance] A skipped event costs one extra branch + continue in the loop. Negligible.
• [security] MalformedSseData.raw truncates to 200 chars to bound the exception body — same hygiene as MalformedSseId.raw[:64] and SseConnectFailed.body[:1024].
• [style] Exception class follows existing pattern (frozen-ish: __init__ stores self.raw, super().__init__(message)). Ruff line-length=100.

---

In-place amendments (the work)

This issue's contract is small because the real work is amendments to issues #1, #3, #4. The amendments are pinned here so reviewers see the whole change in one place; the actual contract files are amended in-place as part of this issue's commit.

Issue #1 (ratatoskr.sse_client) amendments

New public exception:

class MalformedSseData(Exception):
    """Raised when an SSE event's `data:` payload is non-empty but fails JSON parsing."""

    def __init__(self, raw: str) -> None:
        super().__init__(f"malformed SSE data: {raw[:200]!r}")
        self.raw = raw[:200]

_iter_events STEPS amended:

FOR EACH sse in event_source.aiter_sse():
  IF sse.data == "":
    # INV-001: empty-data event is treated as a keepalive; skip silently.
    # ORDERING NOTE: this branch fires BEFORE _parse_sse_id(sse.id). An
    # event with empty data AND a malformed id is therefore silently
    # skipped (MalformedSseId is NOT raised). This is intentional — a
    # keepalive with a bad id is still a keepalive, and the user-visible
    # behavior is "the frame had no content; ignore it". Don't reorder
    # the id-parse before the empty-skip check.
    # Do NOT yield, do NOT update last_sse_id, do NOT raise.
    CONTINUE

  CALL _parse_sse_id(sse.id) → SseId    # existing; only runs for non-empty-data events
  ... existing turn_id checks ...

  TRY:
    body = json.loads(sse.data)
  CATCH json.JSONDecodeError as exc:
    # INV-002: malformed data is a protocol error; surface as named exception.
    RAISE MalformedSseData(raw=sse.data)

  event = _envelope_for_type(body, sse_id=sse_id)
  YIELD event
  ... existing terminal-event handling ...

stream_turn ERROR_ROUTING amended:

MalformedSseData:
  local_handling: log truncated raw `data:` payload; re-raise to caller
  flow_control: abort
  state_recovery: none (payload corruption; can't decode; consumer's call to reconnect)

(Mirrors the existing MalformedSseId entry shape.)

New TESTS entries on _iter_events / stream_turn:

• empty_data_skipped [trace]: stream yields four frames in order — text(42:1), empty-data(42:2 — id present, data empty), text(42:3), done(42:4). Consumer yields exactly three Events: Text(42:1), Text(42:3), Done(42:4) — the empty-data frame at 42:2 is invisible to the caller. Per INV-003, intermediate state after consuming the second Text (i.e., immediately after yielding the event with sse_id=(42, 3)) is last_sse_id == (42, 3) — proves the skip did NOT advance through 42:2. Final state after consuming Done is last_sse_id == (42, 4) (the terminal event advances it as normal).
• malformed_data_raises [error]: stream yields text(42:1) then event with data: not-json → consumer yields one Text then raises MalformedSseData(raw="not-json"). Iteration aborts.
• whitespace_data_raises [adversarial]: stream yields one event with data: (single space) → consumer raises MalformedSseData (whitespace is non-empty and JSON-parse-fails; the skip rule is empty-string only).
• malformed_data_truncation [security]: stream yields one event with 5000-char malformed data → MalformedSseData.raw is exactly the first 200 chars; exception message includes only the truncated form.

assumptions block adds: empty httpx_sse.ServerSentEvent.data behavior across the library's parsing modes.

Issue #3 (ratatoskr.cli) amendments

_run_turn ERROR_ROUTING gains MalformedSseData:

MalformedSseData:
  local_handling: write `[malformed_sse_data] raw={exc.raw!r}` to stderr
  flow_control: abort
  state_recovery: none (server-side wire bug)
  exit_code: 22

New TEST:

• malformed_sse_data [error]: mock yields one Text then an event with data: not-json → _run_turn returns 22; stderr [malformed_sse_data] is present; raw=... truncated.

Issue #4 (ratatoskr.tui) amendments

_stream_turn_worker ERROR_ROUTING gains MalformedSseData:

MalformedSseData:
  local_handling: log.write(f"[malformed_sse_data] raw={exc.raw!r}") to transcript
  flow_control: abort (the iteration aborts; finally-block restores state)
  state_recovery: state → idle; footer hint reset (INV-008: mid-session errors do NOT exit the app)

New TEST:

• malformed_sse_data_returns_to_idle [error]: mock yields text + event with malformed data → transcript has [malformed_sse_data]; state → idle; app does NOT exit.

---

Acceptance

• All three amended contracts (#1, #3, #4) drift-check clean.
• All existing tests + new MalformedSseData coverage GREEN under uv run pytest tests/.
• uv run ruff check src/ tests/ clean.
• Boundary smoke tests/test_no_worldtree_imports.py still passes.
• Regression smoke: re-run the long-conversation TUI smoke that previously crashed (mimic the 2026-05-22 crash conditions — a turn long enough that mimir emits ≥1000 events). Confirm no crash; the conversation completes cleanly to [done].
• The pre-existing happy-path smokes (mimir short responses) still work — empty-data skip is invisible when no empty-data events occur.