ratatoskr/docs/contracts/issues/7.contract.md

---
contract_version: "2.1"
target_module: "ratatoskr.sse_client"
scope: "Stop crashing mid-stream when an SSE event with empty `data:` is dispatched. Two-rule fix in `_iter_events`: (a) when `sse.data` is empty, SKIP the event silently (treat as keepalive per SSE spec — don't yield, don't update `last_sse_id`, don't crash); (b) when `sse.data` is non-empty but fails `json.loads`, raise a new `MalformedSseData(raw=sse.data[:200])` exception (mirrors the existing `MalformedSseId` shape — wire-level protocol error, surfaceable to presenters). Presenters (`cli._run_turn`, `tui._stream_turn_worker`) gain a `MalformedSseData` case in their ERROR_ROUTING. No new modules; in-place amendments to issues #1 (sse_client), #3 (cli), #4 (tui). Discovered by mid-stream crash 2026-05-22: turn 93 emitted 1077 events successfully, then event 1078 arrived with `id: 93:1078` but empty `data:`; `json.loads('')` raised `JSONDecodeError`, propagated through Textual's worker, crashed the app."
depends_on:
  - "httpx"
  - "httpx-sse"
used_by: []
language: "python"
complexity: "low"
estimated_loc: 25
confidence: 0.9
assumptions:
  - "`httpx_sse.ServerSentEvent` surfaces events with an `id:` line but no `data:` line (or empty `data:` content) as a `ServerSentEvent` instance with `data=''`. This is a known behavior-vs-spec divergence in `httpx_sse` — the SSE RFC says empty-data events SHOULD NOT be dispatched, but the library is permissive. We handle the library's actual behavior, not the RFC's prescription."
  - "Worldtree's spec declares every event has JSON-shaped `data:`. An empty-data event from Worldtree is either (a) a server bug, (b) a wire-level keepalive that should have been suppressed by `httpx_sse`, or (c) mid-stream truncation. Cause is upstream and out of scope; ratatoskr's job is to not crash."
  - "Non-empty data that fails `json.loads` is a true protocol error (server emitted malformed JSON). Surface as a named exception so presenters can render a clean error label, not propagate raw `JSONDecodeError`."
  - "Whitespace-only `sse.data` (e.g., `' '`, `'\\n'`) IS non-empty by Python truthiness and `json.loads` fails on it. Treat as malformed (raise) rather than skip. The skip path is only for the exact empty-string case (`sse.data == ''`), matching what `httpx_sse` produces for a truly data-less event."
  - "Truncation of `raw` to 200 chars in `MalformedSseData` mirrors `MalformedSseId.raw = raw[:64]` (sse_client.py:132) but allows more chars because JSON payloads are typically longer than wire IDs. Keep the exception body bounded so it stays loggable."
open_questions:
  - "Telemetry on skipped empty-data events: should ratatoskr emit a stderr label like `[sse_keepalive]` so the operator can see they're happening? Draft: no for v1. Empty-data events are silently dropped; if frequency turns out to be high (or zero, suggesting the bug is actually elsewhere), revisit. The TUI's RichLog already gets all yielded events labeled, so a skip is genuinely invisible by design."
  - "Worldtree-side investigation: is mimir emitting empty-data events legitimately (keepalive) or is this a bug? Worth filing on worldtree-dev's tracker if frequency is non-trivial — but ratatoskr's robustness fix is independent of the upstream root cause."
prd:
  issue: 7
  issue_url: "https://gitea.phasefinal.com/vh/ratatoskr/issues/7"
  body_sha256_16: "155e21850365543f"
  lock_in_comment_id: null
  lock_in_sha256_16: null
  lock_in_at: null
  pinned_at: "2026-05-22T05:22:38+00:00"
dependencies:
  - issue: 1
    path: "src/ratatoskr/sse_client.py"
    reason: "In-place contract amendment: `_iter_events` STEPS gain the empty-skip + malformed-raise rules; new `MalformedSseData` exception added to the public surface; `stream_turn` ERROR_ROUTING lists `MalformedSseData` alongside `MalformedSseId`; new TESTS for the skip and the raise paths."
  - issue: 3
    path: "src/ratatoskr/cli.py"
    reason: "In-place contract amendment: `_run_turn` ERROR_ROUTING gains a `MalformedSseData` case → stderr `[malformed_sse_data] raw=...` + exit code 22 (protocol-failure bucket, same as MalformedSseId/TurnIdFlip)."
  - issue: 4
    path: "src/ratatoskr/tui.py"
    reason: "In-place contract amendment: `_stream_turn_worker` ERROR_ROUTING gains a `MalformedSseData` case → transcript `[malformed_sse_data] raw=...` line + state → idle per INV-008 (mid-session errors don't exit the app)."
---

# Mid-stream robustness — empty SSE data is skipped, malformed is raised cleanly

## Context

During a TUI conversation with `mimir` on 2026-05-22, ratatoskr consumed
1077 SSE events successfully then crashed on event 1078:

```
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  at src/ratatoskr/sse_client.py:301
  in event = _envelope_for_type(json.loads(sse.data), sse_id=sse_id)

  sse = ServerSentEvent(event='message', id='93:1078')
  data = ''                        # ← EMPTY but id present
  sse_id = SseId(turn_id=93, seq=1078)
```

`_iter_events` unconditionally calls `json.loads(sse.data)` on every
dispatched event. When `httpx_sse` surfaces an event with empty `data`,
the JSON parser fails, the exception propagates through `stream_turn` →
`_stream_turn_worker` → Textual's `Worker` → app abort → traceback dumps
to the operator's terminal. The turn is lost.

The SSE spec allows empty-data events (commonly keepalives). The Worldtree
spec says every event carries JSON `data:`, so an empty-data frame from
Worldtree is either a server bug, an `httpx_sse` permissive-parsing quirk,
or mid-stream truncation. The cause is upstream; ratatoskr's responsibility
is to not crash the operator's conversation over a single anomalous frame.

This issue adds two robustness rules in `_iter_events`: empty data is
silently skipped (keepalive semantics); non-empty malformed JSON raises
a new named exception (`MalformedSseData`) that presenters surface as a
clean error label, not a raw traceback.

## Data flow

**Input change:**
- `_iter_events` continues to consume `ServerSentEvent` objects from
  `httpx_sse.EventSource`. Now distinguishes three cases on `sse.data`:
  - **Empty (`sse.data == ''`)**: skip — don't yield, don't update
    `last_sse_id`, don't increment any counter. Continue to next.
  - **Non-empty + parses as JSON**: yield as today (typed Event per
    `_envelope_for_type`).
  - **Non-empty + JSONDecodeError**: raise `MalformedSseData(raw=sse.data[:200])`.

**Output change:**
- A stream containing K empty-data events interleaved with N regular
  events yields N events (the K empty are invisible to the caller).
- A stream containing a malformed-data event raises `MalformedSseData`
  the same way `MalformedSseId` is raised today (consumer catches in
  ERROR_ROUTING).

**No side effects:** purely an iterator-internal behavior change. No new
disk I/O, no new HTTP calls, no new state.

## Invariants

- **INV-001 [hard]**: `_iter_events` MUST silently skip events where
  `sse.data == ''` (exact empty string equality — NOT truthy-check,
  NOT a stripped-whitespace check, NOT a "looks like a keepalive" pattern).
  Whitespace-only data (`' '`, `'\n'`, `'\t'`) is NOT empty and MUST go
  through the malformed-raise path (INV-002). The "keepalive" framing
  is intent-of-the-skip, not a generalized rule — implementers MUST
  apply the literal `sse.data == ''` check. Skip means: don't yield,
  don't update `last_sse_id`, don't set `terminal_seen` to True, don't
  call `_parse_sse_id` on `sse.id`, don't raise. The iteration continues
  to the next event from `event_source.aiter_sse()`. (No other counters
  or accumulators exist in `_iter_events` — `last_sse_id` and
  `terminal_seen` are the only state to preserve. Future implementations
  must not silently grow the state set without re-evaluating this
  invariant.)
- **INV-002 [hard]**: `_iter_events` MUST raise `MalformedSseData` (NOT
  `JSONDecodeError`, NOT `SseConnectionDropped`) when `sse.data` is
  non-empty but fails `json.loads(...)`. Truncation: `raw=sse.data[:200]`
  in the exception. The raw field is the only payload-derived data on
  the exception — keep it bounded.
- **INV-003 [hard]**: Skip does NOT advance `last_sse_id`. Rationale:
  `last_sse_id` is what callers use for `reconnect_turn` on
  `SseConnectionDropped`. A skipped empty-data event was not user-visible
  content; the caller should resume from the last *real* event, not from
  the skipped one. (If `last_sse_id` advanced through the skip and a drop
  happened immediately after, the server would replay nothing of interest
  — the operator loses no data, but the precision is wrong.)
- **INV-004 [hard]**: `MalformedSseData` is a sibling of `MalformedSseId`
  in the public exception surface. Both indicate wire-level protocol
  violations from the server; both abort the stream (not recoverable in
  process); both belong to exit-code bucket 22 (protocol failures) in
  cli's table. `MalformedSseData.raw` mirrors `MalformedSseId.raw` —
  both stored as truncated `str` attributes on the exception.
- **INV-005 [hard]**: No `core.*` / `worldtree.*` imports (existing
  boundary; unchanged).

## Out of scope

- **Diagnosing upstream root cause.** Whether Worldtree is emitting
  empty-data events legitimately (keepalive) or as a server bug is a
  worldtree-dev question worth filing separately if frequency is
  non-trivial. Ratatoskr's robustness fix is independent.
- **Telemetry on skipped events.** No stderr-label or counter for
  `[sse_keepalive]`-style. v1 silent skip; revisit if frequency turns
  out to matter for observability.
- **Auto-reconnect on `MalformedSseData`.** `reconnect_turn` exists for
  clean drops; whether to invoke it after a malformed frame is policy
  and currently the consumer's call. Presenters today don't auto-
  reconnect on any error class; that pattern stays unchanged here.
- **Whitespace-only data handling beyond malformed-raise.** Treating
  `' '` or `'\\n'` as malformed (per the raise rule) is the v1 stance;
  if a real-world server emits whitespace keepalives, file a follow-up.
- **`SseConnectionDropped` reclassification.** Existing wire-level
  exceptions (`ReadError`, `RemoteProtocolError`, `ReadTimeout`,
  clean-EOF-before-terminal) continue to map to `SseConnectionDropped`.
  `MalformedSseData` is a distinct class: payload-level malformation,
  not transport-level failure.

## Constraints

- **[compatibility]** Spec pin unchanged. The empty-data handling is a
  client-side robustness change; nothing about the wire contract moves.
- **[performance]** A skipped event costs one extra branch + continue in
  the loop. Negligible.
- **[security]** `MalformedSseData.raw` truncates to 200 chars to bound
  the exception body — same hygiene as `MalformedSseId.raw[:64]` and
  `SseConnectFailed.body[:1024]`.
- **[style]** Exception class follows existing pattern (frozen-ish:
  `__init__` stores `self.raw`, `super().__init__(message)`). Ruff
  line-length=100.

---

## In-place amendments (the work)

This issue's contract is small because the real work is amendments to
issues #1, #3, #4. The amendments are pinned here so reviewers see the
whole change in one place; the actual contract files are amended in-place
as part of this issue's commit.

### Issue #1 (`ratatoskr.sse_client`) amendments

**New public exception:**

```python
class MalformedSseData(Exception):
    """Raised when an SSE event's `data:` payload is non-empty but fails JSON parsing."""

    def __init__(self, raw: str) -> None:
        super().__init__(f"malformed SSE data: {raw[:200]!r}")
        self.raw = raw[:200]
```

**`_iter_events` STEPS amended:**

```
FOR EACH sse in event_source.aiter_sse():
  IF sse.data == "":
    # INV-001: empty-data event is treated as a keepalive; skip silently.
    # ORDERING NOTE: this branch fires BEFORE _parse_sse_id(sse.id). An
    # event with empty data AND a malformed id is therefore silently
    # skipped (MalformedSseId is NOT raised). This is intentional — a
    # keepalive with a bad id is still a keepalive, and the user-visible
    # behavior is "the frame had no content; ignore it". Don't reorder
    # the id-parse before the empty-skip check.
    # Do NOT yield, do NOT update last_sse_id, do NOT raise.
    CONTINUE

  CALL _parse_sse_id(sse.id) → SseId    # existing; only runs for non-empty-data events
  ... existing turn_id checks ...

  TRY:
    body = json.loads(sse.data)
  CATCH json.JSONDecodeError as exc:
    # INV-002: malformed data is a protocol error; surface as named exception.
    RAISE MalformedSseData(raw=sse.data)

  event = _envelope_for_type(body, sse_id=sse_id)
  YIELD event
  ... existing terminal-event handling ...
```

**`stream_turn` ERROR_ROUTING amended:**

```
MalformedSseData:
  local_handling: log truncated raw `data:` payload; re-raise to caller
  flow_control: abort
  state_recovery: none (payload corruption; can't decode; consumer's call to reconnect)
```

(Mirrors the existing `MalformedSseId` entry shape.)

**New TESTS entries on `_iter_events` / `stream_turn`:**

- `empty_data_skipped [trace]`: stream yields four frames in order —
  `text(42:1)`, `empty-data(42:2 — id present, data empty)`, `text(42:3)`,
  `done(42:4)`. Consumer yields exactly three Events: `Text(42:1)`,
  `Text(42:3)`, `Done(42:4)` — the empty-data frame at `42:2` is
  invisible to the caller. Per INV-003, intermediate state after
  consuming the second `Text` (i.e., immediately after yielding the
  event with `sse_id=(42, 3)`) is `last_sse_id == (42, 3)` — proves the
  skip did NOT advance through `42:2`. Final state after consuming
  `Done` is `last_sse_id == (42, 4)` (the terminal event advances it as
  normal).
- `malformed_data_raises [error]`: stream yields text(42:1) then event
  with `data: not-json` → consumer yields one Text then raises
  `MalformedSseData(raw="not-json")`. Iteration aborts.
- `whitespace_data_raises [adversarial]`: stream yields one event with
  `data: ` (single space) → consumer raises `MalformedSseData` (whitespace
  is non-empty and JSON-parse-fails; the skip rule is empty-string only).
- `malformed_data_truncation [security]`: stream yields one event with
  5000-char malformed data → `MalformedSseData.raw` is exactly the first
  200 chars; exception message includes only the truncated form.

**`assumptions` block adds**: empty `httpx_sse.ServerSentEvent.data`
behavior across the library's parsing modes.

### Issue #3 (`ratatoskr.cli`) amendments

**`_run_turn` ERROR_ROUTING gains `MalformedSseData`:**

```
MalformedSseData:
  local_handling: write `[malformed_sse_data] raw={exc.raw!r}` to stderr
  flow_control: abort
  state_recovery: none (server-side wire bug)
  exit_code: 22
```

**New TEST:**

- `malformed_sse_data [error]`: mock yields one Text then an event with
  `data: not-json` → `_run_turn` returns 22; stderr `[malformed_sse_data]`
  is present; `raw=...` truncated.

### Issue #4 (`ratatoskr.tui`) amendments

**`_stream_turn_worker` ERROR_ROUTING gains `MalformedSseData`:**

```
MalformedSseData:
  local_handling: log.write(f"[malformed_sse_data] raw={exc.raw!r}") to transcript
  flow_control: abort (the iteration aborts; finally-block restores state)
  state_recovery: state → idle; footer hint reset (INV-008: mid-session errors do NOT exit the app)
```

**New TEST:**

- `malformed_sse_data_returns_to_idle [error]`: mock yields text + event
  with malformed data → transcript has `[malformed_sse_data]`; state →
  idle; app does NOT exit.

---

## Acceptance

- All three amended contracts (#1, #3, #4) drift-check clean.
- All existing tests + new `MalformedSseData` coverage GREEN under
  `uv run pytest tests/`.
- `uv run ruff check src/ tests/` clean.
- Boundary smoke `tests/test_no_worldtree_imports.py` still passes.
- **Regression smoke**: re-run the long-conversation TUI smoke that
  previously crashed (mimic the 2026-05-22 crash conditions — a turn
  long enough that mimir emits ≥1000 events). Confirm no crash; the
  conversation completes cleanly to `[done]`.
- The pre-existing happy-path smokes (mimir short responses) still
  work — empty-data skip is invisible when no empty-data events
  occur.