contract(issue#7): author + amend #1/#3/#4 for empty-skip + MalformedSseData
Issue #7: mid-stream robustness fix discovered via 2026-05-22 crash. Long mimir TUI conversation (turn 93, 1077 events consumed) crashed on event 1078 with JSONDecodeError("Expecting value: line 1 column 1 (char 0)") from json.loads('') on an empty-data SSE frame. _iter_events unconditionally called json.loads on every dispatched event; when httpx_sse surfaces a frame with id: present but data: empty/missing (a known library-vs-spec divergence), parsing fails and propagates. Two-rule fix in _iter_events: - Empty sse.data (exact `== ''`): SKIP silently per SSE spec (keepalive semantics). Don't yield, don't advance last_sse_id, don't set terminal_seen. ORDERING: skip fires BEFORE _parse_sse_id, so a keepalive with a malformed id is still a keepalive (intentional). - Non-empty sse.data that fails json.loads: raise new MalformedSseData (sibling to MalformedSseId, mirrors raw[:200] truncation pattern). Wire-level protocol error; presenters route to [malformed_sse_data] + exit 22 in cli, transcript label + state→idle in tui (INV-008). Volva paraphrase round: 4 ambiguities, all amended: 1. INV-001 prose tightened — exact `sse.data == ''` rule made prominent; "keepalive" framing demoted to intent-not-rule; whitespace-only data explicitly listed as malformed (not skipped); specific state names (last_sse_id, terminal_seen) instead of vague "any counter". 2. STEPS pseudocode spells out the ordering — empty-skip happens BEFORE _parse_sse_id; empty-data with bad id is silently swallowed. 3. empty_data_skipped test description fixed (had off-by-one count + wrong wording around last_sse_id intermediate state). 4. (paired with #1 above). Volva code-review post-implementation: 3 findings, all addressed: F1 (test-gap): empty_data_skipped proves yielded events but not internal last_sse_id non-advancement. New empty_data_skip_preserves_last_seen_sse_id test probes via SseConnectionDropped.last_seen_sse_id after a drop following the skipped frame — if the skip had transiently advanced last_sse_id, the exception payload would carry the wrong value. F2 (precision, contract amend): MalformedSseData ERROR_ROUTING said "log truncated raw" but stream_turn doesn't log — sse_client is a library, presenters own observability. Amended to "propagate to caller (no logging at sse_client layer); presenters log exc.raw." F3 (test-gap): cli malformed_sse_data test asserted label but not `raw='X'` shape and not truncation. Tightened existing test + added malformed_sse_data_truncation with 5000-char payload — verifies MalformedSseData.raw truncation carries through the presenter's repr() rendering. **Calibration milestone**: issue #7 is the first issue with ZERO drift findings from Volva code-review. TDD caught all runtime behavior cleanly. The 3 findings were assertion-precision and architectural-correctness-of-wording, not behavioral. Hypothesis: tighter contract spec + smaller code surface shifts Volva's role from "catch behavioral drift" to "tighten observability + wording". Cumulative calibration table: #1 (4 findings, 3 drift + 1 test-gap), #2 (3, 1+1+1 precision), #3 (5, 3+1+1), #4 (8, 5+2+1), #7 (3, 0 drift + 2 test-gap + 1 precision). Contracts touched (all drift-check clean): - docs/contracts/issues/7.contract.md (new): the coordinating record. - docs/contracts/issues/1.contract.md: _iter_events STEP 3.0 empty-skip + ordering note; STEP 3.c JSONDecodeError → MalformedSseData; new MalformedSseData ERROR_ROUTING (propagate-to-caller wording per F2); 4 new TESTS entries including F1's last-seen probe. - docs/contracts/issues/3.contract.md: _run_turn ERROR_ROUTING + malformed_sse_data tests (incl. F3 truncation). - docs/contracts/issues/4.contract.md: INV-008 mentions MalformedSseData; _stream_turn_worker ERROR_ROUTING + new TEST.
This commit is contained in:
@@ -117,6 +117,11 @@ ERROR_ROUTING:
|
||||
local_handling: log truncated raw `id:` value; re-raise to caller
|
||||
flow_control: abort
|
||||
state_recovery: none (stream is corrupt; cannot resume safely)
|
||||
MalformedSseData:
|
||||
local_handling: propagate to caller (no logging at sse_client layer — sse_client is a library, not a logging sink; presenters own observability). The exception carries `exc.raw` (already truncated to 200 chars at MalformedSseData.__init__) for presenters to render.
|
||||
flow_control: abort
|
||||
state_recovery: none (payload corruption; can't decode; consumer's call to reconnect)
|
||||
note: empty `sse.data == ''` is SKIPPED, not raised — see _iter_events STEP 3.0 (issue #7 INV-001). Only non-empty-but-malformed-JSON raises MalformedSseData.
|
||||
TurnIdFlip:
|
||||
local_handling: log both turn_ids; re-raise to caller
|
||||
flow_control: abort
|
||||
@@ -127,6 +132,13 @@ STEPS:
|
||||
ON httpx.HTTPStatusError before stream opens:
|
||||
RAISE SseConnectFailed
|
||||
3. [loop, flexibility=prescriptive] FOR EACH sse_event in event_source.aiter_sse():
|
||||
0. [branch, flexibility=prescriptive] IF sse_event.data == "":
|
||||
# Issue #7 INV-001: empty-data frames are treated as keepalives.
|
||||
# ORDERING: this check fires BEFORE _parse_sse_id; an empty-data event
|
||||
# with a malformed id is silently swallowed (a keepalive with a bad id
|
||||
# is still a keepalive). Skip means: don't yield, don't update last_sse_id,
|
||||
# don't set terminal_seen, don't raise. CONTINUE to next sse_event.
|
||||
CONTINUE
|
||||
a. [sequential] CALL _parse_sse_id(sse_event.id) → (turn_id, seq)
|
||||
ON ValueError:
|
||||
RAISE MalformedSseId(raw=sse_event.id[:64])
|
||||
@@ -135,6 +147,10 @@ STEPS:
|
||||
ELSE:
|
||||
SET established_turn_id = turn_id (on first event)
|
||||
c. [sequential] Parse sse_event.data as JSON → body dict
|
||||
ON json.JSONDecodeError:
|
||||
# Issue #7 INV-002: non-empty data that fails json.loads is a protocol
|
||||
# error (server emitted malformed JSON). Surface as named exception.
|
||||
RAISE MalformedSseData(raw=sse_event.data) # raw truncated to 200 in __init__
|
||||
d. [sequential] CALL _envelope_for_type(body, sse_id=(turn_id, seq)) → typed Event
|
||||
e. [sequential] YIELD event
|
||||
f. [branch] IF isinstance(event, (Done, Error, Cancelled)):
|
||||
@@ -151,6 +167,11 @@ TESTS:
|
||||
turn_id_flip [adversarial]: mock emits text events with ids `42:1` then `99:2` → consumer raises TurnIdFlip; only the first event was yielded
|
||||
connection_drop [error]: mock raises httpx.RemoteProtocolError after one text event → consumer raises SseConnectionDropped(last_seen_sse_id=(42, 1))
|
||||
no_text_aggregation [trace]: assert consumer does NOT concatenate text deltas internally — each text event yields as its own Text envelope; aggregation is the caller's job
|
||||
empty_data_skipped [trace,issue#7]: stream yields 4 frames — text(42:1), empty-data(42:2), text(42:3), done(42:4) — consumer yields exactly 3 events (Text, Text, Done); empty-data frame invisible. Intermediate state after the second Text: last_sse_id == (42, 3) (proves skip did not advance through 42:2).
|
||||
empty_data_skip_preserves_last_seen_sse_id [trace,issue#7]: stream yields text(42:1), empty-data(42:2), then RemoteProtocolError → consumer raises SseConnectionDropped with `last_seen_sse_id == SseId(42, 1)`. Directly probes INV-003 via the exception payload — if the skip had transiently advanced last_sse_id through 42:2, the exception would carry (42, 2) and the test would fail.
|
||||
malformed_data_raises [error,issue#7]: stream yields text(42:1) + event with `data: not-json` → consumer yields one Text then raises MalformedSseData(raw="not-json"); iteration aborts.
|
||||
whitespace_data_raises [adversarial,issue#7]: stream yields event with `data:` followed by single space → MalformedSseData (whitespace is non-empty per INV-001 exact-equality skip rule).
|
||||
malformed_data_truncation [security,issue#7]: stream yields event with 5000-char malformed data → MalformedSseData.raw is exactly first 200 chars; exception message truncated too.
|
||||
```
|
||||
|
||||
```contract
|
||||
|
||||
@@ -427,6 +427,11 @@ ERROR_ROUTING:
|
||||
flow_control: abort
|
||||
state_recovery: none (server-side wire bug; surface honestly)
|
||||
exit_code: 22
|
||||
MalformedSseData:
|
||||
local_handling: write `[malformed_sse_data] raw={exc.raw!r}` to stderr (issue #7)
|
||||
flow_control: abort
|
||||
state_recovery: none (server-side wire bug; payload corruption)
|
||||
exit_code: 22
|
||||
TurnIdFlip:
|
||||
local_handling: write `[turn_id_flip] expected={exc.established} got={exc.got}` to stderr
|
||||
flow_control: abort
|
||||
@@ -494,6 +499,8 @@ TESTS:
|
||||
sse_connect_failed_404 [error]: mock returns 404 before stream opens → returns 20; stderr "[sse_connect_failed] status=404 ..."
|
||||
connection_dropped [error]: mock raises RemoteProtocolError mid-stream → returns 21; stderr "[connection_dropped]"
|
||||
malformed_sse_id [error]: mock yields an event with `id: 42` (no seq) → returns 22; stderr "[malformed_sse_id]"
|
||||
malformed_sse_data [error,issue#7]: mock yields text + event with `data: not-json` → returns 22; stderr contains literal "[malformed_sse_data] raw='not-json'" (exact label+raw shape per ERROR_ROUTING)
|
||||
malformed_sse_data_truncation [security,issue#7]: mock yields text + event with 5000-char malformed data → returns 22; stderr "[malformed_sse_data]" present; full 5000-char payload NOT in stderr; truncated 200-char form IS present (verifies MalformedSseData.raw truncation carries through the presenter's repr() rendering)
|
||||
turn_id_flip [error]: mock yields events 42:1 then 99:2 → returns 22; stderr "[turn_id_flip] expected=42 got=99"
|
||||
sigint_before_first_event [scenario]: sigint_event set BEFORE the mock has yielded anything → returns 3; respx tracked zero cancel_turn POSTs (INV-008)
|
||||
sigint_mid_stream_drains_to_cancelled [scenario,tracer]: mock yields text(42:1) → caller sets sigint_event → mock yields cancelled(42:2) → returns 3; respx tracked exactly one POST /sessions/{id}/turns/42/cancel (INV-007)
|
||||
|
||||
@@ -90,7 +90,7 @@ The shell is the load-bearing primary surface. Together with `--send`, it makes
|
||||
- **INV-005 [hard]**: Markdown rendering on agent output is default-on; `--raw` is the opt-out. With markdown enabled, `Text` event deltas stream as raw text appended to the RichLog as they arrive (no mid-stream markdown attempt — partial markdown like `**hel` would render ugly), and on `Done` a separator + the full markdown-rendered assistant message is appended below the streamed deltas. **This means the assistant's response visibly appears TWICE in the transcript by design — once as the streamed raw deltas, once as the post-Done markdown render — separated by a horizontal-rule separator.** This is the v1 accepted trade-off for streaming-visibility-without-mid-stream-markdown-ugliness; the cleaner Static-then-commit pattern (streaming into a replaceable widget, then committing the markdown version in place) is documented in `open_questions:` as the follow-up if the double-display proves empirically noisy. Implementers MUST NOT attempt the Static-then-commit pattern in this shell — it's deferred. With `--raw`, only the streamed deltas appear; no post-Done re-render; no double-display.
|
||||
- **INV-006 [hard]**: User-prompt echo in the transcript MUST visibly distinguish user input from assistant output. Format: `❯ <content>` for user lines (with a literal `❯` prefix); assistant lines have no prefix. The prefix is also a screen-reader-friendly affordance.
|
||||
- **INV-007 [hard]**: One `httpx.AsyncClient` per app lifetime — opened in `on_mount`, closed in `on_unmount` via the async-with context manager pattern. The client is NOT recreated per turn (would burn the TCP connection pool).
|
||||
- **INV-008 [hard]**: Mid-session network/protocol errors (`SseConnectionDropped`, `SseConnectFailed`, `MalformedSseId`, `TurnIdFlip`) during a streaming turn render as error lines in the transcript and return the app to **idle** state — they do NOT exit the app. Only initial session-create errors exit (per Data flow exit codes).
|
||||
- **INV-008 [hard]**: Mid-session network/protocol errors (`SseConnectionDropped`, `SseConnectFailed`, `MalformedSseId`, `MalformedSseData` (issue #7), `TurnIdFlip`) during a streaming turn render as error lines in the transcript and return the app to **idle** state — they do NOT exit the app. Only initial session-create errors exit (per Data flow exit codes).
|
||||
- **INV-009 [hard]**: No `core.*` / `worldtree.*` imports. The boundary smoke (`tests/test_no_worldtree_imports.py`) covers `src/ratatoskr/` as a whole including the new tui.py.
|
||||
|
||||
## Out of scope
|
||||
@@ -272,7 +272,7 @@ POST: [POST-002 side_effect] each event passed through _render_event_to_log exac
|
||||
POST: [POST-003 side_effect] for Done events with NOT args.raw: a separator line + the markdown-rendered Done.response appended to RichLog (INV-005)
|
||||
POST: [POST-004 state_change] active_turn_id is set to event.sse_id.turn_id on the FIRST yielded event (for cancel_turn use by action_interrupt)
|
||||
ERROR_ROUTING:
|
||||
SseConnectFailed | SseConnectionDropped | MalformedSseId | TurnIdFlip:
|
||||
SseConnectFailed | SseConnectionDropped | MalformedSseId | MalformedSseData | TurnIdFlip:
|
||||
local_handling: append `[<label>] <details>` to RichLog (mirror cli.py's error labels)
|
||||
flow_control: abort (the iteration aborts; finally-block restores state)
|
||||
state_recovery: state → idle; footer hint reset; active_turn_id cleared. (INV-008: mid-session errors do NOT exit the app.)
|
||||
@@ -304,6 +304,7 @@ TESTS:
|
||||
active_turn_id_set_on_first_event [trace]: mock yields text(42:1) then waits; after first render, self.active_turn_id == 42 (verifies the cancel path can pick it up)
|
||||
sse_connect_failed_returns_to_idle [error]: mock returns 404 → "[sse_connect_failed]" label in RichLog; state → idle; app does NOT exit (INV-008)
|
||||
connection_dropped_returns_to_idle [error]: mock raises RemoteProtocolError mid-stream → "[connection_dropped]" label; state → idle
|
||||
malformed_sse_data_returns_to_idle [error,issue#7]: mock yields text + event with `data: not-json` → "[malformed_sse_data]" label; state → idle; app does NOT exit (INV-008)
|
||||
rendered_event_per_event [trace]: spy on _render_event_to_log; mock yields N events; call_count == N (terminal events included, since Done/Error/Cancelled also render through it)
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,306 @@
|
||||
---
|
||||
contract_version: "2.1"
|
||||
target_module: "ratatoskr.sse_client"
|
||||
scope: "Stop crashing mid-stream when an SSE event with empty `data:` is dispatched. Two-rule fix in `_iter_events`: (a) when `sse.data` is empty, SKIP the event silently (treat as keepalive per SSE spec — don't yield, don't update `last_sse_id`, don't crash); (b) when `sse.data` is non-empty but fails `json.loads`, raise a new `MalformedSseData(raw=sse.data[:200])` exception (mirrors the existing `MalformedSseId` shape — wire-level protocol error, surfaceable to presenters). Presenters (`cli._run_turn`, `tui._stream_turn_worker`) gain a `MalformedSseData` case in their ERROR_ROUTING. No new modules; in-place amendments to issues #1 (sse_client), #3 (cli), #4 (tui). Discovered by mid-stream crash 2026-05-22: turn 93 emitted 1077 events successfully, then event 1078 arrived with `id: 93:1078` but empty `data:`; `json.loads('')` raised `JSONDecodeError`, propagated through Textual's worker, crashed the app."
|
||||
depends_on:
|
||||
- "httpx"
|
||||
- "httpx-sse"
|
||||
used_by: []
|
||||
language: "python"
|
||||
complexity: "low"
|
||||
estimated_loc: 25
|
||||
confidence: 0.9
|
||||
assumptions:
|
||||
- "`httpx_sse.ServerSentEvent` surfaces events with an `id:` line but no `data:` line (or empty `data:` content) as a `ServerSentEvent` instance with `data=''`. This is a known behavior-vs-spec divergence in `httpx_sse` — the SSE RFC says empty-data events SHOULD NOT be dispatched, but the library is permissive. We handle the library's actual behavior, not the RFC's prescription."
|
||||
- "Worldtree's spec declares every event has JSON-shaped `data:`. An empty-data event from Worldtree is either (a) a server bug, (b) a wire-level keepalive that should have been suppressed by `httpx_sse`, or (c) mid-stream truncation. Cause is upstream and out of scope; ratatoskr's job is to not crash."
|
||||
- "Non-empty data that fails `json.loads` is a true protocol error (server emitted malformed JSON). Surface as a named exception so presenters can render a clean error label, not propagate raw `JSONDecodeError`."
|
||||
- "Whitespace-only `sse.data` (e.g., `' '`, `'\\n'`) IS non-empty by Python truthiness and `json.loads` fails on it. Treat as malformed (raise) rather than skip. The skip path is only for the exact empty-string case (`sse.data == ''`), matching what `httpx_sse` produces for a truly data-less event."
|
||||
- "Truncation of `raw` to 200 chars in `MalformedSseData` mirrors `MalformedSseId.raw = raw[:64]` (sse_client.py:132) but allows more chars because JSON payloads are typically longer than wire IDs. Keep the exception body bounded so it stays loggable."
|
||||
open_questions:
|
||||
- "Telemetry on skipped empty-data events: should ratatoskr emit a stderr label like `[sse_keepalive]` so the operator can see they're happening? Draft: no for v1. Empty-data events are silently dropped; if frequency turns out to be high (or zero, suggesting the bug is actually elsewhere), revisit. The TUI's RichLog already gets all yielded events labeled, so a skip is genuinely invisible by design."
|
||||
- "Worldtree-side investigation: is mimir emitting empty-data events legitimately (keepalive) or is this a bug? Worth filing on worldtree-dev's tracker if frequency is non-trivial — but ratatoskr's robustness fix is independent of the upstream root cause."
|
||||
prd:
|
||||
issue: 7
|
||||
issue_url: "https://gitea.phasefinal.com/vh/ratatoskr/issues/7"
|
||||
body_sha256_16: "155e21850365543f"
|
||||
lock_in_comment_id: null
|
||||
lock_in_sha256_16: null
|
||||
lock_in_at: null
|
||||
pinned_at: "2026-05-22T05:22:38+00:00"
|
||||
dependencies:
|
||||
- issue: 1
|
||||
path: "src/ratatoskr/sse_client.py"
|
||||
reason: "In-place contract amendment: `_iter_events` STEPS gain the empty-skip + malformed-raise rules; new `MalformedSseData` exception added to the public surface; `stream_turn` ERROR_ROUTING lists `MalformedSseData` alongside `MalformedSseId`; new TESTS for the skip and the raise paths."
|
||||
- issue: 3
|
||||
path: "src/ratatoskr/cli.py"
|
||||
reason: "In-place contract amendment: `_run_turn` ERROR_ROUTING gains a `MalformedSseData` case → stderr `[malformed_sse_data] raw=...` + exit code 22 (protocol-failure bucket, same as MalformedSseId/TurnIdFlip)."
|
||||
- issue: 4
|
||||
path: "src/ratatoskr/tui.py"
|
||||
reason: "In-place contract amendment: `_stream_turn_worker` ERROR_ROUTING gains a `MalformedSseData` case → transcript `[malformed_sse_data] raw=...` line + state → idle per INV-008 (mid-session errors don't exit the app)."
|
||||
---
|
||||
|
||||
# Mid-stream robustness — empty SSE data is skipped, malformed is raised cleanly
|
||||
|
||||
## Context
|
||||
|
||||
During a TUI conversation with `mimir` on 2026-05-22, ratatoskr consumed
|
||||
1077 SSE events successfully then crashed on event 1078:
|
||||
|
||||
```
|
||||
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||||
at src/ratatoskr/sse_client.py:301
|
||||
in event = _envelope_for_type(json.loads(sse.data), sse_id=sse_id)
|
||||
|
||||
sse = ServerSentEvent(event='message', id='93:1078')
|
||||
data = '' # ← EMPTY but id present
|
||||
sse_id = SseId(turn_id=93, seq=1078)
|
||||
```
|
||||
|
||||
`_iter_events` unconditionally calls `json.loads(sse.data)` on every
|
||||
dispatched event. When `httpx_sse` surfaces an event with empty `data`,
|
||||
the JSON parser fails, the exception propagates through `stream_turn` →
|
||||
`_stream_turn_worker` → Textual's `Worker` → app abort → traceback dumps
|
||||
to the operator's terminal. The turn is lost.
|
||||
|
||||
The SSE spec allows empty-data events (commonly keepalives). The Worldtree
|
||||
spec says every event carries JSON `data:`, so an empty-data frame from
|
||||
Worldtree is either a server bug, an `httpx_sse` permissive-parsing quirk,
|
||||
or mid-stream truncation. The cause is upstream; ratatoskr's responsibility
|
||||
is to not crash the operator's conversation over a single anomalous frame.
|
||||
|
||||
This issue adds two robustness rules in `_iter_events`: empty data is
|
||||
silently skipped (keepalive semantics); non-empty malformed JSON raises
|
||||
a new named exception (`MalformedSseData`) that presenters surface as a
|
||||
clean error label, not a raw traceback.
|
||||
|
||||
## Data flow
|
||||
|
||||
**Input change:**
|
||||
- `_iter_events` continues to consume `ServerSentEvent` objects from
|
||||
`httpx_sse.EventSource`. Now distinguishes three cases on `sse.data`:
|
||||
- **Empty (`sse.data == ''`)**: skip — don't yield, don't update
|
||||
`last_sse_id`, don't increment any counter. Continue to next.
|
||||
- **Non-empty + parses as JSON**: yield as today (typed Event per
|
||||
`_envelope_for_type`).
|
||||
- **Non-empty + JSONDecodeError**: raise `MalformedSseData(raw=sse.data[:200])`.
|
||||
|
||||
**Output change:**
|
||||
- A stream containing K empty-data events interleaved with N regular
|
||||
events yields N events (the K empty are invisible to the caller).
|
||||
- A stream containing a malformed-data event raises `MalformedSseData`
|
||||
the same way `MalformedSseId` is raised today (consumer catches in
|
||||
ERROR_ROUTING).
|
||||
|
||||
**No side effects:** purely an iterator-internal behavior change. No new
|
||||
disk I/O, no new HTTP calls, no new state.
|
||||
|
||||
## Invariants
|
||||
|
||||
- **INV-001 [hard]**: `_iter_events` MUST silently skip events where
|
||||
`sse.data == ''` (exact empty string equality — NOT truthy-check,
|
||||
NOT a stripped-whitespace check, NOT a "looks like a keepalive" pattern).
|
||||
Whitespace-only data (`' '`, `'\n'`, `'\t'`) is NOT empty and MUST go
|
||||
through the malformed-raise path (INV-002). The "keepalive" framing
|
||||
is intent-of-the-skip, not a generalized rule — implementers MUST
|
||||
apply the literal `sse.data == ''` check. Skip means: don't yield,
|
||||
don't update `last_sse_id`, don't set `terminal_seen` to True, don't
|
||||
call `_parse_sse_id` on `sse.id`, don't raise. The iteration continues
|
||||
to the next event from `event_source.aiter_sse()`. (No other counters
|
||||
or accumulators exist in `_iter_events` — `last_sse_id` and
|
||||
`terminal_seen` are the only state to preserve. Future implementations
|
||||
must not silently grow the state set without re-evaluating this
|
||||
invariant.)
|
||||
- **INV-002 [hard]**: `_iter_events` MUST raise `MalformedSseData` (NOT
|
||||
`JSONDecodeError`, NOT `SseConnectionDropped`) when `sse.data` is
|
||||
non-empty but fails `json.loads(...)`. Truncation: `raw=sse.data[:200]`
|
||||
in the exception. The raw field is the only payload-derived data on
|
||||
the exception — keep it bounded.
|
||||
- **INV-003 [hard]**: Skip does NOT advance `last_sse_id`. Rationale:
|
||||
`last_sse_id` is what callers use for `reconnect_turn` on
|
||||
`SseConnectionDropped`. A skipped empty-data event was not user-visible
|
||||
content; the caller should resume from the last *real* event, not from
|
||||
the skipped one. (If `last_sse_id` advanced through the skip and a drop
|
||||
happened immediately after, the server would replay nothing of interest
|
||||
— the operator loses no data, but the precision is wrong.)
|
||||
- **INV-004 [hard]**: `MalformedSseData` is a sibling of `MalformedSseId`
|
||||
in the public exception surface. Both indicate wire-level protocol
|
||||
violations from the server; both abort the stream (not recoverable in
|
||||
process); both belong to exit-code bucket 22 (protocol failures) in
|
||||
cli's table. `MalformedSseData.raw` mirrors `MalformedSseId.raw` —
|
||||
both stored as truncated `str` attributes on the exception.
|
||||
- **INV-005 [hard]**: No `core.*` / `worldtree.*` imports (existing
|
||||
boundary; unchanged).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Diagnosing upstream root cause.** Whether Worldtree is emitting
|
||||
empty-data events legitimately (keepalive) or as a server bug is a
|
||||
worldtree-dev question worth filing separately if frequency is
|
||||
non-trivial. Ratatoskr's robustness fix is independent.
|
||||
- **Telemetry on skipped events.** No stderr-label or counter for
|
||||
`[sse_keepalive]`-style. v1 silent skip; revisit if frequency turns
|
||||
out to matter for observability.
|
||||
- **Auto-reconnect on `MalformedSseData`.** `reconnect_turn` exists for
|
||||
clean drops; whether to invoke it after a malformed frame is policy
|
||||
and currently the consumer's call. Presenters today don't auto-
|
||||
reconnect on any error class; that pattern stays unchanged here.
|
||||
- **Whitespace-only data handling beyond malformed-raise.** Treating
|
||||
`' '` or `'\\n'` as malformed (per the raise rule) is the v1 stance;
|
||||
if a real-world server emits whitespace keepalives, file a follow-up.
|
||||
- **`SseConnectionDropped` reclassification.** Existing wire-level
|
||||
exceptions (`ReadError`, `RemoteProtocolError`, `ReadTimeout`,
|
||||
clean-EOF-before-terminal) continue to map to `SseConnectionDropped`.
|
||||
`MalformedSseData` is a distinct class: payload-level malformation,
|
||||
not transport-level failure.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **[compatibility]** Spec pin unchanged. The empty-data handling is a
|
||||
client-side robustness change; nothing about the wire contract moves.
|
||||
- **[performance]** A skipped event costs one extra branch + continue in
|
||||
the loop. Negligible.
|
||||
- **[security]** `MalformedSseData.raw` truncates to 200 chars to bound
|
||||
the exception body — same hygiene as `MalformedSseId.raw[:64]` and
|
||||
`SseConnectFailed.body[:1024]`.
|
||||
- **[style]** Exception class follows existing pattern (frozen-ish:
|
||||
`__init__` stores `self.raw`, `super().__init__(message)`). Ruff
|
||||
line-length=100.
|
||||
|
||||
---
|
||||
|
||||
## In-place amendments (the work)
|
||||
|
||||
This issue's contract is small because the real work is amendments to
|
||||
issues #1, #3, #4. The amendments are pinned here so reviewers see the
|
||||
whole change in one place; the actual contract files are amended in-place
|
||||
as part of this issue's commit.
|
||||
|
||||
### Issue #1 (`ratatoskr.sse_client`) amendments
|
||||
|
||||
**New public exception:**
|
||||
|
||||
```python
|
||||
class MalformedSseData(Exception):
|
||||
"""Raised when an SSE event's `data:` payload is non-empty but fails JSON parsing."""
|
||||
|
||||
def __init__(self, raw: str) -> None:
|
||||
super().__init__(f"malformed SSE data: {raw[:200]!r}")
|
||||
self.raw = raw[:200]
|
||||
```
|
||||
|
||||
**`_iter_events` STEPS amended:**
|
||||
|
||||
```
|
||||
FOR EACH sse in event_source.aiter_sse():
|
||||
IF sse.data == "":
|
||||
# INV-001: empty-data event is treated as a keepalive; skip silently.
|
||||
# ORDERING NOTE: this branch fires BEFORE _parse_sse_id(sse.id). An
|
||||
# event with empty data AND a malformed id is therefore silently
|
||||
# skipped (MalformedSseId is NOT raised). This is intentional — a
|
||||
# keepalive with a bad id is still a keepalive, and the user-visible
|
||||
# behavior is "the frame had no content; ignore it". Don't reorder
|
||||
# the id-parse before the empty-skip check.
|
||||
# Do NOT yield, do NOT update last_sse_id, do NOT raise.
|
||||
CONTINUE
|
||||
|
||||
CALL _parse_sse_id(sse.id) → SseId # existing; only runs for non-empty-data events
|
||||
... existing turn_id checks ...
|
||||
|
||||
TRY:
|
||||
body = json.loads(sse.data)
|
||||
CATCH json.JSONDecodeError as exc:
|
||||
# INV-002: malformed data is a protocol error; surface as named exception.
|
||||
RAISE MalformedSseData(raw=sse.data)
|
||||
|
||||
event = _envelope_for_type(body, sse_id=sse_id)
|
||||
YIELD event
|
||||
... existing terminal-event handling ...
|
||||
```
|
||||
|
||||
**`stream_turn` ERROR_ROUTING amended:**
|
||||
|
||||
```
|
||||
MalformedSseData:
|
||||
local_handling: log truncated raw `data:` payload; re-raise to caller
|
||||
flow_control: abort
|
||||
state_recovery: none (payload corruption; can't decode; consumer's call to reconnect)
|
||||
```
|
||||
|
||||
(Mirrors the existing `MalformedSseId` entry shape.)
|
||||
|
||||
**New TESTS entries on `_iter_events` / `stream_turn`:**
|
||||
|
||||
- `empty_data_skipped [trace]`: stream yields four frames in order —
|
||||
`text(42:1)`, `empty-data(42:2 — id present, data empty)`, `text(42:3)`,
|
||||
`done(42:4)`. Consumer yields exactly three Events: `Text(42:1)`,
|
||||
`Text(42:3)`, `Done(42:4)` — the empty-data frame at `42:2` is
|
||||
invisible to the caller. Per INV-003, intermediate state after
|
||||
consuming the second `Text` (i.e., immediately after yielding the
|
||||
event with `sse_id=(42, 3)`) is `last_sse_id == (42, 3)` — proves the
|
||||
skip did NOT advance through `42:2`. Final state after consuming
|
||||
`Done` is `last_sse_id == (42, 4)` (the terminal event advances it as
|
||||
normal).
|
||||
- `malformed_data_raises [error]`: stream yields text(42:1) then event
|
||||
with `data: not-json` → consumer yields one Text then raises
|
||||
`MalformedSseData(raw="not-json")`. Iteration aborts.
|
||||
- `whitespace_data_raises [adversarial]`: stream yields one event with
|
||||
`data: ` (single space) → consumer raises `MalformedSseData` (whitespace
|
||||
is non-empty and JSON-parse-fails; the skip rule is empty-string only).
|
||||
- `malformed_data_truncation [security]`: stream yields one event with
|
||||
5000-char malformed data → `MalformedSseData.raw` is exactly the first
|
||||
200 chars; exception message includes only the truncated form.
|
||||
|
||||
**`assumptions` block adds**: empty `httpx_sse.ServerSentEvent.data`
|
||||
behavior across the library's parsing modes.
|
||||
|
||||
### Issue #3 (`ratatoskr.cli`) amendments
|
||||
|
||||
**`_run_turn` ERROR_ROUTING gains `MalformedSseData`:**
|
||||
|
||||
```
|
||||
MalformedSseData:
|
||||
local_handling: write `[malformed_sse_data] raw={exc.raw!r}` to stderr
|
||||
flow_control: abort
|
||||
state_recovery: none (server-side wire bug)
|
||||
exit_code: 22
|
||||
```
|
||||
|
||||
**New TEST:**
|
||||
|
||||
- `malformed_sse_data [error]`: mock yields one Text then an event with
|
||||
`data: not-json` → `_run_turn` returns 22; stderr `[malformed_sse_data]`
|
||||
is present; `raw=...` truncated.
|
||||
|
||||
### Issue #4 (`ratatoskr.tui`) amendments
|
||||
|
||||
**`_stream_turn_worker` ERROR_ROUTING gains `MalformedSseData`:**
|
||||
|
||||
```
|
||||
MalformedSseData:
|
||||
local_handling: log.write(f"[malformed_sse_data] raw={exc.raw!r}") to transcript
|
||||
flow_control: abort (the iteration aborts; finally-block restores state)
|
||||
state_recovery: state → idle; footer hint reset (INV-008: mid-session errors do NOT exit the app)
|
||||
```
|
||||
|
||||
**New TEST:**
|
||||
|
||||
- `malformed_sse_data_returns_to_idle [error]`: mock yields text + event
|
||||
with malformed data → transcript has `[malformed_sse_data]`; state →
|
||||
idle; app does NOT exit.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance
|
||||
|
||||
- All three amended contracts (#1, #3, #4) drift-check clean.
|
||||
- All existing tests + new `MalformedSseData` coverage GREEN under
|
||||
`uv run pytest tests/`.
|
||||
- `uv run ruff check src/ tests/` clean.
|
||||
- Boundary smoke `tests/test_no_worldtree_imports.py` still passes.
|
||||
- **Regression smoke**: re-run the long-conversation TUI smoke that
|
||||
previously crashed (mimic the 2026-05-22 crash conditions — a turn
|
||||
long enough that mimir emits ≥1000 events). Confirm no crash; the
|
||||
conversation completes cleanly to `[done]`.
|
||||
- The pre-existing happy-path smokes (mimir short responses) still
|
||||
work — empty-data skip is invisible when no empty-data events
|
||||
occur.
|
||||
Reference in New Issue
Block a user