feat(#1): shared SSE resume orchestration; wire cli --send

v1 coverage-audit slice b1. The audit found reconnect_turn had no
caller — every presenter dropped the stream on disconnect instead of
resuming, leaving the "reference SSE-resume implementation" (design-
brief §3/§8d) unreachable. Add stream_turn_resilient as the single
shared resume surface (design-brief §8b "share the consumer, branch
the presenter") and route cli --send through it.

- stream_turn_resilient wraps stream_turn + reconnect_turn: on
  SseConnectionDropped (mid-stream drop or clean EOF before terminal),
  resume from the last-seen sse_id via reconnect_turn (Last-Event-ID),
  up to max_reconnects (default 5). last_seen persists across attempts.
- Non-drop reconnect failures (412/410/400/TurnIdFlip/SseConnectFailed)
  propagate unchanged, per contract #1's "surface, not recover".
- cli.py: --send consumer now drives stream_turn_resilient (transparent
  reconnect). tui/web still consume bare stream_turn (follow-up).
- Contract #1 amended (FN stream_turn_resilient) + validated; 8 TDD
  cases (happy, resume-after-1/2-drops, clean-EOF resume, unresumable
  zero-event, max-reconnects-exhausted, zero-budget, buffer-expired-
  propagates). Suite 518 green; ruff + mypy clean on touched code.
This commit is contained in:
vh
2026-06-30 15:42:33 -07:00
parent 1f289098ba
commit 0c7660791f
7 changed files with 255 additions and 4 deletions
+48
View File
@@ -297,3 +297,51 @@ TESTS:
trailing_whitespace [adversarial]: "42:3 " → ValueError (strict; do not strip; the server emits clean ids)
truncation [security]: input is 5000-char string with no colon → ValueError message includes only `raw[:64]` (not the full 5000)
```
## Amendment 2026-06-30 — shared resume orchestration (v1 coverage-audit, slice b1)
The original contract specs resume as **caller-owned** (§Resume semantics: "the
caller MAY invoke `reconnect_turn`"). The v1 coverage-audit found `reconnect_turn`
had **no caller** — every presenter (cli/tui/web) let a mid-stream drop propagate
instead of resuming, so the "reference SSE-resume implementation" (design-brief
§3/§8d) was unreachable. Per design-brief §8b ("share the consumer, branch the
presenter") the resume loop is a **single shared orchestration surface**, not
duplicated per presenter. This adds `stream_turn_resilient` as that surface;
presenters call it instead of `stream_turn` when they want transparent reconnect.
`stream_turn` and `reconnect_turn` are unchanged (still the primitives); this is
purely additive.
```contract
FN stream_turn_resilient(client: httpx.AsyncClient, session_id: str, content: str, *, max_reconnects: int = 5) -> AsyncIterator[Event]
BRIEF: The shared resume-orchestration wrapper over stream_turn + reconnect_turn. Yields a SINGLE continuous typed Event stream; on SseConnectionDropped (mid-stream drop OR clean EOF before terminal), transparently resumes via reconnect_turn from the last-seen sse_id, up to max_reconnects times, until a terminal Done/Error/Cancelled arrives. The one surface all presenters consume for resilient streaming (design-brief §8b). Cross-process resume stays deferred to v2 (§8d): last-seen lives only in this generator's frame.
PRE: [PRE-001 hard] client is not None -- assert client is not None
PRE: [PRE-002 hard] session_id is non-empty str -- assert session_id and isinstance(session_id, str)
PRE: [PRE-003 hard] content is non-empty str -- assert content and isinstance(content, str)
PRE: [PRE-004 hard] max_reconnects is a non-negative int -- assert isinstance(max_reconnects, int) and max_reconnects >= 0
POST: [POST-001 return_value] yielded events are the concatenation of each attempt's events in wire order; the wrapper does NOT re-yield events it already saw (the server replays only seq>last_seen) -- assert seq is non-decreasing within a turn_id across the seam
POST: [POST-002 return_value] a fully-consumed stream terminates at exactly one Done/Error/Cancelled (INV-001 holds across reconnects) -- assert isinstance(events[-1], (Done, Error, Cancelled))
POST: [POST-003 state_change] reconnect_turn is invoked with last_event_id == f"{last_seen.turn_id}:{last_seen.seq}" of the most recently yielded event -- assert the Last-Event-ID header on attempt N+1 == the last sse_id yielded before the drop
ERROR_ROUTING:
SseConnectionDropped (from stream_turn or reconnect_turn):
local_handling: IF a last-seen sse_id exists AND reconnects < max_reconnects → increment reconnects, resume via reconnect_turn(last_event_id=f"{turn_id}:{seq}"); ELSE re-raise
flow_control: continue (resume) | abort (re-raise when no last-seen id, or budget exhausted)
state_recovery: server replays buffered events seq>last_seen then streams live (spec §Reconnect flow)
ResumeBufferExpired | ResumeTurnFinished | InvalidLastEventId | TurnIdFlip | SseConnectFailed (from reconnect_turn):
local_handling: propagate unchanged — NOT a transient drop; caller policy is abandon/restart (§Resume semantics "surface, not recover")
flow_control: abort
state_recovery: none
STEPS:
1. [setup, flexibility=prescriptive] Validate PRE-001..PRE-004; SET last_seen=None, reconnects=0, gen=stream_turn(client, session_id, content)
2. [loop, flexibility=prescriptive] async-for event in gen: SET last_seen=event.sse_id; YIELD event. On clean generator completion (terminal reached): RETURN.
3. [branch, flexibility=prescriptive] ON SseConnectionDropped d: SET seen = last_seen or d.last_seen_sse_id. IF seen is None OR reconnects >= max_reconnects: RE-RAISE. ELSE: reconnects += 1; gen = reconnect_turn(client, session_id, content, last_event_id=f"{seen.turn_id}:{seen.seq}"); GOTO step 2.
4. [error_handler, flexibility=prescriptive] Any non-drop exception from gen (ResumeBufferExpired/ResumeTurnFinished/InvalidLastEventId/TurnIdFlip/SseConnectFailed) is NOT caught — it propagates unchanged.
TESTS:
happy_no_drop [happy]: stream yields text(42:1), done(42:2) cleanly → wrapper yields exactly those 2; endpoint hit ONCE (no reconnect).
resume_after_one_drop [scenario,tracer]: attempt 1 yields text(42:1) then RemoteProtocolError; reconnect replays text(42:2)+done(42:3) → wrapper yields 42:1,42:2,42:3 as ONE stream; 2nd request carried Last-Event-ID "42:1".
resume_after_clean_eof [scenario]: attempt 1 yields text(42:1) then clean EOF (no terminal); reconnect yields done(42:2) → continuous (resumes on the INV-001 clean-eof drop too).
two_drops_then_done [scenario]: drops after 42:1 then after 42:2; third attempt yields done(42:3) → all 3 events; reconnects==2; Last-Event-ID headers "42:1" then "42:2".
unresumable_zero_event_drop [adversarial]: attempt 1 drops with ZERO events seen (last_seen None) → SseConnectionDropped propagates; only 1 request issued.
max_reconnects_exhausted [adversarial]: every attempt drops after one event; max_reconnects=2 → after initial + 2 reconnects (3 requests), SseConnectionDropped propagates.
buffer_expired_propagates [error]: attempt 1 drops after 42:1; reconnect returns 412 → ResumeBufferExpired propagates (not retried as a transient drop).
zero_budget_no_resume [adversarial]: max_reconnects=0; attempt 1 drops after 42:1 → SseConnectionDropped propagates immediately (no reconnect attempted).
```
+1
View File
@@ -149,6 +149,7 @@ decision. Captures rationale that won't be obvious from code alone.
- `[2026-06-30]` **Finding P-1 (pin drift) + pin-remediation PENDING.** We vendor the PROSE markdown (`docs/conversation-api-spec.md`), which is **byte-identical to live WT's** but frozen at v0.35.16-era content (last WT edit 2026-05-31) — it does NOT capture b2: 7 new endpoints (admin/keys/bulk, admin/persona/{archive,erase}, admin/usage, embed, judgments, me/usage), the 409/503 on messages-POST (#331), the unified error envelope (#328), or the SSE schema. **WT's authoritative v1 truth is now the FROZEN OpenAPI 2.2.0 + SSE-schema JSON** (`Worldtree/docs/v1-schema-freeze-manifest.md`). So the previously-deferred "re-vendor markdown to b2" is a **near-no-op** (markdown content identical). **Pending operator nod:** re-pin to the machine-readable artifacts (recommended — drift-checkable via `canonical_drift.py`, makes the coverage map reproducible vs a frozen diffable target) vs markdown-only. Deferred (not auto-applied) because it adds vendored artifacts + a canonical-sync pin = substrate change with CI-gating reach. **→ RESOLVED 2026-06-30 (operator: "a then b").** Vendored `conversation-api-openapi.json` (2.2.0) + `conversation-api-sse-events.schema.json` + re-copied the prose markdown; pinned all three in `.corviduo-canonicals.toml` (OpenAPI+SSE = strict drift gates, markdown = `tolerate_drift` reference); advanced `worldtree-spec-rev` f1b59f8→5810a26 + `worldtree-version` v0.29.0(STALE, never bumped from the v0.35.16 pin)→v1.0.0b2 + `pinned-on`→2026-06-30; SPEC-PIN.md history row added. `canonical_drift.py` green (10/10). `pin:`-only, no version bump (no client-facing code change; the b2 409/503 + error-envelope were already consumed in v0.18.3/.4).
- `[2026-06-30]` **(b) Tier-1 frontier SCOPED, ready for a contract-first TDD cycle (next focused work).** The primitives already exist + are contracted + tested; the gap is PRESENTER-level wiring. Two slices: **(b1) SSE-resume** — contract #1 (`ratatoskr.sse_client`) DELIBERATELY makes resume caller-owned ("on `SseConnectionDropped`, the caller MAY invoke `reconnect_turn`"); `reconnect_turn` (sse_client.py:524) has NO caller. Gap = a SHARED resume-orchestration wrapper (catch `SseConnectionDropped` → track last-seen `sse_id` → `reconnect_turn` → continue), consumed by all 3 presenters per design-brief §8b "share the consumer, branch the presenter" (NOT per-presenter — that forks the consumer). New function block → **amend contract #1** (additive FN, e.g. `stream_turn_resilient`) then TDD (RED: drop-mid-stream→resume continuity; GREEN: wrapper; wire `cli --send` first as the tracer). Resume design pre-locked: in-process Last-Event-ID only, cross-process deferred to v2 (design-brief §8d). **(b2) session-picker** — `list_sessions` (sessions.py:198) has NO caller; add a Textual DataTable startup picker (>1 session) + `--session <id>`/`--new` CLI flags (design-brief §4, decisions pre-locked). Both pre-locked → heid-contract-review likely skippable as ceremony (small additive amendments to mature specs); heid-code-review still valuable. **#11 AdminEvents stays BLOCKED** on `admin.events.read` scope (infra-ops).
- `[2026-06-30]` **(b1) SSE-resume SHIPPED (`v0.18.5`) — `stream_turn_resilient` (sse_client.py).** The shared resume-orchestration surface (design-brief §8b): wraps `stream_turn`+`reconnect_turn`, catches `SseConnectionDropped` (mid-stream drop OR clean-EOF-before-terminal) → resumes from last-seen `sse_id` via `reconnect_turn` (Last-Event-ID), up to `max_reconnects` (default 5); non-drop reconnect failures (412/410/400/TurnIdFlip/SseConnectFailed) PROPAGATE per contract #1's "surface, not recover". `last_seen` persists ACROSS attempts (a zero-event reconnect drop falls back to the prior attempt's id). Direct in-session TDD against a contract-#1 amendment (8 cases incl. two-drops, max-reconnects-exhausted, zero-budget, buffer-expired-propagates, unresumable-zero-event). Wired `cli --send` (`cli.py:396`, swapped `stream_turn`→`stream_turn_resilient`; **tui/web still on bare `stream_turn` — follow-up to route them through the wrapper**). Suite 518 green; ruff+mypy clean on touched code (pre-existing cli.py:400/543 mypy warts left untouched per surgical rule); contract #1 validates OK. **heid-code-review NOT run** (small additive well-TDD'd wrapper; offered to operator). **b2 (session-picker + `--session`/`--new` flags) still pending.**
_41 older entries (2026-05-* — the original debug-TUI/web build era) archived to archival-memory.md._
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "ratatoskr"
version = "0.18.4"
version = "0.18.5"
description = "Worldtree Conversation API debug TUI — multi-pane observability dashboard"
readme = "README.md"
requires-python = ">=3.12"
+2 -2
View File
@@ -47,7 +47,7 @@ from ratatoskr.sse_client import (
TurnIdFlip,
WorkerPhase,
cancel_turn,
stream_turn,
stream_turn_resilient,
)
@@ -393,7 +393,7 @@ async def _run_turn(
cancelling = False
sigint_task: asyncio.Task[bool] | None = None
cancel_task: asyncio.Task[None] | None = None # strong ref to fire-and-forget cancel
aiter_obj = stream_turn(client, session_id, content).__aiter__()
aiter_obj = stream_turn_resilient(client, session_id, content).__aiter__()
try:
while True:
+49
View File
@@ -549,6 +549,55 @@ async def reconnect_turn(
yield event
async def stream_turn_resilient(
client: httpx.AsyncClient,
session_id: str,
content: str,
*,
max_reconnects: int = 5,
) -> AsyncIterator[Event]:
"""Resume-orchestration wrapper over stream_turn + reconnect_turn.
Yields ONE continuous Event stream; on `SseConnectionDropped` (mid-stream
drop or clean EOF before a terminal), resumes from the last-seen `sse_id`
via `reconnect_turn`, up to `max_reconnects` times, until a terminal
Done/Error/Cancelled arrives. The single shared surface presenters consume
for resilient streaming (design-brief §8b: "share the consumer, branch the
presenter"). Cross-process resume stays deferred to v2 (§8d): `last_seen`
lives only in this generator's frame. See contract FN stream_turn_resilient
(amendment 2026-06-30).
"""
assert client is not None
assert session_id and isinstance(session_id, str)
assert content and isinstance(content, str)
assert isinstance(max_reconnects, int) and max_reconnects >= 0
last_seen: SseId | None = None
reconnects = 0
gen = stream_turn(client, session_id, content)
while True:
try:
async for event in gen:
last_seen = event.sse_id
yield event
return # generator completed cleanly → terminal event reached (INV-001)
except SseConnectionDropped as drop:
# Prefer the id we tracked from a yielded event; fall back to the one
# the drop carries (covers a drop on the very first frame). Non-drop
# reconnect failures (412/410/400/flip) are NOT caught here — they
# propagate per the contract's "surface, not recover" policy.
seen = last_seen or drop.last_seen_sse_id
if seen is None or reconnects >= max_reconnects:
raise
reconnects += 1
gen = reconnect_turn(
client,
session_id,
content,
last_event_id=f"{seen.turn_id}:{seen.seq}",
)
def _parse_sse_id(raw: str) -> SseId:
"""Parse the SSE wire `id:` as composite `{turn_id}:{seq}`. See contract FN _parse_sse_id."""
assert isinstance(raw, str)
+153
View File
@@ -19,6 +19,7 @@ from ratatoskr.sse_client import (
ResumeBufferExpired,
ResumeTurnFinished,
SseConnectFailed,
SseConnectionDropped,
SseId,
Text,
TurnIdFlip,
@@ -27,6 +28,7 @@ from ratatoskr.sse_client import (
cancel_turn,
reconnect_turn,
stream_turn,
stream_turn_resilient,
)
_DONE_42_6 = {
@@ -51,6 +53,36 @@ def _sse_chunk(sse_id: str, body: dict[str, object]) -> bytes:
return f"id: {sse_id}\ndata: {json.dumps(body)}\n\n".encode()
_EVENT_STREAM = {"content-type": "text/event-stream"}
class _DropStream(httpx.AsyncByteStream):
"""Yield the given chunks, then raise a mid-stream drop (RemoteProtocolError).
Mirrors the inline `_DropAfter` used by TestStreamTurn.test_connection_drop;
hoisted to module scope because the resilient-wrapper tests reuse it.
"""
def __init__(self, chunks: list[bytes]) -> None:
self._chunks = chunks
async def __aiter__(self): # type: ignore[no-untyped-def]
for c in self._chunks:
yield c
raise httpx.RemoteProtocolError("simulated mid-stream drop")
async def aclose(self) -> None:
return None
def _drop_response(chunks: list[bytes]) -> httpx.Response:
return httpx.Response(200, headers=_EVENT_STREAM, stream=_DropStream(chunks))
def _stream_response(content: bytes) -> httpx.Response:
return httpx.Response(200, headers=_EVENT_STREAM, content=content)
class TestParseSseId:
def test_happy_simple(self) -> None:
"""happy_simple [happy,tracer]: '42:3' -> SseId(turn_id=42, seq=3)."""
@@ -1108,3 +1140,124 @@ class TestAwaitingLlmFirstToken:
elapsed = [b.elapsed_ms_since_building_prompt for b in beats]
assert elapsed == sorted(elapsed) # monotonically increasing
assert all(b.turn_id == 42 for b in beats)
_URL = "https://w.example/sessions/s1/messages"
class TestStreamTurnResilient:
"""docs/contracts/issues/1.contract.md FN stream_turn_resilient (amendment 2026-06-30)."""
@respx.mock
async def test_happy_no_drop(self) -> None:
"""happy_no_drop [happy]: clean stream passes through; no reconnect issued."""
stream = _sse_chunk("42:1", {"type": "text", "content": "a"}) + _sse_chunk(
"42:2", _DONE_42_6
)
route = respx.post(_URL).mock(return_value=_stream_response(stream))
async with httpx.AsyncClient(base_url="https://w.example") as client:
events = [e async for e in stream_turn_resilient(client, "s1", "hi")]
assert [e.sse_id for e in events] == [SseId(42, 1), SseId(42, 2)]
assert isinstance(events[-1], Done)
assert route.call_count == 1 # POST-001: no reconnect on a clean stream
@respx.mock
async def test_resume_after_one_drop(self) -> None:
"""resume_after_one_drop [tracer]: a mid-stream drop resumes via reconnect; one stream."""
first = _drop_response([_sse_chunk("42:1", {"type": "text", "content": "a"})])
resume = _stream_response(
_sse_chunk("42:2", {"type": "text", "content": "b"})
+ _sse_chunk("42:3", _DONE_42_6)
)
route = respx.post(_URL).mock(side_effect=[first, resume])
async with httpx.AsyncClient(base_url="https://w.example") as client:
events = [e async for e in stream_turn_resilient(client, "s1", "hi")]
assert [e.sse_id for e in events] == [SseId(42, 1), SseId(42, 2), SseId(42, 3)]
assert isinstance(events[-1], Done)
assert route.call_count == 2
# POST-003: reconnect carries the last yielded pre-drop event's id.
assert route.calls[1].request.headers.get("Last-Event-ID") == "42:1"
# PRE/wire: first attempt does NOT carry a Last-Event-ID.
assert route.calls[0].request.headers.get("Last-Event-ID") is None
@respx.mock
async def test_resume_after_clean_eof(self) -> None:
"""resume_after_clean_eof: a clean EOF before terminal also triggers resume (INV-001)."""
first = _stream_response(_sse_chunk("42:1", {"type": "text", "content": "a"}))
resume = _stream_response(_sse_chunk("42:2", _DONE_42_6))
route = respx.post(_URL).mock(side_effect=[first, resume])
async with httpx.AsyncClient(base_url="https://w.example") as client:
events = [e async for e in stream_turn_resilient(client, "s1", "hi")]
assert [e.sse_id for e in events] == [SseId(42, 1), SseId(42, 2)]
assert isinstance(events[-1], Done)
assert route.calls[1].request.headers.get("Last-Event-ID") == "42:1"
@respx.mock
async def test_two_drops_then_done(self) -> None:
"""two_drops_then_done: two transient drops, third attempt completes; ids thread through."""
a1 = _drop_response([_sse_chunk("42:1", {"type": "text", "content": "a"})])
a2 = _drop_response([_sse_chunk("42:2", {"type": "text", "content": "b"})])
a3 = _stream_response(_sse_chunk("42:3", _DONE_42_6))
route = respx.post(_URL).mock(side_effect=[a1, a2, a3])
async with httpx.AsyncClient(base_url="https://w.example") as client:
events = [e async for e in stream_turn_resilient(client, "s1", "hi")]
assert [e.sse_id for e in events] == [SseId(42, 1), SseId(42, 2), SseId(42, 3)]
assert route.call_count == 3
assert route.calls[1].request.headers.get("Last-Event-ID") == "42:1"
assert route.calls[2].request.headers.get("Last-Event-ID") == "42:2"
@respx.mock
async def test_unresumable_zero_event_drop(self) -> None:
"""unresumable_zero_event_drop [adversarial]: drop before any event → propagate."""
route = respx.post(_URL).mock(side_effect=[_drop_response([])])
async with httpx.AsyncClient(base_url="https://w.example") as client:
with pytest.raises(SseConnectionDropped):
_ = [e async for e in stream_turn_resilient(client, "s1", "hi")]
assert route.call_count == 1 # no id to resume from → no reconnect
@respx.mock
async def test_max_reconnects_exhausted(self) -> None:
"""max_reconnects_exhausted [adversarial]: every attempt drops; budget caps reconnects."""
side = [
_drop_response([_sse_chunk(f"42:{n}", {"type": "text", "content": "x"})])
for n in (1, 2, 3)
]
route = respx.post(_URL).mock(side_effect=side)
async with httpx.AsyncClient(base_url="https://w.example") as client:
with pytest.raises(SseConnectionDropped):
_ = [
e
async for e in stream_turn_resilient(
client, "s1", "hi", max_reconnects=2
)
]
assert route.call_count == 3 # initial + 2 reconnects, then give up
@respx.mock
async def test_zero_budget_no_resume(self) -> None:
"""zero_budget_no_resume [adversarial]: max_reconnects=0 → first drop propagates."""
first = _drop_response([_sse_chunk("42:1", {"type": "text", "content": "a"})])
route = respx.post(_URL).mock(side_effect=[first])
async with httpx.AsyncClient(base_url="https://w.example") as client:
with pytest.raises(SseConnectionDropped):
_ = [
e
async for e in stream_turn_resilient(
client, "s1", "hi", max_reconnects=0
)
]
assert route.call_count == 1
@respx.mock
async def test_buffer_expired_propagates(self) -> None:
"""buffer_expired_propagates [error]: a 412 on reconnect surfaces, not retried."""
first = _drop_response([_sse_chunk("42:1", {"type": "text", "content": "a"})])
resume = httpx.Response(412, json={"turn_id": 42, "buffered_from_seq": 7})
route = respx.post(_URL).mock(side_effect=[first, resume])
async with httpx.AsyncClient(base_url="https://w.example") as client:
collected: list[object] = []
with pytest.raises(ResumeBufferExpired):
async for e in stream_turn_resilient(client, "s1", "hi"):
collected.append(e)
assert [e.sse_id for e in collected] == [SseId(42, 1)] # type: ignore[attr-defined]
assert route.call_count == 2
Generated
+1 -1
View File
@@ -1052,7 +1052,7 @@ wheels = [
[[package]]
name = "ratatoskr"
version = "0.18.4"
version = "0.18.5"
source = { editable = "." }
dependencies = [
{ name = "httpx" },