memory: snapshot — memory provider live-proven (persist/dispatch/search); recall-injection upstream; #17 filed

persistent-memory.md

@@ -56,18 +56,21 @@ background shell (`ratatoskr-provider`, `0.0.0.0:8390`, env-sourced from
 Worldtree **v0.35.2** (`10.250.50.152`): handshake 200 + `affect.emit` 200 →
 durable row persisted, opacity held.
 
-**MEMORY plane: SHIPPED** (v0.17.3, commit `cd12951`). SQLite+sqlite-vec store
-(`src/ratatoskr/provider/memory_store.py`) + dev shell (`serve_memory.py`,
-`ratatoskr-memory-provider`, port 8391, env `RATATOSKR_MEMORY_DB` /
-`RATATOSKR_MEMORY_EMBEDDING_DIM`). All 7 contract blocks; 26 store + 4 serve
-tests; **#195 parity** (upsert / search / expected_revisions) vs
-`InMemoryMemoryStore` through the real `dispatch_memory_call`; heid-code-reviewed
-(panel Gróa/Hulda/Regin — zero true drift). NEXT (the affect-parallel
-**live-proven** milestone): smoke `ratatoskr-memory-provider` against personal
-Worldtree **v0.35.3** — needs the real `PINNED_EMBEDDER_DIM`
-(`RATATOSKR_MEMORY_EMBEDDING_DIM`, required, no default — a wrong dim silently
-breaks vec search) + a heimdall key (broker via infra-ops) + `provider.env`.
-That smoke is what earns a possible minor bump + worldtree-dev announce.
+**MEMORY plane: PROVIDER LIVE-PROVEN** (v0.17.3, commit `cd12951`; provider
+running on `0.0.0.0:8391` as a dev bg shell). Store + dev shell shipped (7
+contract blocks, 26+4 tests, #195 parity, heid-reviewed zero-drift) AND the
+live round-trip ran against personal **v0.35.3**: handshake + `upsert_many`
+(persist) + `search` (recall-dispatch) all green; `memory.db` holds 3 durable
+chunks (2 facts + 1 promoted question, scope `end_user:smoke-user`, 1024-d
+vec-indexed). **Proven: persist + dispatch + our search-correctness** — direct
+probe (embed the recall query via gateway `qwen3-embedding`, search our live
+store) recalls the right fact at cosine 0.60, correctly ranked. **NOT proven,
+and upstream (not ours): end-to-end recall-INJECTION** — Worldtree's
+recall-assembly doesn't surface our hits into the prompt (cold-session probe:
+search dispatched 200 but the model said "no memory"). worldtree-dev looped in
+(thread `01KV7JH8…`; #295 held open). Provider stays up with the live fixture.
+So NO minor bump / "all-green" announce until worldtree-dev closes the injection
++ question-promotion gaps.
 
 **Sindra:** a REGISTERED Tier-3 agent (`ratatoskr:sindra`, was model
 `artemis-31b-v1i`) — registration is REQUIRED to use a Tier-3 character (a
@@ -83,10 +86,10 @@ registration.
 `~/.config/ratatoskr/provider.env` (mode 600, nh3-dev) — `consumer="ratatoskr"`,
 HS256 = the API-key STRING utf-8-encoded; rotate via infra-ops.
 
-**Unpushed:** `main` is 5 commits ahead of origin (origin at `5cdb69a`) —
-`d85ab43` (canonical sync), `335c835` (contract refresh), `e9434b9` (sindra),
-`cf411cb` (graph report), `cd12951` (memory plane v0.17.3) + the local `v0.17.3`
-tag are unpushed. Push is the operator's call.
+**Pushed:** `main` is in sync with `origin` (pushed through `e57b054`); tags
+`v0.17.3` + a straggler `v0.8.2` pushed. Only `graphify-out/GRAPH_REPORT.md`
+runs dirty (auto-regenerated by the commit hook — generated artifact, not
+chased).
 
 **Still standing from before:** Worldtree spec pin v0.29.0 (`562001a`) for the
 conversation-API/TUI surface (untouched by the Bifrost work). Codex-first pilot
@@ -179,3 +182,7 @@ defense against re-attempting the same cul-de-sac.
 - `[2026-06-15]` **Canonical sync retired the issue-scoped parser staleness** (predicted by the 2026-05-21 entry's "until canonical bumps"). `contract_parser.py` synced to v2.1 (`f1fdfdb6→e10a4460`, commit `d85ab43`): now validates issue-scoped frontmatter (`target_module`/`scope`/`prd`) + four v2.1 test categories (scenario/trace/adversarial/property). Issues #3/#4 went FAIL→WARN (0 errors). The old "treat parser ERROR-on-issue-scoped as expected" note no longer applies.
 - `[2026-06-15]` **Refreshed #3/#4 presenter contracts to the shipped TUI model** (commit `335c835`). Both still described the abandoned single-`RichLog` double-display model; rewrote to the 4-pane live-Markdown reality (v0.5.0–v0.14.0 + Worldtree #201/#204) across INV-005, the `[performance]` constraint, the COMPOSE sketch, the `CLASS TuiPresenterState` block, both `render`/`_stream_turn_worker` blocks, and the `_cancel_via_sse` call site — plus the STEPS the v2.1 parser flagged missing. Code unchanged; contract-truth catching up to shipped code. Scope ballooned one-block→contract-wide mid-task; surfaced to operator before rewriting the INV-005 trade-off invariant.
 - `[2026-06-15]` **Memory plane TDD'd + shipped** (commit `cd12951`, v0.17.3). Impl decisions worth keeping: vec0 `distance_metric=cosine` set at table creation (`score = 1 − distance`); `search` over-fetches ALL candidates by cosine then scope-filters in Python so `top_k` counts IN-SCOPE hits (INV-005, contract STEP 2 `indicative`); idempotency_id = reference 4-tuple `("default",verb,_ctx_actor(ctx),key)` pipe-joined as the SQLite PK, digest = sha256 canonical-JSON; `_ctx_actor` = `job_id|jwt_sub|session_id` (memory reference's 3-level, vs affect's 2-level). **heid-code-review panel returned zero true drift**; adopted 5 cheap contract-anchored fixups (scope_filter dict guard, `top_k≤0→[]`, stronger scope-isolation / delete-hit-search / handshake-POST tests), accepted 6 with reasoning. **Partial-map optimistic-lock semantics pinned to the reference via an `expected_revisions` parity test** — resolved a Hulda finding deterministically (the affect-plane lesson: TDD against the shipped lib is the gate, not judgment).
+- `[2026-06-15]` **Memory provider LIVE-PROVEN against personal v0.35.3 (persist + dispatch + search-correctness); recall-injection is upstream.** worldtree-dev's Tier-3 promotion recipe (via infra-ops): memory-call fires from Tier-3 PROMOTION, gated at `service.py:2623` on `ctx.kind=="consumer_defined"` AND `ctx.memory_config is not None` (the agent must be DEFINED WITH a `memory` block — `ValidatedMemoryConfig {tier3_dreaming:false}`, dim 1024) AND handshake-granted memory caps AND `embedding_dim==1024`. `memory.agent_self_enabled` is NOT the gate (only the #248 self-candidate branch). Binding = `POST /sessions BifrostBindingRequest{endpoint_url}`, handshake `caps=["affect","memory"]`, **`binding.scope` null** (per-op scopes auto-minted: upsert_many→`memory:write`, search→`memory:read`). A `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist gates the endpoint (Worldtree-side config — infra-ops added `:8391`). HTTP + HS256 both work in dev. (smoke wiring thread `01KV7D82MJYB…`)
+- `[2026-06-15]` **Diagnostic: our recall-search is SOUND — the cross-session recall gap is UPSTREAM, not the store — and it caught an upstream bug.** Embedded the recall query via gateway `qwen3-embedding` + searched our live store directly → the dark-chocolate fact recalls at cosine 0.60, correctly ranked above the unrelated name fact (0.16). So the cold-session recall failure is Worldtree's recall-assembly/injection (hits not reaching the prompt), NOT our search. ALSO found a latent UPSTREAM bug: a recall QUESTION got promoted as a durable chunk and ranks **#1 (0.70 > the fact's 0.60)**, polluting recall. Relayed to worldtree-dev (thread `01KV7JH8…`). **This is exactly #17's thesis — ratatoskr-as-provider caught an upstream bug invisible from the chat side.**
+- `[2026-06-15]` **"Wire 200 ≠ recall works" — prove recall efficacy at the model's answer in a COLD (history-free) session, not on the wire.** A `search`/memory-call returns 200 whether or not its results are injected into the prompt, and same-session "recall" can be plain session history. infra-ops' cold cross-session probe caught my premature "all-green" (search dispatched 200, model had no memory). Don't call cross-session recall proven from a clean wire.
+- `[2026-06-15]` **Issue #17 filed — Bifrost-binding for the chat client (self-drive + correlated-log affect/memory ops).** REVERSES design-brief §6's "no Bifrost-binding consumer support" — that negative clause predates ratatoskr's provider identity (2026-06-14), so the canary now owns both ends but its client can't drive its own provider (`create_session` sends only `{agent_id, end_user_id}`; no Bifrost `endpoint_url`). Today's smoke proved the substrate (bind→dispatch→persist); only the observe/log channel design (open question #5) remains. The recall-injection caveat is upstream and doesn't block #17. NEXT on #17: `/heid` consult on the now-grounded framing → contract → TDD. (tracked: Gitea #17, labels enhancement/observability/tui)