From b154bb388527b733e8bfb81c63fd37a59a1c1b4f Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 15 Jul 2026 22:21:48 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20verify=20spawn?= =?UTF-8?q?ed=20cross-project=20R39=20(bifrost=20cursor=20ruling,=20WT=20#?= =?UTF-8?q?364,=20brokkr=20export);=20personal-data=20export=20gated=20on?= =?UTF-8?q?=20operator=20consent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index 77b38f8..298896e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -50,7 +50,9 @@ _As of 2026-07-15:_ **⚠️ DEPLOY-BREAKER caught by driving the live bind (v0.20.14, `f46ccba`):** I first advertised `sortable_chunk_fields=[{"name":"updated_at"}]` WITHOUT `type` — bifrost's `handshake_response` `SortableChunkField` schema requires BOTH name+type (`additionalProperties:false`), so the handshake_response failed wire-schema validation → `bifrost.schema_validation_failed` → the ENTIRE bind (memory+affect) broke, not just sort. Unit tests + worldtree-dev's name-only service parser + the heid-bug-hunt ALL missed it; only DRIVING the real handshake (`/verify` discipline) caught it. Fixed → `{"name":"updated_at","type":"timestamp"}` (matches reference; type advisory-only) + regression guard in the caps test (asserts required name+type, no extra keys). LESSON: validate `describe_store` against the bifrost wire schema, not just our own caps assertions. -**LIVE VERIFY (2026-07-15, drove a bound Sindra turn via `ratatoskr --new --agent ratatoskr:sindra --send … --bifrost-url http://10.100.10.50:8392 --end-user-id ratatoskr-tui`):** GATE LIT + scan fired ONCE at turn 1 with worldtree-dev's exact args (`scope_all={end_user:ratatoskr-tui} cursor=null limit=3 sort={updated_at,desc}`) → 3 records in ~5ms (no 500ms fail-open); injection confirmed in Sindra's CoT; **cross-session recognition WORKS** (she recalls him as a distinct person + patterns — blank-slate GONE). **BUT name-recall FAILS** — `"Name is Vuong."` is the OLDEST chunk (07:09) → excluded from top-3-by-recency AND scores 0.354 (sub-0.45) on the query path, so injects via NEITHER path; meanwhile a STALE contradictory `"user has not yet provided their name"` (07:58, 0.46) IS injected → Sindra concludes she lacks the name. **ROOT CAUSE = WORLDTREE ranking/hygiene** (not our wire): recency-top-3 under-weights early identity facts + the "no name" fact was never `mark_superseded` when "Name is Vuong" landed → both live+contradictory. Reported to worldtree-dev (thread `01KXK86PZ9…`) with 3 options: memory hygiene (supersede), person-prime salience-weighting, or higher N. AWAITING their call; our harness + 7 real facts sit ready for a re-drive. WHY: Sindra didn't remember Vuong's name across sessions; worldtree-dev root-caused it — WT injects a recalled fact only if combined score (sim×salience) ≥ **0.45** (`auto_inject_combined_score_threshold`, `core/memory/context_promotion/config.py`), and recall is per-turn **query-gated**, so moderate-sim durable facts (name hit ~0.40) never inject. The designed **turn-0** fix is WT **#349 person-prime** — a query-LESS top-N-by-recency durable-fact injection — capability-gated on the store advertising **`updated_at` in `sort_fields`** at the Bifrost handshake, DARK for our provider until now. Fix (ZERO Worldtree change): implement the sorted `scan` verb + advertise the capability. **✅ CODE COMPLETE + COMMITTED (2026-07-15):** contract amended + validated; `scan` impl in `src/ratatoskr/provider/memory_store.py` (globally-ordered by indexed `updated_at`, live-only via `_is_live`, offset-cursor; `has_more` peek so an exact-fill page emits no empty trailing cursor = reference-parity; `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; expression index `idx_chunks_updated_at`); **TDD 7/7 GREEN** (scan_recency tracer + live_only + scope_isolation + unadvertised_sort + person_prime_record_shape + cursor pagination + **`parity_vs_reference` #195**); NUANCE resolved — the bifrost ref `InMemoryMemoryStore` **READS** `updated_at` from the record (never stamps it; its own comment confirms), identical to ours, so order-parity holds over the live set (ref does NOT lifecycle-filter, so parity is defined over live chunks only). **Full suite 638 green.** Committed as **two patches**: `8fc757a` scan (**v0.20.11**) + `a9c521a` Sindra holodesk first-message preset (**v0.20.12**, split out as a separate concern from the soong-lab redefine). **heid-bug-hunt RETURNED + TRIAGED** (panel Gróa/Hulda/Regin, thread `01KXK5XTYHV8TGEDRAZV8GRXWC`; base-ref 39050c3, code-only diff). **FIXED (v0.20.13, `25ccb5c`):** the non-dict `sort` crash — a truthy non-dict `sort` (`"updated_at"`/`["updated_at"]`/int) hit `(sort or {}).get(...)` → AttributeError instead of InvalidArguments; added isinstance guard (Gróa#1/Hulda#2, confirmed-from-code). **SURFACED (operator design decision, NOT auto-adopted):** offset-cursor cross-page duplicate/drop under concurrent mutation (all 3 arms) — real divergence from bifrost `InMemoryMemoryStore` (snapshots the ordered id-list per opaque cursor + `ScanCursorExpired` TTL); BUT person-prime uses a SINGLE page (no cursor) so the actual use isn't exercised → operator picks: adopt reference snapshot-cursors vs keyset vs accept + honestly scope INV-010 to a static store. **NOTED (reference-parity / accept-known-risk, no fork):** full-table materialize-then-Python-filter (matches ref's own O(store) iteration), unbound limit, falsy scope coercion `[]`→`{}` (parity w/ search + ref), lexical timestamp ordering (WT writes consistent ISO-8601 UTC); `_is_live` dead-marker completeness (Gróa#6 needs-repro) → confirm canonical marker set w/ worldtree-dev, don't guess. **REFUTED:** Regin's mixed-type ORDER BY "crash" (SQLite orders by storage class, doesn't raise) + its self-retracted `_is_live`/None item. **✅ DEPLOYED (2026-07-15, operator-authorized):** (5) **DONE** — restarted BOTH `:8392` combined provider (new pid, old 982284 gone) AND `:8765` web on **v0.20.13** via the env-preserving `scratchpad/relaunch_by_pid.py ` (captures /proc cmdline+environ+cwd → byte-identical config, only code changes; self-daemonizes). Verified: `:8392` up (uvicorn startup complete; `/`→404 = no root route = serving), deployed `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; `:8765` up HTTP 200 (log "ratatoskr-web v0.20.13"). **Added the missing REQUIRED `RATATOSKR_MEMORY_EMBEDDING_DIM=1024` to `env.sh`** (gitignored; was absent → bare `source env.sh` restart would crash). (6) **DONE** — pinged **worldtree-dev** (thread `01KXK86PZ9M8J53A4SFXREEMW7`) to verify turn-1 recall against personal :8081. Store has **7 REAL live Sindra durable facts** ready (agent_id `ratatoskr:sindra`, worldtree_scope `end_user`, scope `{end_user:ratatoskr-tui}`, all carry `updated_at`, accumulated across 3 sessions today) → top-N-by-recency has real data to inject. AWAITING their turn-1 verdict (monitor armed). (7) **F3 cursor RESOLVED — operator ACCEPTED offset-for-v1** (keep offset cursor; person-prime single-page is correct/conformant). INV-010 amended to mark the cursor **v1-provisional / KNOWN DEVIATION** (multi-page continuation diverges from bifrost's protocol snapshot-cursor contract — the `bifrost.memory` dispatch drops `sort` on continuation citing "snapshotted order is authoritative" + maps `ScanCursorExpired→410`; our offset cursor doesn't snapshot → dup/drop under concurrent write, and never expires). **Durable fix = a BIFROST completeness concern (operator's catch), ROUTED to bifrost-dev** (thread `01KXK7MDTY0B6VEDF3SVGSKY9R`): bifrost's conformance suite has ZERO scan/cursor coverage, so a non-snapshot cursor passes silently — asked bifrost-dev to (1) confirm snapshot-cursor+ScanCursorExpired is the NORMATIVE store contract, (2) add scan/cursor conformance coverage; we'll adopt the reference snapshot-cursor semantics IF they rule it normative (offered to be their canary as first non-reference scan implementer). AWAITING bifrost-dev ruling; monitor armed. +**LIVE VERIFY (2026-07-15, drove a bound Sindra turn via `ratatoskr --new --agent ratatoskr:sindra --send … --bifrost-url http://10.100.10.50:8392 --end-user-id ratatoskr-tui`):** GATE LIT + scan fired ONCE at turn 1 with worldtree-dev's exact args (`scope_all={end_user:ratatoskr-tui} cursor=null limit=3 sort={updated_at,desc}`) → 3 records in ~5ms (no 500ms fail-open); injection confirmed in Sindra's CoT; **cross-session recognition WORKS** (she recalls him as a distinct person + patterns — blank-slate GONE). **BUT name-recall FAILS** — `"Name is Vuong."` is the OLDEST chunk (07:09) → excluded from top-3-by-recency AND scores 0.354 (sub-0.45) on the query path, so injects via NEITHER path; meanwhile a STALE contradictory `"user has not yet provided their name"` (07:58, 0.46) IS injected → Sindra concludes she lacks the name. **ROOT CAUSE = WORLDTREE ranking/hygiene** (not our wire): recency-top-3 under-weights early identity facts + the "no name" fact was never `mark_superseded` when "Name is Vuong" landed → both live+contradictory. Reported to worldtree-dev (thread `01KXK86PZ9…`) with 3 options: memory hygiene (supersede), person-prime salience-weighting, or higher N. + +**CROSS-PROJECT FOLLOW-UPS (2026-07-16, spawned by the verify — 4 inbound drained):** (1) **bifrost-dev RULED** scan cursor snapshot-semantics NORMATIVE (offset NOT blessed) + shipped conformance coverage in **bifrost 1.1.3** (`bifrost.conformance.memory_store_conformance`, 4 probes; our offset bug ships as their negative-canary fixture). TODO (OPERATOR-SEQUENCED, not urgent — single-page person-prime already conformant): bump pin 1.1.1→1.1.3, adopt snapshot cursor (frozen id-list per opaque token + `ScanCursorExpired`→410, match `InMemoryMemoryStore`), run `run_all(include_optional=True)` → expect P1/P2/P3 RED→green; we're their canary. Replied ack'd (thread `01KXK7MDTY…`). (2) **worldtree-dev FILED the hygiene bug as WT #364** (promotion writes negative-knowledge facts w/o contradiction-reconciliation; our specimen = evidence base); #349 ranking/N decision (N=3 NOT consumer-tunable) is going to **Vuong** with their recommendation. (3) **brokkr R39 Phase-1 Arm-0 fusion bake-off** wants our frozen 7-fact store exported DECOMPOSED (data contract in `~/development/brokkr-smithy/research/R39-memory-salience-dreams-surfacing/phase-1/re-drive-design.md`; #364 reconciliation GATED on brokkr's re-drive-complete). **KEY STRUCTURAL FINDINGS:** our store persists ONLY `salience` (+embedding) — `similarity`/`combined` are WT-side query-time (`bifrost_memory_store.py:702`, combined=sim×salience); **person-prime is query-LESS → carries NO similarity** (= brokkr's Arm-1 finding); recovered per-fact similarity from the verify SEARCH log (name=0.354, stale=0.463, ×salience-1.0 = combined, matches brokkr's 0.354/0.46 grounding); **BOTH the true "Name is Vuong" AND the stale negative have salience=1.0** (salience can't discriminate — stale wins on recency + sim only); they share the (user,name) supersession slot (Phase-0 Q3). **⚠️ SPECIMEN CONTAMINATED BY MY VERIFY:** the turn-1 drive ADDED 3 chunks (16:28, re-extractions incl. a 3RD "name unknown" negative) → store is now **10, not 7**; the original 7 intact (≤08:42). **⛔ EXPORT BLOCKED on operator consent:** classifier denied writing the personal memory facts (name/behaviors/intimate prefs) to shared `/mnt/smithy` — his personal data → his call on approve + destination. HOLDING brokkr + worldtree-dev PII-touching replies pending Vuong's OK (bifrost-dev reply was technical, sent). **HOLD HONORED — NO cleanup on the specimen** (R39 needs it frozen-dirty; the earlier operator "nudge" option is now OFF). Our harness + facts sit ready for the re-drive. WHY: Sindra didn't remember Vuong's name across sessions; worldtree-dev root-caused it — WT injects a recalled fact only if combined score (sim×salience) ≥ **0.45** (`auto_inject_combined_score_threshold`, `core/memory/context_promotion/config.py`), and recall is per-turn **query-gated**, so moderate-sim durable facts (name hit ~0.40) never inject. The designed **turn-0** fix is WT **#349 person-prime** — a query-LESS top-N-by-recency durable-fact injection — capability-gated on the store advertising **`updated_at` in `sort_fields`** at the Bifrost handshake, DARK for our provider until now. Fix (ZERO Worldtree change): implement the sorted `scan` verb + advertise the capability. **✅ CODE COMPLETE + COMMITTED (2026-07-15):** contract amended + validated; `scan` impl in `src/ratatoskr/provider/memory_store.py` (globally-ordered by indexed `updated_at`, live-only via `_is_live`, offset-cursor; `has_more` peek so an exact-fill page emits no empty trailing cursor = reference-parity; `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; expression index `idx_chunks_updated_at`); **TDD 7/7 GREEN** (scan_recency tracer + live_only + scope_isolation + unadvertised_sort + person_prime_record_shape + cursor pagination + **`parity_vs_reference` #195**); NUANCE resolved — the bifrost ref `InMemoryMemoryStore` **READS** `updated_at` from the record (never stamps it; its own comment confirms), identical to ours, so order-parity holds over the live set (ref does NOT lifecycle-filter, so parity is defined over live chunks only). **Full suite 638 green.** Committed as **two patches**: `8fc757a` scan (**v0.20.11**) + `a9c521a` Sindra holodesk first-message preset (**v0.20.12**, split out as a separate concern from the soong-lab redefine). **heid-bug-hunt RETURNED + TRIAGED** (panel Gróa/Hulda/Regin, thread `01KXK5XTYHV8TGEDRAZV8GRXWC`; base-ref 39050c3, code-only diff). **FIXED (v0.20.13, `25ccb5c`):** the non-dict `sort` crash — a truthy non-dict `sort` (`"updated_at"`/`["updated_at"]`/int) hit `(sort or {}).get(...)` → AttributeError instead of InvalidArguments; added isinstance guard (Gróa#1/Hulda#2, confirmed-from-code). **SURFACED (operator design decision, NOT auto-adopted):** offset-cursor cross-page duplicate/drop under concurrent mutation (all 3 arms) — real divergence from bifrost `InMemoryMemoryStore` (snapshots the ordered id-list per opaque cursor + `ScanCursorExpired` TTL); BUT person-prime uses a SINGLE page (no cursor) so the actual use isn't exercised → operator picks: adopt reference snapshot-cursors vs keyset vs accept + honestly scope INV-010 to a static store. **NOTED (reference-parity / accept-known-risk, no fork):** full-table materialize-then-Python-filter (matches ref's own O(store) iteration), unbound limit, falsy scope coercion `[]`→`{}` (parity w/ search + ref), lexical timestamp ordering (WT writes consistent ISO-8601 UTC); `_is_live` dead-marker completeness (Gróa#6 needs-repro) → confirm canonical marker set w/ worldtree-dev, don't guess. **REFUTED:** Regin's mixed-type ORDER BY "crash" (SQLite orders by storage class, doesn't raise) + its self-retracted `_is_live`/None item. **✅ DEPLOYED (2026-07-15, operator-authorized):** (5) **DONE** — restarted BOTH `:8392` combined provider (new pid, old 982284 gone) AND `:8765` web on **v0.20.13** via the env-preserving `scratchpad/relaunch_by_pid.py ` (captures /proc cmdline+environ+cwd → byte-identical config, only code changes; self-daemonizes). Verified: `:8392` up (uvicorn startup complete; `/`→404 = no root route = serving), deployed `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; `:8765` up HTTP 200 (log "ratatoskr-web v0.20.13"). **Added the missing REQUIRED `RATATOSKR_MEMORY_EMBEDDING_DIM=1024` to `env.sh`** (gitignored; was absent → bare `source env.sh` restart would crash). (6) **DONE** — pinged **worldtree-dev** (thread `01KXK86PZ9M8J53A4SFXREEMW7`) to verify turn-1 recall against personal :8081. Store has **7 REAL live Sindra durable facts** ready (agent_id `ratatoskr:sindra`, worldtree_scope `end_user`, scope `{end_user:ratatoskr-tui}`, all carry `updated_at`, accumulated across 3 sessions today) → top-N-by-recency has real data to inject. AWAITING their turn-1 verdict (monitor armed). (7) **F3 cursor RESOLVED — operator ACCEPTED offset-for-v1** (keep offset cursor; person-prime single-page is correct/conformant). INV-010 amended to mark the cursor **v1-provisional / KNOWN DEVIATION** (multi-page continuation diverges from bifrost's protocol snapshot-cursor contract — the `bifrost.memory` dispatch drops `sort` on continuation citing "snapshotted order is authoritative" + maps `ScanCursorExpired→410`; our offset cursor doesn't snapshot → dup/drop under concurrent write, and never expires). **Durable fix = a BIFROST completeness concern (operator's catch), ROUTED to bifrost-dev** (thread `01KXK7MDTY0B6VEDF3SVGSKY9R`): bifrost's conformance suite has ZERO scan/cursor coverage, so a non-snapshot cursor passes silently — asked bifrost-dev to (1) confirm snapshot-cursor+ScanCursorExpired is the NORMATIVE store contract, (2) add scan/cursor conformance coverage; we'll adopt the reference snapshot-cursor semantics IF they rule it normative (offered to be their canary as first non-reference scan implementer). AWAITING bifrost-dev ruling; monitor armed. **✅ soong-lab interop ESTABLISHED (this session) — soong-lab is now our Tier-3 agent-authoring studio.** ratatoskr consumes soong-lab bundles → WT `agents.define`, and ships agents back as bundles. Sindra round-trip PROVEN (soong imported her `resume` half through real `import_bundle`). soong-lab export/importer contracts PINNED via canonical-sync (`canonical_source=soong-lab` @ f434016, commit `39050c3`). Role-grounding (worldtree-dev): the 4 soong-lab ROLE_CHOICES = WT model-role slugs 1:1 BY NAME — `character`/`thoughtful-character` need the `character` grant (HELD via Sindra → define-able now); `assistant`/`thoughtful-assistant` need `foundational` (confirm/mint routed to infra-ops, in flight; only for assistant-type bundles). `character-rp` ≡ `thoughtful-character` (same Deckard seat + grant). **Sindra REDEFINED from the bundle** (`/tmp/sindra.json`, operator "update to match"): persona immutable → DELETE+redefine (payload validated via a throwaway `sindra-probe2`→201 BEFORE deleting the real one), `memory:{}` preserved, motivational string→WT-object mapped (Tried/abandoned), role→`thoughtful-character`, first_message preset updated. Verified live (thoughtful-character, 25574-char prompt).