diff --git a/persistent-memory.md b/persistent-memory.md index 133f991..78a5ed2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -52,7 +52,7 @@ _As of 2026-07-15:_ **LIVE VERIFY (2026-07-15, drove a bound Sindra turn via `ratatoskr --new --agent ratatoskr:sindra --send … --bifrost-url http://10.100.10.50:8392 --end-user-id ratatoskr-tui`):** GATE LIT + scan fired ONCE at turn 1 with worldtree-dev's exact args (`scope_all={end_user:ratatoskr-tui} cursor=null limit=3 sort={updated_at,desc}`) → 3 records in ~5ms (no 500ms fail-open); injection confirmed in Sindra's CoT; **cross-session recognition WORKS** (she recalls him as a distinct person + patterns — blank-slate GONE). **BUT name-recall FAILS** — `"Name is Vuong."` is the OLDEST chunk (07:09) → excluded from top-3-by-recency AND scores 0.354 (sub-0.45) on the query path, so injects via NEITHER path; meanwhile a STALE contradictory `"user has not yet provided their name"` (07:58, 0.46) IS injected → Sindra concludes she lacks the name. **ROOT CAUSE = WORLDTREE ranking/hygiene** (not our wire): recency-top-3 under-weights early identity facts + the "no name" fact was never `mark_superseded` when "Name is Vuong" landed → both live+contradictory. Reported to worldtree-dev (thread `01KXK86PZ9…`) with 3 options: memory hygiene (supersede), person-prime salience-weighting, or higher N. -**CROSS-PROJECT FOLLOW-UPS (2026-07-16, spawned by the verify — 4 inbound drained):** (1) **bifrost-dev RULED** scan cursor snapshot-semantics NORMATIVE (offset NOT blessed) + shipped conformance coverage in **bifrost 1.1.3** (`bifrost.conformance.memory_store_conformance`, 4 probes; our offset bug ships as their negative-canary fixture). TODO (OPERATOR-SEQUENCED, not urgent — single-page person-prime already conformant): bump pin 1.1.1→1.1.3, adopt snapshot cursor (frozen id-list per opaque token + `ScanCursorExpired`→410, match `InMemoryMemoryStore`), run `run_all(include_optional=True)` → expect P1/P2/P3 RED→green; we're their canary. Replied ack'd (thread `01KXK7MDTY…`). (2) **worldtree-dev FILED the hygiene bug as WT #364** (promotion writes negative-knowledge facts w/o contradiction-reconciliation; our specimen = evidence base); #349 ranking/N decision (N=3 NOT consumer-tunable) is going to **Vuong** with their recommendation. (3) **brokkr R39 Phase-1 Arm-0 fusion bake-off** wants our frozen 7-fact store exported DECOMPOSED (data contract in `~/development/brokkr-smithy/research/R39-memory-salience-dreams-surfacing/phase-1/re-drive-design.md`; #364 reconciliation GATED on brokkr's re-drive-complete). **KEY STRUCTURAL FINDINGS:** our store persists ONLY `salience` (+embedding) — `similarity`/`combined` are WT-side query-time (`bifrost_memory_store.py:702`, combined=sim×salience); **person-prime is query-LESS → carries NO similarity** (= brokkr's Arm-1 finding); recovered per-fact similarity from the verify SEARCH log (name=0.354, stale=0.463, ×salience-1.0 = combined, matches brokkr's 0.354/0.46 grounding); **BOTH the true "Name is Vuong" AND the stale negative have salience=1.0** (salience can't discriminate — stale wins on recency + sim only); they share the (user,name) supersession slot (Phase-0 Q3). **⚠️ SPECIMEN CONTAMINATED BY MY VERIFY:** the turn-1 drive ADDED 3 chunks (16:28, re-extractions incl. a 3RD "name unknown" negative) → store is now **10, not 7**; the original 7 intact (≤08:42). **✅ EXPORT SENT (operator approved VERBATIM 2026-07-16):** "nothing there is really a concern." Classifier had blocked the `/mnt/smithy` shared-mount write (PII) → routed to Vuong → he chose full verbatim; delivered INLINE in brokkr thread `01KXMN7NR54…` (scoped to brokkr, not the open mount) as 7-row JSONL (fact_text + similarity[from verify SEARCH log]/salience[stored]/combined/recency/subject/relation/is_true_identity/is_stale_negative/source/provenance_note). **NEW #364 EVIDENCE — SUBJECT-PROVENANCE bug (operator-caught):** fact 75aa3110 ("prefers clear parameters Intensity/Mood/Willingness…") is NOT a user fact — Vuong never said it; it's **SINDRA'S OWN system-prompt scripted behavior mis-extracted into the USER memory partition** (subject flipped→assistant in export); e35b9dfe (closeness) maybe same (unsure). So the extractor leaks CHARACTER-self facts into user memory — a subject-correctness axis orthogonal to the recency/threshold/stale-negative story; flagged to brokkr (Arm-1?) + worldtree-dev (widen #364?). **SPECIMEN now 10 rows** (my verify added 3 re-extractions incl. a 3rd "name unknown"); original 7 intact + exported; flagged both peers. **HOLD HONORED — NO cleanup** (R39 needs it frozen-dirty; earlier "nudge" option OFF). All 3 peer loops replied (bifrost-dev/brokkr/worldtree-dev); harness ready for re-drive on brokkr/WT signal. WHY: Sindra didn't remember Vuong's name across sessions; worldtree-dev root-caused it — WT injects a recalled fact only if combined score (sim×salience) ≥ **0.45** (`auto_inject_combined_score_threshold`, `core/memory/context_promotion/config.py`), and recall is per-turn **query-gated**, so moderate-sim durable facts (name hit ~0.40) never inject. The designed **turn-0** fix is WT **#349 person-prime** — a query-LESS top-N-by-recency durable-fact injection — capability-gated on the store advertising **`updated_at` in `sort_fields`** at the Bifrost handshake, DARK for our provider until now. Fix (ZERO Worldtree change): implement the sorted `scan` verb + advertise the capability. **✅ CODE COMPLETE + COMMITTED (2026-07-15):** contract amended + validated; `scan` impl in `src/ratatoskr/provider/memory_store.py` (globally-ordered by indexed `updated_at`, live-only via `_is_live`, offset-cursor; `has_more` peek so an exact-fill page emits no empty trailing cursor = reference-parity; `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; expression index `idx_chunks_updated_at`); **TDD 7/7 GREEN** (scan_recency tracer + live_only + scope_isolation + unadvertised_sort + person_prime_record_shape + cursor pagination + **`parity_vs_reference` #195**); NUANCE resolved — the bifrost ref `InMemoryMemoryStore` **READS** `updated_at` from the record (never stamps it; its own comment confirms), identical to ours, so order-parity holds over the live set (ref does NOT lifecycle-filter, so parity is defined over live chunks only). **Full suite 638 green.** Committed as **two patches**: `8fc757a` scan (**v0.20.11**) + `a9c521a` Sindra holodesk first-message preset (**v0.20.12**, split out as a separate concern from the soong-lab redefine). **heid-bug-hunt RETURNED + TRIAGED** (panel Gróa/Hulda/Regin, thread `01KXK5XTYHV8TGEDRAZV8GRXWC`; base-ref 39050c3, code-only diff). **FIXED (v0.20.13, `25ccb5c`):** the non-dict `sort` crash — a truthy non-dict `sort` (`"updated_at"`/`["updated_at"]`/int) hit `(sort or {}).get(...)` → AttributeError instead of InvalidArguments; added isinstance guard (Gróa#1/Hulda#2, confirmed-from-code). **SURFACED (operator design decision, NOT auto-adopted):** offset-cursor cross-page duplicate/drop under concurrent mutation (all 3 arms) — real divergence from bifrost `InMemoryMemoryStore` (snapshots the ordered id-list per opaque cursor + `ScanCursorExpired` TTL); BUT person-prime uses a SINGLE page (no cursor) so the actual use isn't exercised → operator picks: adopt reference snapshot-cursors vs keyset vs accept + honestly scope INV-010 to a static store. **NOTED (reference-parity / accept-known-risk, no fork):** full-table materialize-then-Python-filter (matches ref's own O(store) iteration), unbound limit, falsy scope coercion `[]`→`{}` (parity w/ search + ref), lexical timestamp ordering (WT writes consistent ISO-8601 UTC); `_is_live` dead-marker completeness (Gróa#6 needs-repro) → confirm canonical marker set w/ worldtree-dev, don't guess. **REFUTED:** Regin's mixed-type ORDER BY "crash" (SQLite orders by storage class, doesn't raise) + its self-retracted `_is_live`/None item. **✅ DEPLOYED (2026-07-15, operator-authorized):** (5) **DONE** — restarted BOTH `:8392` combined provider (new pid, old 982284 gone) AND `:8765` web on **v0.20.13** via the env-preserving `scratchpad/relaunch_by_pid.py ` (captures /proc cmdline+environ+cwd → byte-identical config, only code changes; self-daemonizes). Verified: `:8392` up (uvicorn startup complete; `/`→404 = no root route = serving), deployed `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; `:8765` up HTTP 200 (log "ratatoskr-web v0.20.13"). **Added the missing REQUIRED `RATATOSKR_MEMORY_EMBEDDING_DIM=1024` to `env.sh`** (gitignored; was absent → bare `source env.sh` restart would crash). (6) **DONE** — pinged **worldtree-dev** (thread `01KXK86PZ9M8J53A4SFXREEMW7`) to verify turn-1 recall against personal :8081. Store has **7 REAL live Sindra durable facts** ready (agent_id `ratatoskr:sindra`, worldtree_scope `end_user`, scope `{end_user:ratatoskr-tui}`, all carry `updated_at`, accumulated across 3 sessions today) → top-N-by-recency has real data to inject. AWAITING their turn-1 verdict (monitor armed). (7) **F3 cursor RESOLVED — operator ACCEPTED offset-for-v1** (keep offset cursor; person-prime single-page is correct/conformant). INV-010 amended to mark the cursor **v1-provisional / KNOWN DEVIATION** (multi-page continuation diverges from bifrost's protocol snapshot-cursor contract — the `bifrost.memory` dispatch drops `sort` on continuation citing "snapshotted order is authoritative" + maps `ScanCursorExpired→410`; our offset cursor doesn't snapshot → dup/drop under concurrent write, and never expires). **Durable fix = a BIFROST completeness concern (operator's catch), ROUTED to bifrost-dev** (thread `01KXK7MDTY0B6VEDF3SVGSKY9R`): bifrost's conformance suite has ZERO scan/cursor coverage, so a non-snapshot cursor passes silently — asked bifrost-dev to (1) confirm snapshot-cursor+ScanCursorExpired is the NORMATIVE store contract, (2) add scan/cursor conformance coverage; we'll adopt the reference snapshot-cursor semantics IF they rule it normative (offered to be their canary as first non-reference scan implementer). AWAITING bifrost-dev ruling; monitor armed. +**CROSS-PROJECT FOLLOW-UPS (2026-07-16, spawned by the verify — 4 inbound drained):** (1) **bifrost-dev RULED** scan cursor snapshot-semantics NORMATIVE (offset NOT blessed) + shipped conformance coverage in **bifrost 1.1.3** (`bifrost.conformance.memory_store_conformance`, 4 probes; our offset bug ships as their negative-canary fixture). TODO (OPERATOR-SEQUENCED, not urgent — single-page person-prime already conformant): bump pin 1.1.1→1.1.3, adopt snapshot cursor (frozen id-list per opaque token + `ScanCursorExpired`→410, match `InMemoryMemoryStore`), run `run_all(include_optional=True)` → expect P1/P2/P3 RED→green; we're their canary. Replied ack'd (thread `01KXK7MDTY…`). (2) **worldtree-dev FILED the hygiene bug as WT #364** (promotion writes negative-knowledge facts w/o contradiction-reconciliation; our specimen = evidence base); #349 ranking/N decision is **ALREADY RULED (CORRECTION 2026-07-16 from worldtree-dev — NOT inbound to Vuong as I'd said)**: the operator ruled 2026-07-15 via brokkr's R39 thread — **NO top-N recency band-aid ships; person-prime goes straight to R39's researched identity-class pinning**, sequenced on #364. Next person-prime change = the R39-designed one. Both my flags folded into **#364** (now a MULTI-class contract: (i) stale-negative contradiction-reconciliation, (ii) subject-attribution gate at promotion for the character-self-into-user leak, same pipeline stage). Nothing lands before brokkr's re-drive-complete signal. (3) **brokkr R39 Phase-1 Arm-0 fusion bake-off** wants our frozen 7-fact store exported DECOMPOSED (data contract in `~/development/brokkr-smithy/research/R39-memory-salience-dreams-surfacing/phase-1/re-drive-design.md`; #364 reconciliation GATED on brokkr's re-drive-complete). **KEY STRUCTURAL FINDINGS:** our store persists ONLY `salience` (+embedding) — `similarity`/`combined` are WT-side query-time (`bifrost_memory_store.py:702`, combined=sim×salience); **person-prime is query-LESS → carries NO similarity** (= brokkr's Arm-1 finding); recovered per-fact similarity from the verify SEARCH log (name=0.354, stale=0.463, ×salience-1.0 = combined, matches brokkr's 0.354/0.46 grounding); **BOTH the true "Name is Vuong" AND the stale negative have salience=1.0** (salience can't discriminate — stale wins on recency + sim only); they share the (user,name) supersession slot (Phase-0 Q3). **⚠️ SPECIMEN CONTAMINATED BY MY VERIFY:** the turn-1 drive ADDED 3 chunks (16:28, re-extractions incl. a 3RD "name unknown" negative) → store is now **10, not 7**; the original 7 intact (≤08:42). **✅ EXPORT SENT (operator approved VERBATIM 2026-07-16):** "nothing there is really a concern." Classifier had blocked the `/mnt/smithy` shared-mount write (PII) → routed to Vuong → he chose full verbatim; delivered INLINE in brokkr thread `01KXMN7NR54…` (scoped to brokkr, not the open mount) as 7-row JSONL (fact_text + similarity[from verify SEARCH log]/salience[stored]/combined/recency/subject/relation/is_true_identity/is_stale_negative/source/provenance_note). **NEW #364 EVIDENCE — SUBJECT-PROVENANCE bug (operator-caught):** fact 75aa3110 ("prefers clear parameters Intensity/Mood/Willingness…") is NOT a user fact — Vuong never said it; it's **SINDRA'S OWN system-prompt scripted behavior mis-extracted into the USER memory partition** (subject flipped→assistant in export); e35b9dfe (closeness) maybe same (unsure). So the extractor leaks CHARACTER-self facts into user memory — a subject-correctness axis orthogonal to the recency/threshold/stale-negative story; flagged to brokkr (Arm-1?) + worldtree-dev (widen #364?). **SPECIMEN now 10 rows** (my verify added 3 re-extractions incl. a 3rd "name unknown"); original 7 intact + exported; flagged both peers. **HOLD HONORED — NO cleanup** (R39 needs it frozen-dirty; earlier "nudge" option OFF). All 3 peer loops replied (bifrost-dev/brokkr/worldtree-dev); harness ready for re-drive on brokkr/WT signal. WHY: Sindra didn't remember Vuong's name across sessions; worldtree-dev root-caused it — WT injects a recalled fact only if combined score (sim×salience) ≥ **0.45** (`auto_inject_combined_score_threshold`, `core/memory/context_promotion/config.py`), and recall is per-turn **query-gated**, so moderate-sim durable facts (name hit ~0.40) never inject. The designed **turn-0** fix is WT **#349 person-prime** — a query-LESS top-N-by-recency durable-fact injection — capability-gated on the store advertising **`updated_at` in `sort_fields`** at the Bifrost handshake, DARK for our provider until now. Fix (ZERO Worldtree change): implement the sorted `scan` verb + advertise the capability. **✅ CODE COMPLETE + COMMITTED (2026-07-15):** contract amended + validated; `scan` impl in `src/ratatoskr/provider/memory_store.py` (globally-ordered by indexed `updated_at`, live-only via `_is_live`, offset-cursor; `has_more` peek so an exact-fill page emits no empty trailing cursor = reference-parity; `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; expression index `idx_chunks_updated_at`); **TDD 7/7 GREEN** (scan_recency tracer + live_only + scope_isolation + unadvertised_sort + person_prime_record_shape + cursor pagination + **`parity_vs_reference` #195**); NUANCE resolved — the bifrost ref `InMemoryMemoryStore` **READS** `updated_at` from the record (never stamps it; its own comment confirms), identical to ours, so order-parity holds over the live set (ref does NOT lifecycle-filter, so parity is defined over live chunks only). **Full suite 638 green.** Committed as **two patches**: `8fc757a` scan (**v0.20.11**) + `a9c521a` Sindra holodesk first-message preset (**v0.20.12**, split out as a separate concern from the soong-lab redefine). **heid-bug-hunt RETURNED + TRIAGED** (panel Gróa/Hulda/Regin, thread `01KXK5XTYHV8TGEDRAZV8GRXWC`; base-ref 39050c3, code-only diff). **FIXED (v0.20.13, `25ccb5c`):** the non-dict `sort` crash — a truthy non-dict `sort` (`"updated_at"`/`["updated_at"]`/int) hit `(sort or {}).get(...)` → AttributeError instead of InvalidArguments; added isinstance guard (Gróa#1/Hulda#2, confirmed-from-code). **SURFACED (operator design decision, NOT auto-adopted):** offset-cursor cross-page duplicate/drop under concurrent mutation (all 3 arms) — real divergence from bifrost `InMemoryMemoryStore` (snapshots the ordered id-list per opaque cursor + `ScanCursorExpired` TTL); BUT person-prime uses a SINGLE page (no cursor) so the actual use isn't exercised → operator picks: adopt reference snapshot-cursors vs keyset vs accept + honestly scope INV-010 to a static store. **NOTED (reference-parity / accept-known-risk, no fork):** full-table materialize-then-Python-filter (matches ref's own O(store) iteration), unbound limit, falsy scope coercion `[]`→`{}` (parity w/ search + ref), lexical timestamp ordering (WT writes consistent ISO-8601 UTC); `_is_live` dead-marker completeness (Gróa#6 needs-repro) → confirm canonical marker set w/ worldtree-dev, don't guess. **REFUTED:** Regin's mixed-type ORDER BY "crash" (SQLite orders by storage class, doesn't raise) + its self-retracted `_is_live`/None item. **✅ DEPLOYED (2026-07-15, operator-authorized):** (5) **DONE** — restarted BOTH `:8392` combined provider (new pid, old 982284 gone) AND `:8765` web on **v0.20.13** via the env-preserving `scratchpad/relaunch_by_pid.py ` (captures /proc cmdline+environ+cwd → byte-identical config, only code changes; self-daemonizes). Verified: `:8392` up (uvicorn startup complete; `/`→404 = no root route = serving), deployed `describe_store` advertises `sortable_chunk_fields=[{updated_at}]`; `:8765` up HTTP 200 (log "ratatoskr-web v0.20.13"). **Added the missing REQUIRED `RATATOSKR_MEMORY_EMBEDDING_DIM=1024` to `env.sh`** (gitignored; was absent → bare `source env.sh` restart would crash). (6) **DONE** — pinged **worldtree-dev** (thread `01KXK86PZ9M8J53A4SFXREEMW7`) to verify turn-1 recall against personal :8081. Store has **7 REAL live Sindra durable facts** ready (agent_id `ratatoskr:sindra`, worldtree_scope `end_user`, scope `{end_user:ratatoskr-tui}`, all carry `updated_at`, accumulated across 3 sessions today) → top-N-by-recency has real data to inject. AWAITING their turn-1 verdict (monitor armed). (7) **F3 cursor RESOLVED — operator ACCEPTED offset-for-v1** (keep offset cursor; person-prime single-page is correct/conformant). INV-010 amended to mark the cursor **v1-provisional / KNOWN DEVIATION** (multi-page continuation diverges from bifrost's protocol snapshot-cursor contract — the `bifrost.memory` dispatch drops `sort` on continuation citing "snapshotted order is authoritative" + maps `ScanCursorExpired→410`; our offset cursor doesn't snapshot → dup/drop under concurrent write, and never expires). **Durable fix = a BIFROST completeness concern (operator's catch), ROUTED to bifrost-dev** (thread `01KXK7MDTY0B6VEDF3SVGSKY9R`): bifrost's conformance suite has ZERO scan/cursor coverage, so a non-snapshot cursor passes silently — asked bifrost-dev to (1) confirm snapshot-cursor+ScanCursorExpired is the NORMATIVE store contract, (2) add scan/cursor conformance coverage; we'll adopt the reference snapshot-cursor semantics IF they rule it normative (offered to be their canary as first non-reference scan implementer). AWAITING bifrost-dev ruling; monitor armed. **✅ soong-lab interop ESTABLISHED (this session) — soong-lab is now our Tier-3 agent-authoring studio.** ratatoskr consumes soong-lab bundles → WT `agents.define`, and ships agents back as bundles. Sindra round-trip PROVEN (soong imported her `resume` half through real `import_bundle`). soong-lab export/importer contracts PINNED via canonical-sync (`canonical_source=soong-lab` @ f434016, commit `39050c3`). Role-grounding (worldtree-dev): the 4 soong-lab ROLE_CHOICES = WT model-role slugs 1:1 BY NAME — `character`/`thoughtful-character` need the `character` grant (HELD via Sindra → define-able now); `assistant`/`thoughtful-assistant` need `foundational` (confirm/mint routed to infra-ops, in flight; only for assistant-type bundles). `character-rp` ≡ `thoughtful-character` (same Deckard seat + grant). **Sindra REDEFINED from the bundle** (`/tmp/sindra.json`, operator "update to match"): persona immutable → DELETE+redefine (payload validated via a throwaway `sindra-probe2`→201 BEFORE deleting the real one), `memory:{}` preserved, motivational string→WT-object mapped (Tried/abandoned), role→`thoughtful-character`, first_message preset updated. Verified live (thoughtful-character, 25574-char prompt).