Files
ratatoskr/persistent-memory.d/2026-07-18-368-silo-test-passed.md
T
vh 13efdf43a5 memory: compress #368 forensic history into the detail file
Move the consolidated #368 in-flight narrative (diagnosis -> two-channel
investigation -> both scrubs -> marker repro -> enforcement read) out of
persistent-memory.md Current state into the silo-test detail file, leaving
a compact pointer. Index 439 -> 339 lines.
2026-07-17 21:44:10 -07:00

177 lines
17 KiB
Markdown

`[2026-07-18]` **#368 (user, character) memory silo test — DONE, PASSED live against Worldtree b127.**
The immediate post-clear action (the reason for the 2026-07-18 clear). Closed with a decisive PASS on both the
write and read halves of the silo. See sibling arcs [[2026-07-16-wt364-r39-name-recall]] (name recall) and the
#368 enforcement-read history captured in the Recent-decisions log.
## The bug (recap)
WT wrote Tier-3 memory scoped `{end_user}`-only; `agent_id` rode as record metadata, NOT isolation scope. So every
character under one end_user shared the memory pool (Lofn recited Sindra's facts; the operator caught it). NOT a
ratatoskr defect — our lattice `{end_user,group,tenant,agent_self}` + `_matches_scope` already supported the
agent axis; WT's write AND retrieval both omitted it. Operator ruled Option 1: silo per (user,character) +
backfill. ratatoskr was the ENFORCEMENT half (consumer-side read conformance), never a code-change target.
## The fix (what b127 ships)
Every Tier-3 write is stamped with the conjunctive `{end_user, agent_self}` scope. On each session bind WT probes
our store for the migration sentinel `worldtree:migration:agent-scope-v1`; if absent it runs a store-wide backfill
(single-page scan per our finding F2; sentinel carries a zero-vector at embedding_dim per F3) and only FLIPS reads
to conjunctive once the sentinel is present. Un-bound end_user reads on a migrated store drop fail-closed, never
sent as an empty (match-all) filter (F1). All three enforcement finds folded into WT contract rev 1.3.
## Test design (why it's valid)
Held **end_user CONSTANT** (`silo-368`) across BOTH agents — that isolates the `agent_self` axis under test.
Different end_users would let the pre-existing end_user axis explain any separation and prove nothing about #368.
Drove via the **CLI combined bind** (`--bifrost-url http://10.100.10.50:8392`), not the web path — same canonical
combined bind onto the same :8392 provider, but scriptable/deterministic (captures session ids + turn output).
Sindra/Betty are consumer-defined agents with NO server-side chroma → our Tier-3 store is their SOLE memory source,
so cold-recall is a clean test of our store's conjunctive retrieval (no channel-2 confound; that's foundational-only).
## Results — silo holds end to end
- **Migration verified:** first bind wrote the sentinel (`worldtree:migration:agent-scope-v1`, scope
`{"tenant":"worldtree:migration"}`, content "agent-scope-v1 backfill complete"); backfill a no-op on the
born-empty store. Reads flipped conjunctive.
- **WRITE fix (structural core):** Sindra's 3 promoted chunks each carry `{"end_user":"silo-368",
"agent_self":"ratatoskr:sindra"}` (verbatim texts: "Strongly prefers coffee over tea;", "Morning ritual is always
a strong black coffee.", plus a meta-reinforcement chunk). Real WT→provider promotion, not a synthetic write.
- **READ isolation (behavioral, decisive):** cold-recall in FRESH sessions (no in-session context), same end_user
`silo-368`. Sindra pulled her own chunks via person-prime and answered "Coffee. Strong black, no sugar." Betty,
sharing the exact same end_user, got ZERO hits → "I don't know. I don't have memory across chats." Betty did NOT
bleed Sindra's coffee chunks — her conjunctive `{silo-368, betty}` filter matched none of Sindra's
`{silo-368, sindra}`. That's the precise pre→post flip of the Lofn-recites-Sindra bug.
- Bind identical for both (admin `GET /admin/sessions/{id}/bifrost`: connected, caps `[affect, memory]`).
## The Betty asymmetry (NOT a #368 issue — banked as a data point)
Betty never got her OWN tea fact promoted (18+ min idle, while Sindra promoted in ~8.5 min). Cause: WT's memory
extractor leans on the ASSISTANT's prose, and Betty's terse "brief replies" persona never restated the fact, so
nothing was extracted. Sindra's verbose persona restated "your favorite drink is strong black coffee" out loud →
that got promoted. So the persona shaped the outcome INDIRECTLY (reply verbosity), but the gate is WT-side
extraction, not persona disposition. Known #296/#369 family (extractor favors assistant prose over the raw user
statement). Did NOT weaken the verdict — the decisive cross-bleed direction passed cleanly, and Sindra demonstrated
the positive-recall direction. Operator declined a worldtree-dev FYI on it (tangential to #368).
## Name clarification (operator asked)
The operator's name (Vuong) was NOT part of this test — zero mentions in any sent message or either agent's recall,
no name chunk in the store. This test was beverage-only under a FRESH, scrubbed end_user (`silo-368`). The
name-recall behavior from the earlier contamination saga was under a DIFFERENT partition (`ratatoskr-tui`), scrubbed.
## Cleanup + tooling notes
- Throwaway `ratatoskr:betty` DELETED: `python -m ratatoskr.tier3 delete ratatoskr:betty` → WT 404 + local-registry
removal; Sindra intact (`GET /agents/ratatoskr:sindra` = 200). The auto-mode classifier denied the first attempt
(irreversible remote deletion); operator explicitly authorized.
- **tier3 CLI define/patch adapted to the b127 `role` schema** (was `model`), commit `860e0d5`, v0.21.1. The
`delete` verb works (used above).
- **Store SCRUBBED clean afterward (2026-07-18, operator-directed):** `reset-sindra-stores.sh` (rolling backup
`db-reset-backup/`) → memory 0 / affect 0, combined provider restarted empty (pid 3856058). The next bind
re-runs the no-op migration + sentinel write. Silo-test evidence lives in this file, not the store.
## Still QUEUED
The **backfill live-verify** (synthetic legacy corpus): when exercising the backfill machinery matters, drive a
synthetic pre-migration (`{end_user}`-only) corpus into our store, run WT's backfill against it, confirm 100%
stamped + sentinel writes + reads flip clean — a born-fresh store skips the backfill entirely (this test did).
## Full forensic history (moved from persistent-memory.md Current state, 2026-07-18)
The consolidated in-flight narrative of the #368 arc — diagnosis, the two-channel investigation, both
scrubs, the marker repro, and the enforcement read — as it stood in Current state before compression:
**⚠️ Cross-agent Tier-3 memory BLEED — WORLDTREE defect (found via debug surface 2026-07-17; reported to
worldtree-dev, reply pending).** Loaded Lofn (Tier-1) in the memory explorer under end_user `ratatoskr-tui`
→ saw **Sindra's** chunks, and a live Lofn turn **recalled Vuong's name** (a Sindra fact). ROOT: WT writes
Tier-3 memory scoped `{"end_user": "ratatoskr-tui"}`, `worldtree_scope: "end_user"`, **NO `agent_self`** —
`agent_id: "ratatoskr:sindra"` rides as record metadata only, not isolation scope. So every agent under one
end_user shares the pool. **NOT a ratatoskr defect:** our v0.6 lattice {end_user,group,tenant,agent_self} +
`_matches_scope` support agent_self; `upsert_many` stores scope verbatim (no strip/inject); read-path proof —
Lofn getting Sindra's chunks means WT's retrieval `scope_all` ALSO omits agent_self (else `_scope_subset`
excludes agent_self-less records → Lofn would get nothing). FIX is WT's: stamp `agent_self` on Tier-3 writes
+ include it in turn-retrieval scope_all; ratatoskr enforces the moment it arrives, zero change. Likely the
known per-(user,character) isolation gap (#172). Thread `01KXRY8V1C30R3HGFCTKHPA4PR`. **FORK RELEVANCE:** the
fork's memory isolation inherits this same WT axis — don't design around it, it's Worldtree's to close.
Explorer nuance: its agent filter is LENIENT (excludes a chunk only if it CARRIES a conflicting agent_self),
so agent_self-less chunks show under any agent — a faithful display of user-shared data, not a separate bug.
**RESOLVED-DIRECTION 2026-07-17 (WT #368 "Cross-agent Tier-3 memory bleed — end_user scope is agent-agnostic").**
Operator ruled **Option 1: silo per (user, character) + backfill**. NO amnesia cliff — backfill stamps `agent_self`
from the `record.agent_id` every chunk already carries. Contract arc starting WT-side (blast-radius→contract→heid
review→TDD→code-review+bug-hunt panels); **ratatoskr gets the contract for a consumer-side read BEFORE preflight —
we are the enforcement half.** Confirmed empirically (our `_scope_subset`/`_matches_scope`): a multi-axis scope_any
element is matched CONJUNCTIVELY → **fix rides `scope_any=[{end_user:X, agent_self:Y}]` ALONE, no `scope_all`
threading** (INV-005 / #297 "each AND-matched as a whole"). Cutover ordering: **backfill THEN flip retrieval**
(a conjunctive element excludes 100% of un-backfilled chunks). **READ-PATH WRINKLE (flagged to WT):** labeled Lofn
capture (agent=lofn, session 15fde90d, turn 2253) recited *"Vuong is the name on file"* — behavioral bleed exhibit
— BUT driven via CLI `--bifrost-url` (plane=direct) the turn hit our provider with a HANDSHAKE ONLY, zero
memory-calls, yet still knew the name → the name came from **Worldtree-internal context, not our Tier-3 store** on a
direct bind. CONFIRMED via a SECOND fresh combined-plane turn (session c153d8ec): it TOO recited the name with
handshake-but-ZERO-memory-calls. So the turn-context bleed is Worldtree-INTERNAL on BOTH bind shapes — it never
traverses our Tier-3 retrieval. **TWO channels:** (1) Tier-3 semantic recall to us (the morning memory.search lines;
closed by the conjunctive scope_any fix) and (2) WT-internal assembly (the ACTIVE fresh-session turn-context channel;
needs the same (user,character) conjunction WT-side or the silo leaks exactly where the operator saw it). Both fresh
sessions verified via message logs (name only in Lofn's response, no prior history → not same-session continuity).
WT is tracing channel 2 code-side (Tier-1 per-agent store audited immune). CLI GAP: `--bifrost-plane` offers only
memory/affect (→:8391/:8390), not `combined` (→:8392). Exhibit sessions: 15fde90d (direct), c153d8ec (combined).
**CHANNEL 2 = REAL BUG (discriminator run via GET /search, our key carries search.read, 2026-07-17):** name
introduced ONLY to Sindra (8 user msgs, e.g. session 81952f2e "my name is Vuong"), ZERO to Lofn (0 user, 4 assistant
incl. PRE-EXISTING session cc8d7ee6 @20:47Z which PREDATES my captures — the operator's own observation). So Lofn's
person-prime (#349, turn-1 query-less "what I know about this person") surfaces a Sindra-only fact un-introduced →
cross-agent leak in WT's server-side per-agent store, one layer below the Bifrost plane. WT's channel-2 source: on
foundational agents the Bifrost store is NEVER attached (kind-guard) — context comes from WT's own per-agent store via
#349 person-prime + #221 auto-recall, whose query filter carries NO agent axis (end_user only; isolation rests solely
on physical per-collection boundary). That missing agent-axis looks LOAD-BEARING (channel-2's real fix), not just
defense-in-depth. WT tracing write/query path; **rev 1.2 contract (ch1 + ch2 + agent-axis stamp) comes to ratatoskr
for the enforcement read** (INV-368-9 blocks WT preflight until then).
**SCRUB + REPRO (2026-07-18):** operator ruled a FULL memory scrub both sides (contamination noise, nothing
worth keeping). Server-side (personal/b125): infra-ops wiping all agent chroma collections + shared promotion
state. Ours: **Tier-3 memory.db scrubbed to 0 chunks, :8392 restarted born-empty, no backup** (operator direct
go; new pid 3548269; affect plane untouched). Personal is ALREADY b125 (redeployed 20:35Z; /version 404 =
b125 dropped the route, misled us). Current-code status: the fresh Lofn 'Vuong' chunks are the FEEDBACK LOOP
(Lofn's own recitations re-extracted), NOT proven fresh mis-write; the original cross-agent mis-write ("I'm
Saga, and you are Vuong" → lofn.chroma) is Jun-3 code, UNPROVEN on b125. **QUEUED (after BOTH scrubs confirmed):
the marker repro** — drive ONE fresh Sindra session (combined :8392 bound), user introduces an absurd greppable
marker ("my favorite mineral is rhodochrosite", NOT the name), idle a few min for extraction, report timestamps
→ infra-ops greps lofn.chroma + our store. Marker in lofn.chroma = mis-write ALIVE on b125 (code fix); marker
only in our store = channel 2 is just the feedback loop + agent-axis hardening. Post-scrub our #368 backfill is
a NO-OP (born-conjunctive once the fix ships); contract keeps migration machinery for other consumers. DO NOT
re-ask Lofn the name meanwhile (re-seeds the loop). Awaiting WT infra-ops server-side-scrub confirm to run it.
**RAN 2026-07-18 (both scrubs confirmed clean):** marker repro executed — fresh Sindra session 822c3af7
(combined :8392), user introduced "my favorite mineral is rhodochrosite", 5 more turns drove the idle-scan
promotion at ~6 turns. RESULT: marker promoted to OUR store scoped **{end_user}-only** (chunk 647aeac6;
record.agent_id "ratatoskr:sindra" metadata; worldtree_scope "end_user"; verbatim "Favorite mineral is
rhodochrosite") = **channel-1 baseline exhibit** (pre-fix shape; agent_self absent, to be backfilled from the
agent_id metadata). **KEY WT CORRECTION:** consumer-defined agents (ratatoskr:sindra) have NO server-side
chroma — they write ONLY to our store via Bifrost; only FOUNDATIONAL agents (lofn/forseti/mimir) get
server-side collections. Confirmed on the wire: Sindra IS Bifrost-attached (queried us scan+search) UNLIKE
foundational Lofn (handshake-only, no memory-call). DECISIVE channel-2 grep (marker must be in ZERO
server-side stores; ANY server-side hit = cross-plane bug, no benign reading) still PENDING infra-ops.
**ETA (WT): rev 1.2 amended contract reaches us TODAY 2026-07-18 for the consumer-side enforcement read;
ship-ready LIKELY 2026-07-19; our integration risk LOW** (born-fresh store won't exercise the backfill
machinery, which is the bulk of the impl). ratatoskr is the enforcement half; INV-368-9 blocks WT preflight.
**CHANNEL 2 CLOSED CLEAN + ENFORCEMENT READ DONE (2026-07-18):** marker routed ONLY to our store, zero in every
foundational store → current-b125 write path VERIFIED correct; the contamination was old-code (saga-v1, pre-#211)
seeding + a b125 recited-content re-promotion feedback loop (→ WT #369). Contract FINAL: `Worldtree/docs/contracts/
issues/368.contract.md` rev 1.2 (cda308c). **Consumer-side enforcement read complete — read-path CONFORMANT
(conjunctive-per-element re-verified); 3 findings sent to WT (thread 01KXSBY3A8):** F1[HIGH] INV-368-10 can fail-OPEN
at our store — our empty-scope = MATCH-ALL, so the guard's "drop end_user element" must NOT reach us as an empty
filter (else total bleed); fix = don't-call/never-match. F2[HIGH] backfill scan NOT snapshot-stable — our scan is
offset-over-updated_at (chunk_id NOT sortable), and re-upserts advance updated_at, so multi-page paging under
concurrent mutation can SKIP chunks → un-stamped → post-flip amnesia (INV-368-3 failure); fix = whole-corpus
single-page scan (no consumer change) OR add chunk_id sort (consumer change). F3[MED] sentinel needs a valid 1024-dim
embedding or our vec0 upsert raises → backfill never completes. All 3 are WT-side backfill/guard adjustments, NOT
consumer reworks — the "zero consumer changes" claim holds for reads, needs the qualifier for backfill/guard.
**FINAL 2026-07-18: all 3 findings FOLDED into contract rev 1.3 (3e3f629).** F1 → INV-368-10 now mandates
drop-to-empty = SKIP the wire call (test `drop_to_empty_skips_wire` asserts our store gets NO query). F2 →
option (a) pinned: backfill scans whole corpus in ONE page (test `single_page_snapshot`; immutable-key paging
= recorded scale follow-up that WOULD be a consumer change). F3 → sentinel writes a zero-vector at embedding_dim
(test `sentinel_carries_vector`). Our read-path conformance + conjunctive-per-element check cited in the revision
record. WT implementing against rev 1.3 (TDD after their context-reset; cross-frontier panels follow); ship ~7/19.
**QUEUED — backfill live-verify:** when WT's backfill lands, drive a SYNTHETIC legacy corpus into our store, run
their backfill against it, confirm 100% stamped (no un-attributable skips) + sentinel writes + reads flip clean,
all BEFORE any real flip. **#369** = the recited-content re-promotion feedback-loop policy class (WT owns).
**#368 arc DONE from ratatoskr's side** — diagnosed → reported → operator ruled → exhibits + both scrubs →
channel-2 closed clean → enforcement read (3 findings, all folded).
**✅ LIVE SILO TEST DONE — PASSED (2026-07-18, WT b127).** See the ✅ #368 summary above + full record in
`persistent-memory.d/2026-07-18-368-silo-test-passed.md`. Betty (throwaway) DELETED (WT 404 + local registry);
Sindra intact. **TIER3 CLI: define/patch adapted to the b127 `role` schema** (commit `860e0d5`, v0.21.1) — the
`delete` verb works (used it to remove betty: WT hard-delete + `remove_local_agent`). Drive was via the CLI
combined bind (`--bifrost-url http://10.100.10.50:8392`), not web. **Store SCRUBBED clean afterward (2026-07-18,
`reset-sindra-stores.sh`, rolling backup `db-reset-backup/`): memory 0 / affect 0, provider restarted empty (pid
3856058); next bind re-runs the no-op migration + sentinel write.** Silo-test evidence preserved in the detail file.