13efdf43a5
Move the consolidated #368 in-flight narrative (diagnosis -> two-channel investigation -> both scrubs -> marker repro -> enforcement read) out of persistent-memory.md Current state into the silo-test detail file, leaving a compact pointer. Index 439 -> 339 lines.
177 lines
17 KiB
Markdown
177 lines
17 KiB
Markdown
`[2026-07-18]` **#368 (user, character) memory silo test — DONE, PASSED live against Worldtree b127.**
|
|
|
|
The immediate post-clear action (the reason for the 2026-07-18 clear). Closed with a decisive PASS on both the
|
|
write and read halves of the silo. See sibling arcs [[2026-07-16-wt364-r39-name-recall]] (name recall) and the
|
|
#368 enforcement-read history captured in the Recent-decisions log.
|
|
|
|
## The bug (recap)
|
|
WT wrote Tier-3 memory scoped `{end_user}`-only; `agent_id` rode as record metadata, NOT isolation scope. So every
|
|
character under one end_user shared the memory pool (Lofn recited Sindra's facts; the operator caught it). NOT a
|
|
ratatoskr defect — our lattice `{end_user,group,tenant,agent_self}` + `_matches_scope` already supported the
|
|
agent axis; WT's write AND retrieval both omitted it. Operator ruled Option 1: silo per (user,character) +
|
|
backfill. ratatoskr was the ENFORCEMENT half (consumer-side read conformance), never a code-change target.
|
|
|
|
## The fix (what b127 ships)
|
|
Every Tier-3 write is stamped with the conjunctive `{end_user, agent_self}` scope. On each session bind WT probes
|
|
our store for the migration sentinel `worldtree:migration:agent-scope-v1`; if absent it runs a store-wide backfill
|
|
(single-page scan per our finding F2; sentinel carries a zero-vector at embedding_dim per F3) and only FLIPS reads
|
|
to conjunctive once the sentinel is present. Un-bound end_user reads on a migrated store drop fail-closed, never
|
|
sent as an empty (match-all) filter (F1). All three enforcement finds folded into WT contract rev 1.3.
|
|
|
|
## Test design (why it's valid)
|
|
Held **end_user CONSTANT** (`silo-368`) across BOTH agents — that isolates the `agent_self` axis under test.
|
|
Different end_users would let the pre-existing end_user axis explain any separation and prove nothing about #368.
|
|
Drove via the **CLI combined bind** (`--bifrost-url http://10.100.10.50:8392`), not the web path — same canonical
|
|
combined bind onto the same :8392 provider, but scriptable/deterministic (captures session ids + turn output).
|
|
Sindra/Betty are consumer-defined agents with NO server-side chroma → our Tier-3 store is their SOLE memory source,
|
|
so cold-recall is a clean test of our store's conjunctive retrieval (no channel-2 confound; that's foundational-only).
|
|
|
|
## Results — silo holds end to end
|
|
- **Migration verified:** first bind wrote the sentinel (`worldtree:migration:agent-scope-v1`, scope
|
|
`{"tenant":"worldtree:migration"}`, content "agent-scope-v1 backfill complete"); backfill a no-op on the
|
|
born-empty store. Reads flipped conjunctive.
|
|
- **WRITE fix (structural core):** Sindra's 3 promoted chunks each carry `{"end_user":"silo-368",
|
|
"agent_self":"ratatoskr:sindra"}` (verbatim texts: "Strongly prefers coffee over tea;", "Morning ritual is always
|
|
a strong black coffee.", plus a meta-reinforcement chunk). Real WT→provider promotion, not a synthetic write.
|
|
- **READ isolation (behavioral, decisive):** cold-recall in FRESH sessions (no in-session context), same end_user
|
|
`silo-368`. Sindra pulled her own chunks via person-prime and answered "Coffee. Strong black, no sugar." Betty,
|
|
sharing the exact same end_user, got ZERO hits → "I don't know. I don't have memory across chats." Betty did NOT
|
|
bleed Sindra's coffee chunks — her conjunctive `{silo-368, betty}` filter matched none of Sindra's
|
|
`{silo-368, sindra}`. That's the precise pre→post flip of the Lofn-recites-Sindra bug.
|
|
- Bind identical for both (admin `GET /admin/sessions/{id}/bifrost`: connected, caps `[affect, memory]`).
|
|
|
|
## The Betty asymmetry (NOT a #368 issue — banked as a data point)
|
|
Betty never got her OWN tea fact promoted (18+ min idle, while Sindra promoted in ~8.5 min). Cause: WT's memory
|
|
extractor leans on the ASSISTANT's prose, and Betty's terse "brief replies" persona never restated the fact, so
|
|
nothing was extracted. Sindra's verbose persona restated "your favorite drink is strong black coffee" out loud →
|
|
that got promoted. So the persona shaped the outcome INDIRECTLY (reply verbosity), but the gate is WT-side
|
|
extraction, not persona disposition. Known #296/#369 family (extractor favors assistant prose over the raw user
|
|
statement). Did NOT weaken the verdict — the decisive cross-bleed direction passed cleanly, and Sindra demonstrated
|
|
the positive-recall direction. Operator declined a worldtree-dev FYI on it (tangential to #368).
|
|
|
|
## Name clarification (operator asked)
|
|
The operator's name (Vuong) was NOT part of this test — zero mentions in any sent message or either agent's recall,
|
|
no name chunk in the store. This test was beverage-only under a FRESH, scrubbed end_user (`silo-368`). The
|
|
name-recall behavior from the earlier contamination saga was under a DIFFERENT partition (`ratatoskr-tui`), scrubbed.
|
|
|
|
## Cleanup + tooling notes
|
|
- Throwaway `ratatoskr:betty` DELETED: `python -m ratatoskr.tier3 delete ratatoskr:betty` → WT 404 + local-registry
|
|
removal; Sindra intact (`GET /agents/ratatoskr:sindra` = 200). The auto-mode classifier denied the first attempt
|
|
(irreversible remote deletion); operator explicitly authorized.
|
|
- **tier3 CLI define/patch adapted to the b127 `role` schema** (was `model`), commit `860e0d5`, v0.21.1. The
|
|
`delete` verb works (used above).
|
|
- **Store SCRUBBED clean afterward (2026-07-18, operator-directed):** `reset-sindra-stores.sh` (rolling backup
|
|
`db-reset-backup/`) → memory 0 / affect 0, combined provider restarted empty (pid 3856058). The next bind
|
|
re-runs the no-op migration + sentinel write. Silo-test evidence lives in this file, not the store.
|
|
|
|
## Still QUEUED
|
|
The **backfill live-verify** (synthetic legacy corpus): when exercising the backfill machinery matters, drive a
|
|
synthetic pre-migration (`{end_user}`-only) corpus into our store, run WT's backfill against it, confirm 100%
|
|
stamped + sentinel writes + reads flip clean — a born-fresh store skips the backfill entirely (this test did).
|
|
|
|
## Full forensic history (moved from persistent-memory.md Current state, 2026-07-18)
|
|
|
|
The consolidated in-flight narrative of the #368 arc — diagnosis, the two-channel investigation, both
|
|
scrubs, the marker repro, and the enforcement read — as it stood in Current state before compression:
|
|
|
|
**⚠️ Cross-agent Tier-3 memory BLEED — WORLDTREE defect (found via debug surface 2026-07-17; reported to
|
|
worldtree-dev, reply pending).** Loaded Lofn (Tier-1) in the memory explorer under end_user `ratatoskr-tui`
|
|
→ saw **Sindra's** chunks, and a live Lofn turn **recalled Vuong's name** (a Sindra fact). ROOT: WT writes
|
|
Tier-3 memory scoped `{"end_user": "ratatoskr-tui"}`, `worldtree_scope: "end_user"`, **NO `agent_self`** —
|
|
`agent_id: "ratatoskr:sindra"` rides as record metadata only, not isolation scope. So every agent under one
|
|
end_user shares the pool. **NOT a ratatoskr defect:** our v0.6 lattice {end_user,group,tenant,agent_self} +
|
|
`_matches_scope` support agent_self; `upsert_many` stores scope verbatim (no strip/inject); read-path proof —
|
|
Lofn getting Sindra's chunks means WT's retrieval `scope_all` ALSO omits agent_self (else `_scope_subset`
|
|
excludes agent_self-less records → Lofn would get nothing). FIX is WT's: stamp `agent_self` on Tier-3 writes
|
|
+ include it in turn-retrieval scope_all; ratatoskr enforces the moment it arrives, zero change. Likely the
|
|
known per-(user,character) isolation gap (#172). Thread `01KXRY8V1C30R3HGFCTKHPA4PR`. **FORK RELEVANCE:** the
|
|
fork's memory isolation inherits this same WT axis — don't design around it, it's Worldtree's to close.
|
|
Explorer nuance: its agent filter is LENIENT (excludes a chunk only if it CARRIES a conflicting agent_self),
|
|
so agent_self-less chunks show under any agent — a faithful display of user-shared data, not a separate bug.
|
|
|
|
**RESOLVED-DIRECTION 2026-07-17 (WT #368 "Cross-agent Tier-3 memory bleed — end_user scope is agent-agnostic").**
|
|
Operator ruled **Option 1: silo per (user, character) + backfill**. NO amnesia cliff — backfill stamps `agent_self`
|
|
from the `record.agent_id` every chunk already carries. Contract arc starting WT-side (blast-radius→contract→heid
|
|
review→TDD→code-review+bug-hunt panels); **ratatoskr gets the contract for a consumer-side read BEFORE preflight —
|
|
we are the enforcement half.** Confirmed empirically (our `_scope_subset`/`_matches_scope`): a multi-axis scope_any
|
|
element is matched CONJUNCTIVELY → **fix rides `scope_any=[{end_user:X, agent_self:Y}]` ALONE, no `scope_all`
|
|
threading** (INV-005 / #297 "each AND-matched as a whole"). Cutover ordering: **backfill THEN flip retrieval**
|
|
(a conjunctive element excludes 100% of un-backfilled chunks). **READ-PATH WRINKLE (flagged to WT):** labeled Lofn
|
|
capture (agent=lofn, session 15fde90d, turn 2253) recited *"Vuong is the name on file"* — behavioral bleed exhibit
|
|
— BUT driven via CLI `--bifrost-url` (plane=direct) the turn hit our provider with a HANDSHAKE ONLY, zero
|
|
memory-calls, yet still knew the name → the name came from **Worldtree-internal context, not our Tier-3 store** on a
|
|
direct bind. CONFIRMED via a SECOND fresh combined-plane turn (session c153d8ec): it TOO recited the name with
|
|
handshake-but-ZERO-memory-calls. So the turn-context bleed is Worldtree-INTERNAL on BOTH bind shapes — it never
|
|
traverses our Tier-3 retrieval. **TWO channels:** (1) Tier-3 semantic recall to us (the morning memory.search lines;
|
|
closed by the conjunctive scope_any fix) and (2) WT-internal assembly (the ACTIVE fresh-session turn-context channel;
|
|
needs the same (user,character) conjunction WT-side or the silo leaks exactly where the operator saw it). Both fresh
|
|
sessions verified via message logs (name only in Lofn's response, no prior history → not same-session continuity).
|
|
WT is tracing channel 2 code-side (Tier-1 per-agent store audited immune). CLI GAP: `--bifrost-plane` offers only
|
|
memory/affect (→:8391/:8390), not `combined` (→:8392). Exhibit sessions: 15fde90d (direct), c153d8ec (combined).
|
|
**CHANNEL 2 = REAL BUG (discriminator run via GET /search, our key carries search.read, 2026-07-17):** name
|
|
introduced ONLY to Sindra (8 user msgs, e.g. session 81952f2e "my name is Vuong"), ZERO to Lofn (0 user, 4 assistant
|
|
incl. PRE-EXISTING session cc8d7ee6 @20:47Z which PREDATES my captures — the operator's own observation). So Lofn's
|
|
person-prime (#349, turn-1 query-less "what I know about this person") surfaces a Sindra-only fact un-introduced →
|
|
cross-agent leak in WT's server-side per-agent store, one layer below the Bifrost plane. WT's channel-2 source: on
|
|
foundational agents the Bifrost store is NEVER attached (kind-guard) — context comes from WT's own per-agent store via
|
|
#349 person-prime + #221 auto-recall, whose query filter carries NO agent axis (end_user only; isolation rests solely
|
|
on physical per-collection boundary). That missing agent-axis looks LOAD-BEARING (channel-2's real fix), not just
|
|
defense-in-depth. WT tracing write/query path; **rev 1.2 contract (ch1 + ch2 + agent-axis stamp) comes to ratatoskr
|
|
for the enforcement read** (INV-368-9 blocks WT preflight until then).
|
|
**SCRUB + REPRO (2026-07-18):** operator ruled a FULL memory scrub both sides (contamination noise, nothing
|
|
worth keeping). Server-side (personal/b125): infra-ops wiping all agent chroma collections + shared promotion
|
|
state. Ours: **Tier-3 memory.db scrubbed to 0 chunks, :8392 restarted born-empty, no backup** (operator direct
|
|
go; new pid 3548269; affect plane untouched). Personal is ALREADY b125 (redeployed 20:35Z; /version 404 =
|
|
b125 dropped the route, misled us). Current-code status: the fresh Lofn 'Vuong' chunks are the FEEDBACK LOOP
|
|
(Lofn's own recitations re-extracted), NOT proven fresh mis-write; the original cross-agent mis-write ("I'm
|
|
Saga, and you are Vuong" → lofn.chroma) is Jun-3 code, UNPROVEN on b125. **QUEUED (after BOTH scrubs confirmed):
|
|
the marker repro** — drive ONE fresh Sindra session (combined :8392 bound), user introduces an absurd greppable
|
|
marker ("my favorite mineral is rhodochrosite", NOT the name), idle a few min for extraction, report timestamps
|
|
→ infra-ops greps lofn.chroma + our store. Marker in lofn.chroma = mis-write ALIVE on b125 (code fix); marker
|
|
only in our store = channel 2 is just the feedback loop + agent-axis hardening. Post-scrub our #368 backfill is
|
|
a NO-OP (born-conjunctive once the fix ships); contract keeps migration machinery for other consumers. DO NOT
|
|
re-ask Lofn the name meanwhile (re-seeds the loop). Awaiting WT infra-ops server-side-scrub confirm to run it.
|
|
**RAN 2026-07-18 (both scrubs confirmed clean):** marker repro executed — fresh Sindra session 822c3af7
|
|
(combined :8392), user introduced "my favorite mineral is rhodochrosite", 5 more turns drove the idle-scan
|
|
promotion at ~6 turns. RESULT: marker promoted to OUR store scoped **{end_user}-only** (chunk 647aeac6;
|
|
record.agent_id "ratatoskr:sindra" metadata; worldtree_scope "end_user"; verbatim "Favorite mineral is
|
|
rhodochrosite") = **channel-1 baseline exhibit** (pre-fix shape; agent_self absent, to be backfilled from the
|
|
agent_id metadata). **KEY WT CORRECTION:** consumer-defined agents (ratatoskr:sindra) have NO server-side
|
|
chroma — they write ONLY to our store via Bifrost; only FOUNDATIONAL agents (lofn/forseti/mimir) get
|
|
server-side collections. Confirmed on the wire: Sindra IS Bifrost-attached (queried us scan+search) UNLIKE
|
|
foundational Lofn (handshake-only, no memory-call). DECISIVE channel-2 grep (marker must be in ZERO
|
|
server-side stores; ANY server-side hit = cross-plane bug, no benign reading) still PENDING infra-ops.
|
|
**ETA (WT): rev 1.2 amended contract reaches us TODAY 2026-07-18 for the consumer-side enforcement read;
|
|
ship-ready LIKELY 2026-07-19; our integration risk LOW** (born-fresh store won't exercise the backfill
|
|
machinery, which is the bulk of the impl). ratatoskr is the enforcement half; INV-368-9 blocks WT preflight.
|
|
**CHANNEL 2 CLOSED CLEAN + ENFORCEMENT READ DONE (2026-07-18):** marker routed ONLY to our store, zero in every
|
|
foundational store → current-b125 write path VERIFIED correct; the contamination was old-code (saga-v1, pre-#211)
|
|
seeding + a b125 recited-content re-promotion feedback loop (→ WT #369). Contract FINAL: `Worldtree/docs/contracts/
|
|
issues/368.contract.md` rev 1.2 (cda308c). **Consumer-side enforcement read complete — read-path CONFORMANT
|
|
(conjunctive-per-element re-verified); 3 findings sent to WT (thread 01KXSBY3A8):** F1[HIGH] INV-368-10 can fail-OPEN
|
|
at our store — our empty-scope = MATCH-ALL, so the guard's "drop end_user element" must NOT reach us as an empty
|
|
filter (else total bleed); fix = don't-call/never-match. F2[HIGH] backfill scan NOT snapshot-stable — our scan is
|
|
offset-over-updated_at (chunk_id NOT sortable), and re-upserts advance updated_at, so multi-page paging under
|
|
concurrent mutation can SKIP chunks → un-stamped → post-flip amnesia (INV-368-3 failure); fix = whole-corpus
|
|
single-page scan (no consumer change) OR add chunk_id sort (consumer change). F3[MED] sentinel needs a valid 1024-dim
|
|
embedding or our vec0 upsert raises → backfill never completes. All 3 are WT-side backfill/guard adjustments, NOT
|
|
consumer reworks — the "zero consumer changes" claim holds for reads, needs the qualifier for backfill/guard.
|
|
**FINAL 2026-07-18: all 3 findings FOLDED into contract rev 1.3 (3e3f629).** F1 → INV-368-10 now mandates
|
|
drop-to-empty = SKIP the wire call (test `drop_to_empty_skips_wire` asserts our store gets NO query). F2 →
|
|
option (a) pinned: backfill scans whole corpus in ONE page (test `single_page_snapshot`; immutable-key paging
|
|
= recorded scale follow-up that WOULD be a consumer change). F3 → sentinel writes a zero-vector at embedding_dim
|
|
(test `sentinel_carries_vector`). Our read-path conformance + conjunctive-per-element check cited in the revision
|
|
record. WT implementing against rev 1.3 (TDD after their context-reset; cross-frontier panels follow); ship ~7/19.
|
|
**QUEUED — backfill live-verify:** when WT's backfill lands, drive a SYNTHETIC legacy corpus into our store, run
|
|
their backfill against it, confirm 100% stamped (no un-attributable skips) + sentinel writes + reads flip clean,
|
|
all BEFORE any real flip. **#369** = the recited-content re-promotion feedback-loop policy class (WT owns).
|
|
**#368 arc DONE from ratatoskr's side** — diagnosed → reported → operator ruled → exhibits + both scrubs →
|
|
channel-2 closed clean → enforcement read (3 findings, all folded).
|
|
**✅ LIVE SILO TEST DONE — PASSED (2026-07-18, WT b127).** See the ✅ #368 summary above + full record in
|
|
`persistent-memory.d/2026-07-18-368-silo-test-passed.md`. Betty (throwaway) DELETED (WT 404 + local registry);
|
|
Sindra intact. **TIER3 CLI: define/patch adapted to the b127 `role` schema** (commit `860e0d5`, v0.21.1) — the
|
|
`delete` verb works (used it to remove betty: WT hard-delete + `remove_local_agent`). Drive was via the CLI
|
|
combined bind (`--bifrost-url http://10.100.10.50:8392`), not web. **Store SCRUBBED clean afterward (2026-07-18,
|
|
`reset-sindra-stores.sh`, rolling backup `db-reset-backup/`): memory 0 / affect 0, provider restarted empty (pid
|
|
3856058); next bind re-runs the no-op migration + sentinel write.** Silo-test evidence preserved in the detail file.
|