Memory-plane Bifrost consumer (v1 basic plane): a SQLite+sqlite-vec-backed durable memory store Worldtree persists Tier-3 agent memory chunks into and recalls via vector search.
v1 = worldtree-dev's BASIC PLANE only (search / get / get_many / upsert_many / delete_many + describe_store + health), the ONLY surface Tier-3's live path touches (#294); Worldtree v0.35.3 already negotiates it.
Chunk record field names are taken from the reference store (named inline below) but the AUTHORITATIVE pin is TDD against InMemoryMemoryStore, as it was for affect.
Embedding dimension matches Worldtree's PINNED_EMBEDDER_DIM, supplied as config (env RATATOSKR_MEMORY_EMBEDDING_DIM); the sqlite-vec virtual table is created at that fixed dim.
Whether the memory DB shares one SQLite file with affect or its own — default SEPARATE per plane; the dev-shell entrypoint reads RATATOSKR_MEMORY_DB (analogous to RATATOSKR_AFFECT_DB). Revisit at the combined two-plane server (guide §7).
The second plane of ratatoskr's Tier-3 Bifrost consumer (after the shipped affect
plane). A SQLite + sqlite-vec durable store Worldtree writes agent memory
chunks into (upsert_many) and recalls from by vector similarity
(search), plus point reads (get/get_many) and deletes (delete_many).
v1 is worldtree-dev's basic plane — the only surface Tier-3's live path uses;
the gated verbs (edges, scan, atomic_supersede, mark_*, patch, maintenance) are
deferred. We implement bifrost's ownMemoryDataStore Protocol and hand it to
build_memory_app. Conformance is #195 parity vs InMemoryMemoryStore.
The boundary (ADR-0001/0002/0009): Worldtree owns intelligence — appraisal,
consolidation, trust; we own permanence. But unlike affect (blind conduit),
memory is a structural index: we read a few fields of each chunk —
record["embedding"] (rank), record["scope"] (isolation), record["id"] +
revision (optimistic locking), and origin/injection_source (the consistency
rule). The semantic content, record["distillate"], and inert fields
(trust_tier/provenance/source_role) are persisted verbatim and never
interpreted.
Data flow
In: Worldtree → POST /bifrost/memory-call → library validates envelope +
per-dispatch JWT → the verb on our store.
Chunk record (key fields we read; rest is opaque payload):id (the chunk
id — reference falls back to chunk_id/memory_id), embedding (the vector —
fallback vector), scope (a {axis: value} dict — the isolation key),
origin + injection_source (consistency rule), distillate (the recall
view). Everything else (content, metadata, trust_tier, …) is stored verbatim.
At rest: SQLite —
memory_chunks(chunk_id PK, record_json, revision, scope_json, origin, ...) —
the verbatim chunk + extracted columns (chunk_id, scope) for isolation.
sqlite-vec virtual table memory_vec(chunk_id, embedding[<dim>]) — the index.
Out:upsert_many → {"upserted": N, "replayed": bool}; search → list of
{chunk, chunk_id, score, recalled_view, revision} where recalled_view =
the chunk's distillate field, or the whole chunk if absent (per the reference);
delete_many → {"deleted": N}; get → the verbatim record + a revision key,
or None; get_many(ids) → the list form of get (found records only).
Invariants
INV-001 [hard]: Persist verbatim (semantic round-trip); read only the
structural surface. The whole chunk is stored and a read deserializes to a
Python object EQUAL to the input (json.loads(record_json) == input) — not
byte-equal (key order / formatting may differ); get additionally attaches a
revision key to the returned object. The store reads ONLY record["embedding"],
record["scope"], record["id"] + revision, and origin/injection_source.
Content / distillate / inert fields (trust_tier/provenance/source_role)
are NOT interpreted.
INV-002 [hard]: Idempotency = replay-or-conflict, actor-scoped (affect-
parallel). idempotency_id = ("default", <verb>, _ctx_actor(ctx), idempotency_key)
— the literal "default" class slot + the verb name, matching the reference
4-tuple (idempotency_class tunes only the cache TTL, not the id). Same digest →
replay (replayed: True, no re-write); different digest → raise
IdempotencyConflict. Actor from ctx, never from the record.
INV-003 [hard]: Optimistic locking. When upsert_many carries
expected_revisions, each record's stored revision must equal the expected; any
mismatch → raise RevisionMismatch and the whole batch rolls back. Each
successful upsert increments the chunk's revision (a first insert → revision 1).
INV-004 [hard]: Atomic batch.upsert_many applies all records + their
vec rows + the idempotency record in one transaction; on any error nothing is
persisted (no partial batch, no orphaned vec rows).
INV-005 [hard]: Scope isolation.search results are filtered to records
whose record["scope"] matches every axis in scope_filter; a search never
returns another scope's chunk.
INV-006 [hard]: Capabilities match implementation (advertise-⇒-implement).
describe_store advertises ONLY what v1 implements: relational_edges_supported=False,
atomic_supersede_supported=False, transaction_supported=False,
optimistic_locking_supported=True, filterable_metadata_fields=[].
(transaction_supported is the bifrost wire-level multi-op transaction
capability — NOT our internal SQLite transactions, which we use for atomic
batches.) The client gates the gated verbs off these.
INV-007 [hard]: origin == "injected_context" requires injection_source; a
non-injected record carrying injection_source is rejected — both raise
InvalidArguments (mirrors the reference).
INV-008 [hard]: The store is REQUIRED (build_memory_app(store=None) raises);
identity/scope/actor come from ctx, never call args.
Concurrency
SQLite WAL (concurrent readers, single writer). upsert_many/delete_many
serialize on the writer; search/get are concurrent reads. sqlite-vec index
writes ride inside the upsert/delete transaction.
Division of labor (library vs store)
The bifrost library owns the wire (envelope validation, per-dispatch JWT,
scope authorization, error mapping of our typed exceptions, capability
negotiation, routes). This contract owns the store (the basic verbs + SQLite
sqlite-vec persistence/index) + the thin build_memory_provider_app wiring.
Conformance (tests):bifrost.consumer.testing.InMemoryMemoryStore + bifrost.memory.dispatch_memory_call (#195). The reference is the authoritative pin for exact field names + wire shapes.
sqlite-vec — the vector index extension loaded into the connection.
Constraints
[security] Never log chunk content / distillate. Index the vector + scope; don't interpret semantics.
[compatibility] Implement bifrost's MemoryDataStore shape exactly; raise its typed exceptions; never fork the wire. Gated verbs are absent + advertised unsupported.
[correctness]search ranks by cosine over record["embedding"]; scope isolation (INV-005) is non-negotiable; top_k counts IN-SCOPE results (see search STEPS).
FN upsert_many(self, records: list[dict], *, idempotency_key: str, ctx, expected_revisions: dict | None = None, idempotency_class: str | None = None) -> dict
BRIEF: Persist chunks verbatim + index their vectors, atomically, replay-or-conflict idempotent, optimistic-locked.
PRE: [PRE-001 hard] idempotency_key non-empty str -- else InvalidArguments
PRE: [PRE-002 hard] each injected_context record has injection_source; non-injected has none -- else InvalidArguments (INV-007)
POST: [POST-001 return_value] {"upserted": len(records), "replayed": False} on persist; {"...","replayed": True} on replay (INV-002) -- assert
POST: [POST-002 state_change] each chunk stored verbatim + vector indexed + revision incremented (first insert → 1); expected_revisions enforced (INV-003) -- assert
POST: [POST-003 side_effect] on ANY error, nothing persisted (INV-004) -- rollback
ERROR_ROUTING:
InvalidArguments: { local_handling: raise on bad key / injection_source rule, flow_control: abort, state_recovery: none }
IdempotencyConflict: { local_handling: raise on key-reuse-different-digest, flow_control: abort, state_recovery: none }
RevisionMismatch: { local_handling: raise on stale expected_revision, flow_control: abort, state_recovery: full batch rollback }
STEPS:
1. [setup] validate idempotency_key; digest over {records, expected_revisions}; idempotency_id = ("default", "upsert_many", _ctx_actor(ctx), idempotency_key) -- matches INV-002
2. [branch] idempotency lookup: same digest → RETURN replayed; different → RAISE IdempotencyConflict
3. [sequential, flexibility=prescriptive] BEGIN; IF expected_revisions: assert each stored revision matches else RAISE RevisionMismatch
4. [loop] FOR each record: validate origin/injection_source; UPSERT memory_chunks (record_json + scope_json, revision+1); UPSERT memory_vec(record["id"], record["embedding"])
5. [sequential] record idempotency (digest, expires_at = now + ttl(idempotency_class)); COMMIT
6. [cleanup] RETURN {"upserted": len(records), "replayed": False}
TESTS:
basic_upsert [happy,tracer]: 2 records → {"upserted":2,"replayed":False}; get() round-trips each verbatim + revision=1
replay [happy]: same key+payload twice → first writes (revision 1), second {"replayed":True} with NO further write (revision stays 1)
conflict [adversarial]: same key, different records → IdempotencyConflict; first batch intact
optimistic_lock [adversarial]: expected_revisions stale → RevisionMismatch; nothing written
injection_rule [adversarial]: injected_context w/o injection_source → InvalidArguments; no write
parity_vs_reference [scenario]: same upsert_many envelopes through dispatch_memory_call vs InMemoryMemoryStore → wire bodies agree (#195)
FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS:
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis])
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS:
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
empty [boundary]: search empty store → []
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
FN get(self, chunk_id: str) -> dict | None
BRIEF: Point read; returns the verbatim chunk + current revision, or None. get_many(ids) is the list form (found records only).
POST: [POST-001 return_value] stored record (verbatim, json.loads) + "revision" key, or None if absent (INV-001) -- assert
STEPS:
1. [sequential] SELECT record_json, revision WHERE chunk_id; RETURN json.loads + revision, or None
TESTS:
get_hit [happy]: after upsert → record equal + revision present
get_absent [boundary]: unknown id → None
get_many [happy]: get_many([present, absent]) → [present record] only
FN delete_many(self, ids: list[str]) -> dict
BRIEF: Delete chunks (+ their vec rows) by id, transactionally.
POST: [POST-001 return_value] {"deleted": N} where N = ids that existed -- assert
POST: [POST-002 state_change] in ONE transaction, deleted chunks gone from memory_chunks AND memory_vec; partial failure rolls back the whole batch (no orphan vec rows) -- assert
STEPS:
1. [sequential, flexibility=prescriptive] BEGIN; FOR each id present: DELETE from memory_chunks + memory_vec; count; COMMIT
2. [cleanup] RETURN {"deleted": count}
TESTS:
delete_hit [happy]: delete 1 of 2 → {"deleted":1}; gone from chunks + vec; search won't surface it
delete_absent [boundary]: unknown id → {"deleted":0}
FN build_memory_provider_app(store: RatatoskrMemoryStore, heimdall_key: bytes, consumer_id: str = "ratatoskr") -> Starlette
BRIEF: Wire JwtVerifier + registration; hand the store to bifrost's build_memory_app.
PRE: [PRE-001 hard] store.describe_store() returns a dict (advertises caps) -- assert (INV-008)
PRE: [PRE-002 hard] heimdall_key non-empty bytes -- assert
POST: [POST-001 return_value] Starlette app exposing POST /bifrost/handshake + POST /bifrost/memory-call -- assert routes
STEPS:
1. [setup] verifier = JwtVerifier(HS256, heimdall_key); registration = ConsumerRegistration(consumer_id)
2. [sequential, flexibility=prescriptive] app = build_memory_app(store=store, verifier=verifier, registration=registration)
3. [cleanup] RETURN app
TESTS:
builds_app [happy,tracer]: valid store + key → app with the two routes (incl. POST)
bad_key [error]: empty heimdall_key → raises at construction
vh
1f94e5f251