ratatoskr/docs/contracts/bifrost_memory_provider.contract.md

contract_version, module, purpose, touches, language, complexity, estimated_loc, confidence, assumptions, open_questions, external_invariants, revisions

contract_version	module	purpose	touches	language	complexity	estimated_loc	confidence	assumptions	open_questions	external_invariants	revisions
2.1	ratatoskr.provider.memory_store	Memory-plane Bifrost consumer (v1 basic plane): a SQLite+sqlite-vec-backed durable memory store Worldtree persists Tier-3 agent memory chunks into and recalls via vector search.	src/ratatoskr/provider/memory_store.py tests/test_provider_memory.py	python	high	320	0.82	bifrost>=0.6.1 exposes build_memory_app, dispatch_memory_call, JwtVerifier, ConsumerRegistration, StoreCapabilities, MemoryDataStore, InvalidArguments, IdempotencyConflict, RevisionMismatch per bifrost/reference_server/memory.py + bifrost.memory. v1 = worldtree-dev's BASIC PLANE only (search / get / get_many / upsert_many / delete_many + describe_store + health), the ONLY surface Tier-3's live path touches (#294); Worldtree v0.35.3 already negotiates it. Chunk record field names are taken from the reference store (named inline below) but the AUTHORITATIVE pin is TDD against InMemoryMemoryStore, as it was for affect. Embedding dimension matches Worldtree's PINNED_EMBEDDER_DIM, supplied as config (env RATATOSKR_MEMORY_EMBEDDING_DIM); the sqlite-vec virtual table is created at that fixed dim.	Whether the memory DB shares one SQLite file with affect or its own — default SEPARATE per plane; the dev-shell entrypoint reads RATATOSKR_MEMORY_DB (analogous to RATATOSKR_AFFECT_DB). Revisit at the combined two-plane server (guide §7).	source invariant_id ~/development/bifrost/bifrost/reference_server/memory.py InMemoryMemoryStore source invariant_id ~/development/bifrost/docs/implementing-a-consumer.md §5 memory plane	version at summary delta 1.2 2026-06-16 Repin bifrost 0.7.0→0.8.0 (wire v0.5→v0.6, #11): search `scope_filter` split into `scope_all` (AND/intersection) + `scope_any` (OR/union over a list of conjunctive scopes). No-compat: `scope_filter` removed. Adds union-visibility recall in one call — the fix for the #295/#297 AND silent-zero foot-gun. Store at parity with the v0.6 reference `_matches_scope` / `_validate_scope`. MODIFIED ADDED search signature: scope_filter -> scope_all + scope_any INV-005 scope isolation -> composed v0.6 (scope_all AND ∧ scope_any OR-union) PRE-003 validates axes in BOTH fields; STEP 1 = _validate_scope (shape + lattice) scope_any_union test (#295/#297 union capability); both-fields-empty match-all version at summary delta 1.1 2026-06-15 Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005. MODIFIED INV-001 byte-equal -> semantic round-trip; named structural field keys inline INV-002 idempotency_id = ("default", verb, actor, key) — reconciled with STEPS INV-005 search only (scan was deferred) search: top_k in-scope, scope_filter shape, recalled_view, metadata_filter-v1 reject delete_many atomicity; get_many clarified; transaction-term clarified

Context

The second plane of ratatoskr's Tier-3 Bifrost consumer (after the shipped affect plane). A SQLite + sqlite-vec durable store Worldtree writes agent memory chunks into (upsert_many) and recalls from by vector similarity (search), plus point reads (get/get_many) and deletes (delete_many). v1 is worldtree-dev's basic plane — the only surface Tier-3's live path uses; the gated verbs (edges, scan, atomic_supersede, mark_*, patch, maintenance) are deferred. We implement bifrost's own MemoryDataStore Protocol and hand it to build_memory_app. Conformance is #195 parity vs InMemoryMemoryStore.

The boundary (ADR-0001/0002/0009): Worldtree owns intelligence — appraisal, consolidation, trust; we own permanence. But unlike affect (blind conduit), memory is a structural index: we read a few fields of each chunk — record["embedding"] (rank), record["scope"] (isolation), record["id"] + revision (optimistic locking), and origin/injection_source (the consistency rule). The semantic content, record["distillate"], and inert fields (trust_tier/provenance/source_role) are persisted verbatim and never interpreted.

Data flow

• In: Worldtree → POST /bifrost/memory-call → library validates envelope + per-dispatch JWT → the verb on our store.
• Chunk record (key fields we read; rest is opaque payload): id (the chunk id — reference falls back to chunk_id/memory_id), embedding (the vector — fallback vector), scope (a {axis: value} dict — the isolation key), origin + injection_source (consistency rule), distillate (the recall view). Everything else (content, metadata, trust_tier, …) is stored verbatim.
• At rest: SQLite —
  • memory_chunks(chunk_id PK, record_json, revision, scope_json, origin, ...) — the verbatim chunk + extracted columns (chunk_id, scope) for isolation.
  • sqlite-vec virtual table memory_vec(chunk_id, embedding[<dim>]) — the index.
  • memory_idempotency(idempotency_id PK, digest, expires_at) — replay/conflict cache (affect-parallel shape).
• Out: upsert_many → {"upserted": N, "replayed": bool}; search → list of {chunk, chunk_id, score, recalled_view, revision} where recalled_view = the chunk's distillate field, or the whole chunk if absent (per the reference); delete_many → {"deleted": N}; get → the verbatim record + a revision key, or None; get_many(ids) → the list form of get (found records only).

Invariants

• INV-001 [hard]: Persist verbatim (semantic round-trip); read only the structural surface. The whole chunk is stored and a read deserializes to a Python object EQUAL to the input (json.loads(record_json) == input) — not byte-equal (key order / formatting may differ); get additionally attaches a revision key to the returned object. The store reads ONLY record["embedding"], record["scope"], record["id"] + revision, and origin/injection_source. Content / distillate / inert fields (trust_tier/provenance/source_role) are NOT interpreted.
• INV-002 [hard]: Idempotency = replay-or-conflict, actor-scoped (affect- parallel). idempotency_id = ("default", <verb>, _ctx_actor(ctx), idempotency_key) — the literal "default" class slot + the verb name, matching the reference 4-tuple (idempotency_class tunes only the cache TTL, not the id). Same digest → replay (replayed: True, no re-write); different digest → raise IdempotencyConflict. Actor from ctx, never from the record.
• INV-003 [hard]: Optimistic locking. When upsert_many carries expected_revisions, each record's stored revision must equal the expected; any mismatch → raise RevisionMismatch and the whole batch rolls back. Each successful upsert increments the chunk's revision (a first insert → revision 1).
• INV-004 [hard]: Atomic batch. upsert_many applies all records + their vec rows + the idempotency record in one transaction; on any error nothing is persisted (no partial batch, no orphaned vec rows).
• INV-005 [hard]: Scope isolation (wire v0.6, #11). search filters by two explicit fields: scope_all (AND/intersection — record ⊇ every named axis) and scope_any (OR/union over a LIST of conjunctive scope dicts — record ⊇ ≥1 element, each element AND-matched as a whole). They compose by AND; both empty → no scope constraint. A search never returns a chunk outside the composed filter. Byte-faithful to the reference _matches_scope. (scope_any is the union-visibility primitive that resolves the #295/#297 silent-zero — a subset-scoped chunk now recalls via an OR member.)
• INV-006 [hard]: Capabilities match implementation (advertise-⇒-implement). describe_store advertises ONLY what v1 implements: relational_edges_supported=False, atomic_supersede_supported=False, transaction_supported=False, optimistic_locking_supported=True, filterable_metadata_fields=[]. (transaction_supported is the bifrost wire-level multi-op transaction capability — NOT our internal SQLite transactions, which we use for atomic batches.) The client gates the gated verbs off these.
• INV-007 [hard]: origin == "injected_context" requires injection_source; a non-injected record carrying injection_source is rejected — both raise InvalidArguments (mirrors the reference).
• INV-008 [hard]: The store is REQUIRED (build_memory_app(store=None) raises); identity/scope/actor come from ctx, never call args.

Concurrency

SQLite WAL (concurrent readers, single writer). upsert_many/delete_many serialize on the writer; search/get are concurrent reads. sqlite-vec index writes ride inside the upsert/delete transaction. The connection is opened check_same_thread=False with PRAGMA busy_timeout=5000 (mirrors the affect store): the provider is an ASGI app, so uvicorn/Starlette (and TestClient always) may run a handler off the connection's creating thread — the event loop serializes the sync sqlite calls, so this is safe; busy_timeout preps the composite/standalone two-process topology over the same db. (Surfaced by a TestClient-driven memory search through the #18 D1 combined provider — the direct-store tests structurally could not.)

Division of labor (library vs store)

The bifrost library owns the wire (envelope validation, per-dispatch JWT, scope authorization, error mapping of our typed exceptions, capability negotiation, routes). This contract owns the store (the basic verbs + SQLite

• sqlite-vec persistence/index) + the thin build_memory_provider_app wiring.

Integration points

• bifrost.consumer.build_memory_app(store, verifier, registration, maintenance_store=None, hooks=None) → Starlette app.
• bifrost.reference_server.JwtVerifier + bifrost.consumer.ConsumerRegistration.
• bifrost.memory.{StoreCapabilities, InvalidArguments, IdempotencyConflict, RevisionMismatch} — typed surface.
• Conformance (tests): bifrost.consumer.testing.InMemoryMemoryStore + bifrost.memory.dispatch_memory_call (#195). The reference is the authoritative pin for exact field names + wire shapes.
• sqlite-vec — the vector index extension loaded into the connection.

Constraints

• [security] Never log chunk content / distillate. Index the vector + scope; don't interpret semantics.
• [compatibility] Implement bifrost's MemoryDataStore shape exactly; raise its typed exceptions; never fork the wire. Gated verbs are absent + advertised unsupported.
• [correctness] search ranks by cosine over record["embedding"]; scope isolation (INV-005) is non-negotiable; top_k counts IN-SCOPE results (see search STEPS).

Out of scope (deferred — do NOT flag as drift)

• Gated/maintenance verbs: upsert_edges/get_edges_for, scan, mark_invalid/mark_superseded, patch_many, atomic_supersede, lease/checkpoint. Absent + advertised-unsupported.
• metadata_filter beyond scope: advertise filterable_metadata_fields=[]; a non-empty metadata_filter is unsupported in v1 (rejected — see search PRE).
• The combined two-plane server (guide §7) — separate memory + affect apps in v1.
• Deployment — dev-box background shell (ratatoskr-memory-provider), no systemd/infra.
• idempotency-cache TTL pruning — expires_at recorded, eviction deferred (affect-parallel).

FN open_memory_store(db_path: str, *, embedding_dim: int) -> RatatoskrMemoryStore
BRIEF: Open the SQLite+sqlite-vec memory store, creating schema + the vec index on first use.
PRE: [PRE-001 hard] db_path writable or ":memory:" -- guard
PRE: [PRE-002 hard] embedding_dim is a positive int (matches Worldtree PINNED_EMBEDDER_DIM) -- assert
POST: [POST-001 return_value] store.describe_store() advertises the v1 capability set (INV-006) -- assert
POST: [POST-002 state_change] memory_chunks + memory_vec(dim) + memory_idempotency exist -- schema present
STEPS:
  1. [setup] CONNECT sqlite3; enable_load_extension; LOAD sqlite-vec; WAL (skip for ":memory:")
  2. [sequential, flexibility=prescriptive] CREATE memory_chunks + memory_idempotency tables IF NOT EXISTS
  3. [sequential, flexibility=prescriptive] CREATE VIRTUAL TABLE memory_vec USING vec0(chunk_id TEXT PRIMARY KEY, embedding float[embedding_dim]) IF NOT EXISTS
  4. [cleanup] RETURN RatatoskrMemoryStore(conn, embedding_dim)
TESTS:
  fresh_db [happy,tracer]: open ":memory:" dim=8 → describe_store() has the v1 caps; tables queryable
  reopen [happy]: open existing file twice → idempotent schema

FN describe_store(self) -> dict
BRIEF: Static capability descriptor (sync, no I/O).
POST: [POST-001 return_value] returns the bifrost StoreCapabilities dict with v1 values (INV-006) -- assert relational_edges/atomic_supersede/transaction False, optimistic_locking True, filterable_metadata_fields []
STEPS:
  1. [cleanup] RETURN StoreCapabilities(relational_edges_supported=False, optimistic_locking_supported=True, atomic_supersede_supported=False, transaction_supported=False, filterable_metadata_fields=[]).to_dict()
TESTS:
  caps [happy]: returns exactly the v1 capability dict; advertise-⇒-implement holds

FN upsert_many(self, records: list[dict], *, idempotency_key: str, ctx, expected_revisions: dict | None = None, idempotency_class: str | None = None) -> dict
BRIEF: Persist chunks verbatim + index their vectors, atomically, replay-or-conflict idempotent, optimistic-locked.
PRE: [PRE-001 hard] idempotency_key non-empty str -- else InvalidArguments
PRE: [PRE-002 hard] each injected_context record has injection_source; non-injected has none -- else InvalidArguments (INV-007)
POST: [POST-001 return_value] {"upserted": len(records), "replayed": False} on persist; {"...","replayed": True} on replay (INV-002) -- assert
POST: [POST-002 state_change] each chunk stored verbatim + vector indexed + revision incremented (first insert → 1); expected_revisions enforced (INV-003) -- assert
POST: [POST-003 side_effect] on ANY error, nothing persisted (INV-004) -- rollback
ERROR_ROUTING:
  InvalidArguments: { local_handling: raise on bad key / injection_source rule, flow_control: abort, state_recovery: none }
  IdempotencyConflict: { local_handling: raise on key-reuse-different-digest, flow_control: abort, state_recovery: none }
  RevisionMismatch: { local_handling: raise on stale expected_revision, flow_control: abort, state_recovery: full batch rollback }
STEPS:
  1. [setup] validate idempotency_key; digest over {records, expected_revisions}; idempotency_id = ("default", "upsert_many", _ctx_actor(ctx), idempotency_key)  -- matches INV-002
  2. [branch] idempotency lookup: same digest → RETURN replayed; different → RAISE IdempotencyConflict
  3. [sequential, flexibility=prescriptive] BEGIN; IF expected_revisions: assert each stored revision matches else RAISE RevisionMismatch
  4. [loop] FOR each record: validate origin/injection_source; UPSERT memory_chunks (record_json + scope_json, revision+1); UPSERT memory_vec(record["id"], record["embedding"])
  5. [sequential] record idempotency (digest, expires_at = now + ttl(idempotency_class)); COMMIT
  6. [cleanup] RETURN {"upserted": len(records), "replayed": False}
TESTS:
  basic_upsert [happy,tracer]: 2 records → {"upserted":2,"replayed":False}; get() round-trips each verbatim + revision=1
  replay [happy]: same key+payload twice → first writes (revision 1), second {"replayed":True} with NO further write (revision stays 1)
  conflict [adversarial]: same key, different records → IdempotencyConflict; first batch intact
  optimistic_lock [adversarial]: expected_revisions stale → RevisionMismatch; nothing written
  injection_rule [adversarial]: injected_context w/o injection_source → InvalidArguments; no write
  parity_vs_reference [scenario]: same upsert_many envelopes through dispatch_memory_call vs InMemoryMemoryStore → wire bodies agree (#195)

FN search(self, vector: list[float], *, top_k: int, scope_all: dict | None = None, scope_any: list | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped by the v0.6 scope_all/scope_any filter, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
PRE: [PRE-003 hard] scope_all is a flat dict and scope_any a list of flat dicts (else InvalidArguments); every axis in BOTH ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.6 lattice, matching the reference _validate_scope (#10 agent_self canonical, #11 scope split)
POST: [POST-001 return_value] returns the top_k highest-cosine records passing the composed v0.6 filter — `(scope_all empty OR record ⊇ scope_all) AND (scope_any empty OR record ⊇ ≥1 element)`; at most top_k, never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS:
  1. [setup] scope_all ← scope_all or {}; scope_any ← scope_any or []; validate via _validate_scope (flat-dict / list-of-dicts shape + every axis ∈ the v0.6 lattice, else InvalidArguments / InvalidFilter)
  2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only records passing _matches_scope(scope_all, scope_any) (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
  3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS:
  basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
  scope_isolation [adversarial]: two scopes, scope_all one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
  scope_any_union [scenario]: scope_any=[{end_user:u},{agent_self:a}] recalls BOTH a subject-scoped and a self-scoped chunk in one call (#295/#297 union capability); scope_all+scope_any compose by AND
  empty [boundary]: search empty store → []; both fields empty → match all
  metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
  lattice_axes [adversarial]: out-of-lattice axis in scope_all OR scope_any → InvalidFilter; non-list scope_any → InvalidArguments; agent_self admitted (wire v0.5, #10)
  parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)

FN get(self, chunk_id: str) -> dict | None
BRIEF: Point read; returns the verbatim chunk + current revision, or None. get_many(ids) is the list form (found records only).
POST: [POST-001 return_value] stored record (verbatim, json.loads) + "revision" key, or None if absent (INV-001) -- assert
STEPS:
  1. [sequential] SELECT record_json, revision WHERE chunk_id; RETURN json.loads + revision, or None
TESTS:
  get_hit [happy]: after upsert → record equal + revision present
  get_absent [boundary]: unknown id → None
  get_many [happy]: get_many([present, absent]) → [present record] only

FN delete_many(self, ids: list[str]) -> dict
BRIEF: Delete chunks (+ their vec rows) by id, transactionally.
POST: [POST-001 return_value] {"deleted": N} where N = ids that existed -- assert
POST: [POST-002 state_change] in ONE transaction, deleted chunks gone from memory_chunks AND memory_vec; partial failure rolls back the whole batch (no orphan vec rows) -- assert
STEPS:
  1. [sequential, flexibility=prescriptive] BEGIN; FOR each id present: DELETE from memory_chunks + memory_vec; count; COMMIT
  2. [cleanup] RETURN {"deleted": count}
TESTS:
  delete_hit [happy]: delete 1 of 2 → {"deleted":1}; gone from chunks + vec; search won't surface it
  delete_absent [boundary]: unknown id → {"deleted":0}

FN build_memory_provider_app(store: RatatoskrMemoryStore, heimdall_key: bytes, consumer_id: str = "ratatoskr") -> Starlette
BRIEF: Wire JwtVerifier + registration; hand the store to bifrost's build_memory_app.
PRE: [PRE-001 hard] store.describe_store() returns a dict (advertises caps) -- assert (INV-008)
PRE: [PRE-002 hard] heimdall_key non-empty bytes -- assert
POST: [POST-001 return_value] Starlette app exposing POST /bifrost/handshake + POST /bifrost/memory-call -- assert routes
STEPS:
  1. [setup] verifier = JwtVerifier(HS256, heimdall_key); registration = ConsumerRegistration(consumer_id)
  2. [sequential, flexibility=prescriptive] app = build_memory_app(store=store, verifier=verifier, registration=registration)
  3. [cleanup] RETURN app
TESTS:
  builds_app [happy,tracer]: valid store + key → app with the two routes (incl. POST)
  bad_key [error]: empty heimdall_key → raises at construction