diff --git a/docs/contracts/bifrost_memory_provider.contract.md b/docs/contracts/bifrost_memory_provider.contract.md index 5bba6ef..f7d4243 100644 --- a/docs/contracts/bifrost_memory_provider.contract.md +++ b/docs/contracts/bifrost_memory_provider.contract.md @@ -8,20 +8,30 @@ touches: language: "python" complexity: "high" estimated_loc: 320 -confidence: 0.78 +confidence: 0.82 assumptions: - "bifrost>=0.6.1 exposes build_memory_app, dispatch_memory_call, JwtVerifier, ConsumerRegistration, StoreCapabilities, MemoryDataStore, InvalidArguments, IdempotencyConflict, RevisionMismatch per bifrost/reference_server/memory.py + bifrost.memory." - - "v1 = worldtree-dev's BASIC PLANE only (search / get / upsert / delete + describe_store + health), which is the ONLY surface Tier-3's live path touches (#294). Worldtree v0.35.3 already requests + maps it — no Worldtree-side blocker." - - "Exact chunk-record field names (embedding vector key, scope keys, id key) are pinned in TDD against InMemoryMemoryStore — the executable spec — as they were for affect." + - "v1 = worldtree-dev's BASIC PLANE only (search / get / get_many / upsert_many / delete_many + describe_store + health), the ONLY surface Tier-3's live path touches (#294); Worldtree v0.35.3 already negotiates it." + - "Chunk record field names are taken from the reference store (named inline below) but the AUTHORITATIVE pin is TDD against InMemoryMemoryStore, as it was for affect." - "Embedding dimension matches Worldtree's PINNED_EMBEDDER_DIM, supplied as config (env RATATOSKR_MEMORY_EMBEDDING_DIM); the sqlite-vec virtual table is created at that fixed dim." open_questions: - - "Whether the memory DB shares one SQLite file with affect or uses its own — default SEPARATE per plane (cleaner); revisit at the combined two-plane server (guide §7)." - - "metadata_filter richness: v1 advertises filterable_metadata_fields=[] (scope_filter only); add fields when a concrete Worldtree filter need lands." + - "Whether the memory DB shares one SQLite file with affect or its own — default SEPARATE per plane; the dev-shell entrypoint reads RATATOSKR_MEMORY_DB (analogous to RATATOSKR_AFFECT_DB). Revisit at the combined two-plane server (guide §7)." external_invariants: - source: ~/development/bifrost/bifrost/reference_server/memory.py - invariant_id: "InMemoryMemoryStore" # executable reference for the wire semantics we parity-prove against + invariant_id: "InMemoryMemoryStore" - source: ~/development/bifrost/docs/implementing-a-consumer.md invariant_id: "§5 memory plane" +revisions: + - version: "1.1" + at: 2026-06-15 + summary: "Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005." + delta: + MODIFIED: + - "INV-001 byte-equal -> semantic round-trip; named structural field keys inline" + - "INV-002 idempotency_id = (\"default\", verb, actor, key) — reconciled with STEPS" + - "INV-005 search only (scan was deferred)" + - "search: top_k in-scope, scope_filter shape, recalled_view, metadata_filter-v1 reject" + - "delete_many atomicity; get_many clarified; transaction-term clarified" --- ## Context @@ -37,65 +47,78 @@ deferred. We implement **bifrost's own** `MemoryDataStore` Protocol and hand it The boundary (ADR-0001/0002/0009): **Worldtree owns intelligence — appraisal, consolidation, trust; we own permanence.** But unlike affect (blind conduit), -memory is a **structural index**: we read the chunk's embedding vector + scope -keys + id/revision + origin/injection_source to serve search and enforce the -wire's rules. The semantic content/distillate + the inert fields (`trust_tier`, -`provenance`, `source_role`) are persisted verbatim and never interpreted. +memory is a **structural index**: we read a few fields of each chunk — +`record["embedding"]` (rank), `record["scope"]` (isolation), `record["id"]` + +revision (optimistic locking), and `origin`/`injection_source` (the consistency +rule). The semantic content, `record["distillate"]`, and inert fields +(`trust_tier`/`provenance`/`source_role`) are persisted verbatim and never +interpreted. ## Data flow - **In:** Worldtree → `POST /bifrost/memory-call` → library validates envelope + per-dispatch JWT → the verb on our store. +- **Chunk record (key fields we read; rest is opaque payload):** `id` (the chunk + id — reference falls back to `chunk_id`/`memory_id`), `embedding` (the vector — + fallback `vector`), `scope` (a `{axis: value}` dict — the isolation key), + `origin` + `injection_source` (consistency rule), `distillate` (the recall + view). Everything else (content, `metadata`, `trust_tier`, …) is stored verbatim. - **At rest:** SQLite — - - `memory_chunks(chunk_id PK, record_json, revision, agent_id, end_user_id, - scope_json, origin, invalid, superseded, ...)` — the verbatim chunk + the - extracted structural columns for scope-filtering + lifecycle. - - a **sqlite-vec** virtual table `memory_vec(chunk_id, embedding[])` — the - embedding index for cosine search. + - `memory_chunks(chunk_id PK, record_json, revision, scope_json, origin, ...)` — + the verbatim chunk + extracted columns (chunk_id, scope) for isolation. + - sqlite-vec virtual table `memory_vec(chunk_id, embedding[])` — the index. - `memory_idempotency(idempotency_id PK, digest, expires_at)` — replay/conflict - cache (same shape as the affect plane). -- **Out:** verb-specific dicts mirroring the reference: `upsert_many` → - `{"upserted": N, "replayed": bool}`; `search` → list of - `{chunk, chunk_id, score, recalled_view, revision}`; `delete_many` → - `{"deleted": N}`; `get` → record + `revision`, or `None`. + cache (affect-parallel shape). +- **Out:** `upsert_many` → `{"upserted": N, "replayed": bool}`; `search` → list of + `{chunk, chunk_id, score, recalled_view, revision}` where **`recalled_view`** = + the chunk's `distillate` field, or the whole chunk if absent (per the reference); + `delete_many` → `{"deleted": N}`; `get` → the verbatim record + a `revision` key, + or `None`; `get_many(ids)` → the list form of `get` (found records only). ## Invariants -- **INV-001** [hard]: **Persist verbatim; read only the structural surface.** The - whole chunk record is stored + round-tripped byte-equal (semantic round-trip). - The store reads ONLY: the embedding vector (search index), scope keys - (scope_filter), chunk id + revision (optimistic locking), and origin + - injection_source (the consistency rule). Content/distillate + inert fields - (`trust_tier`/`provenance`/`source_role`) are NOT interpreted. -- **INV-002** [hard]: **Idempotency = replay-or-conflict, actor-scoped** (same as - affect). `idempotency_id = (verb, _ctx_actor(ctx), idempotency_key)`. Same - digest → replay (`replayed: True`, no re-write); different digest → raise +- **INV-001** [hard]: **Persist verbatim (semantic round-trip); read only the + structural surface.** The whole chunk is stored and a read deserializes to a + Python object EQUAL to the input (`json.loads(record_json) == input`) — **not** + byte-equal (key order / formatting may differ); `get` additionally attaches a + `revision` key to the returned object. The store reads ONLY `record["embedding"]`, + `record["scope"]`, `record["id"]` + revision, and `origin`/`injection_source`. + Content / `distillate` / inert fields (`trust_tier`/`provenance`/`source_role`) + are NOT interpreted. +- **INV-002** [hard]: **Idempotency = replay-or-conflict, actor-scoped** (affect- + parallel). `idempotency_id = ("default", , _ctx_actor(ctx), idempotency_key)` + — the literal `"default"` class slot + the verb name, matching the reference + 4-tuple (`idempotency_class` tunes only the cache TTL, not the id). Same digest → + replay (`replayed: True`, no re-write); different digest → raise `IdempotencyConflict`. Actor from `ctx`, never from the record. - **INV-003** [hard]: **Optimistic locking.** When `upsert_many` carries - `expected_revisions`, each record's stored revision must equal the expected; - any mismatch → raise `RevisionMismatch` and the whole batch rolls back. Each - successful upsert increments the chunk's revision. -- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + the - idempotency record in one transaction; on any error nothing is persisted - (no partial batch, no orphaned vec rows). -- **INV-005** [hard]: **Scope isolation.** `search`/`scan` results are filtered to - records matching `scope_filter`; a search never returns another scope's chunk. -- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement, - INV-007 upstream). `describe_store` advertises ONLY what v1 implements: - `relational_edges_supported=False`, `atomic_supersede_supported=False`, - `transaction_supported=False`, `optimistic_locking_supported=True`, - `filterable_metadata_fields=[]`. The client gates the gated verbs off these. -- **INV-007** [hard]: `origin == "injected_context"` requires `injection_source`; - a non-injected record carrying `injection_source` is rejected — both raise + `expected_revisions`, each record's stored revision must equal the expected; any + mismatch → raise `RevisionMismatch` and the whole batch rolls back. Each + successful upsert increments the chunk's revision (a first insert → revision 1). +- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + their + vec rows + the idempotency record in one transaction; on any error nothing is + persisted (no partial batch, no orphaned vec rows). +- **INV-005** [hard]: **Scope isolation.** `search` results are filtered to records + whose `record["scope"]` matches every axis in `scope_filter`; a search never + returns another scope's chunk. +- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement). + `describe_store` advertises ONLY what v1 implements: `relational_edges_supported=False`, + `atomic_supersede_supported=False`, `transaction_supported=False`, + `optimistic_locking_supported=True`, `filterable_metadata_fields=[]`. + (`transaction_supported` is the bifrost **wire-level** multi-op transaction + capability — NOT our internal SQLite transactions, which we use for atomic + batches.) The client gates the gated verbs off these. +- **INV-007** [hard]: `origin == "injected_context"` requires `injection_source`; a + non-injected record carrying `injection_source` is rejected — both raise `InvalidArguments` (mirrors the reference). -- **INV-008** [hard]: The store is REQUIRED (`build_memory_app(store=None)` - raises); identity/scope/actor come from `ctx`, never call args (INV-006 affect-parallel). +- **INV-008** [hard]: The store is REQUIRED (`build_memory_app(store=None)` raises); + identity/scope/actor come from `ctx`, never call args. ## Concurrency SQLite WAL (concurrent readers, single writer). `upsert_many`/`delete_many` serialize on the writer; `search`/`get` are concurrent reads. sqlite-vec index -writes ride inside the upsert transaction. +writes ride inside the upsert/delete transaction. ## Division of labor (library vs store) @@ -109,22 +132,22 @@ negotiation, routes). **This contract** owns the store (the basic verbs + SQLite - `bifrost.consumer.build_memory_app(store, verifier, registration, maintenance_store=None, hooks=None)` → Starlette app. - `bifrost.reference_server.JwtVerifier` + `bifrost.consumer.ConsumerRegistration`. - `bifrost.memory.{StoreCapabilities, InvalidArguments, IdempotencyConflict, RevisionMismatch}` — typed surface. -- **Conformance (tests):** `bifrost.consumer.testing.InMemoryMemoryStore` + `bifrost.memory.dispatch_memory_call` (#195). -- `sqlite-vec` (the vector index extension) loaded into the SQLite connection. +- **Conformance (tests):** `bifrost.consumer.testing.InMemoryMemoryStore` + `bifrost.memory.dispatch_memory_call` (#195). The reference is the authoritative pin for exact field names + wire shapes. +- `sqlite-vec` — the vector index extension loaded into the connection. ## Constraints -- **[security]** Never log chunk content/distillate. Index the vector + scope; don't interpret semantics. -- **[compatibility]** Implement bifrost's MemoryDataStore shape exactly; raise its typed exceptions; never fork the wire. Gated verbs are simply absent + advertised unsupported. -- **[correctness]** `search` ranking is cosine over the embedding; scope isolation (INV-005) is non-negotiable. +- **[security]** Never log chunk content / `distillate`. Index the vector + scope; don't interpret semantics. +- **[compatibility]** Implement bifrost's MemoryDataStore shape exactly; raise its typed exceptions; never fork the wire. Gated verbs are absent + advertised unsupported. +- **[correctness]** `search` ranks by cosine over `record["embedding"]`; scope isolation (INV-005) is non-negotiable; `top_k` counts IN-SCOPE results (see search STEPS). ## Out of scope (deferred — do NOT flag as drift) -- **Gated/maintenance verbs:** `upsert_edges`/`get_edges_for` (relational edges), `scan`, `mark_invalid`/`mark_superseded`, `patch_many`, `atomic_supersede`, the lease/checkpoint maintenance plane. All advertised-unsupported or absent in v1. -- **metadata_filter** beyond scope (advertise `filterable_metadata_fields=[]`). +- **Gated/maintenance verbs:** `upsert_edges`/`get_edges_for`, `scan`, `mark_invalid`/`mark_superseded`, `patch_many`, `atomic_supersede`, lease/checkpoint. Absent + advertised-unsupported. +- **metadata_filter beyond scope:** advertise `filterable_metadata_fields=[]`; a non-empty `metadata_filter` is unsupported in v1 (rejected — see search PRE). - **The combined two-plane server** (guide §7) — separate memory + affect apps in v1. -- **Deployment** — runs as a dev-box background shell (`ratatoskr-memory-provider`), no systemd/infra. -- **idempotency-cache TTL pruning** — `expires_at` recorded, eviction deferred (affect-parallel INV-009). +- **Deployment** — dev-box background shell (`ratatoskr-memory-provider`), no systemd/infra. +- **idempotency-cache TTL pruning** — `expires_at` recorded, eviction deferred (affect-parallel). ```contract FN open_memory_store(db_path: str, *, embedding_dim: int) -> RatatoskrMemoryStore @@ -159,22 +182,22 @@ BRIEF: Persist chunks verbatim + index their vectors, atomically, replay-or-conf PRE: [PRE-001 hard] idempotency_key non-empty str -- else InvalidArguments PRE: [PRE-002 hard] each injected_context record has injection_source; non-injected has none -- else InvalidArguments (INV-007) POST: [POST-001 return_value] {"upserted": len(records), "replayed": False} on persist; {"...","replayed": True} on replay (INV-002) -- assert -POST: [POST-002 state_change] each chunk stored verbatim + vector indexed + revision incremented; expected_revisions enforced (INV-003) -- assert +POST: [POST-002 state_change] each chunk stored verbatim + vector indexed + revision incremented (first insert → 1); expected_revisions enforced (INV-003) -- assert POST: [POST-003 side_effect] on ANY error, nothing persisted (INV-004) -- rollback ERROR_ROUTING: InvalidArguments: { local_handling: raise on bad key / injection_source rule, flow_control: abort, state_recovery: none } IdempotencyConflict: { local_handling: raise on key-reuse-different-digest, flow_control: abort, state_recovery: none } RevisionMismatch: { local_handling: raise on stale expected_revision, flow_control: abort, state_recovery: full batch rollback } STEPS: - 1. [setup] validate idempotency_key; compute digest over {records, expected_revisions}; idempotency_id = ("default","upsert_many",_ctx_actor(ctx),key) + 1. [setup] validate idempotency_key; digest over {records, expected_revisions}; idempotency_id = ("default", "upsert_many", _ctx_actor(ctx), idempotency_key) -- matches INV-002 2. [branch] idempotency lookup: same digest → RETURN replayed; different → RAISE IdempotencyConflict 3. [sequential, flexibility=prescriptive] BEGIN; IF expected_revisions: assert each stored revision matches else RAISE RevisionMismatch - 4. [loop] FOR each record: validate origin/injection_source; UPSERT memory_chunks (record_json + extracted scope/origin cols, revision+1); UPSERT memory_vec(chunk_id, embedding) - 5. [sequential] record idempotency (digest, expires_at); COMMIT + 4. [loop] FOR each record: validate origin/injection_source; UPSERT memory_chunks (record_json + scope_json, revision+1); UPSERT memory_vec(record["id"], record["embedding"]) + 5. [sequential] record idempotency (digest, expires_at = now + ttl(idempotency_class)); COMMIT 6. [cleanup] RETURN {"upserted": len(records), "replayed": False} TESTS: basic_upsert [happy,tracer]: 2 records → {"upserted":2,"replayed":False}; get() round-trips each verbatim + revision=1 - replay [happy]: same key+payload twice → second {"replayed":True}; one revision bump, not two + replay [happy]: same key+payload twice → first writes (revision 1), second {"replayed":True} with NO further write (revision stays 1) conflict [adversarial]: same key, different records → IdempotencyConflict; first batch intact optimistic_lock [adversarial]: expected_revisions stale → RevisionMismatch; nothing written injection_rule [adversarial]: injected_context w/o injection_source → InvalidArguments; no write @@ -183,38 +206,41 @@ TESTS: ```contract FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict] -BRIEF: Vector (cosine) recall over sqlite-vec, scoped, top-k. +BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks. PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments -POST: [POST-001 return_value] returns ≤ top_k results, each {chunk, chunk_id, score, recalled_view, revision}, ranked by similarity, scope-filtered (INV-005) -- assert +PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments +POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert STEPS: - 1. [setup] validate scope_filter shape - 2. [sequential, flexibility=indicative] sqlite-vec KNN over memory_vec for the query vector, JOIN memory_chunks, FILTER by scope (INV-005), LIMIT top_k - 3. [cleanup] RETURN result rows (chunk verbatim + score + revision) + 1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis]) + 2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed) + 3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision) TESTS: - basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k - scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk (INV-005) + basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present + scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005) empty [boundary]: search empty store → [] + metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195) ``` ```contract FN get(self, chunk_id: str) -> dict | None -BRIEF: Point read; returns the verbatim chunk + current revision, or None. -POST: [POST-001 return_value] stored record (verbatim) + "revision" key, or None if absent (INV-001) -- assert +BRIEF: Point read; returns the verbatim chunk + current revision, or None. get_many(ids) is the list form (found records only). +POST: [POST-001 return_value] stored record (verbatim, json.loads) + "revision" key, or None if absent (INV-001) -- assert STEPS: 1. [sequential] SELECT record_json, revision WHERE chunk_id; RETURN json.loads + revision, or None TESTS: get_hit [happy]: after upsert → record equal + revision present get_absent [boundary]: unknown id → None + get_many [happy]: get_many([present, absent]) → [present record] only ``` ```contract FN delete_many(self, ids: list[str]) -> dict -BRIEF: Delete chunks (+ their vec rows) by id. +BRIEF: Delete chunks (+ their vec rows) by id, transactionally. POST: [POST-001 return_value] {"deleted": N} where N = ids that existed -- assert -POST: [POST-002 state_change] deleted chunks gone from memory_chunks AND memory_vec -- no orphan vec rows +POST: [POST-002 state_change] in ONE transaction, deleted chunks gone from memory_chunks AND memory_vec; partial failure rolls back the whole batch (no orphan vec rows) -- assert STEPS: - 1. [loop] FOR each id present: DELETE from memory_chunks + memory_vec; count + 1. [sequential, flexibility=prescriptive] BEGIN; FOR each id present: DELETE from memory_chunks + memory_vec; count; COMMIT 2. [cleanup] RETURN {"deleted": count} TESTS: delete_hit [happy]: delete 1 of 2 → {"deleted":1}; gone from chunks + vec; search won't surface it