docs(provider): memory contract v1.1 — heid-contract-review fixup

Panel review (Gróa/Hulda/Regin) → 11 spec-tightening fixes, no design change:
- INV-001 byte-equal → semantic round-trip (the slip that rode the affect copy-paste)
- search returns top_k IN-SCOPE results (filter-then-limit) — Regin's correctness catch
- idempotency_id reconciled to the reference's ("default", verb, actor, key)
- inline the reference's field keys (id/embedding/scope/distillate) + recalled_view + scope_filter shape
- drop scan from INV-005; clarify metadata_filter-v1 reject, transaction-term, delete atomicity, get_many, revision-on-replay
- revisions: marker records the v1.1 delta
This commit is contained in:
2026-06-15 01:21:24 -07:00
parent eebab46812
commit 1f94e5f251
@@ -8,20 +8,30 @@ touches:
language: "python"
complexity: "high"
estimated_loc: 320
confidence: 0.78
confidence: 0.82
assumptions:
- "bifrost>=0.6.1 exposes build_memory_app, dispatch_memory_call, JwtVerifier, ConsumerRegistration, StoreCapabilities, MemoryDataStore, InvalidArguments, IdempotencyConflict, RevisionMismatch per bifrost/reference_server/memory.py + bifrost.memory."
- "v1 = worldtree-dev's BASIC PLANE only (search / get / upsert / delete + describe_store + health), which is the ONLY surface Tier-3's live path touches (#294). Worldtree v0.35.3 already requests + maps it — no Worldtree-side blocker."
- "Exact chunk-record field names (embedding vector key, scope keys, id key) are pinned in TDD against InMemoryMemoryStore — the executable spec — as they were for affect."
- "v1 = worldtree-dev's BASIC PLANE only (search / get / get_many / upsert_many / delete_many + describe_store + health), the ONLY surface Tier-3's live path touches (#294); Worldtree v0.35.3 already negotiates it."
- "Chunk record field names are taken from the reference store (named inline below) but the AUTHORITATIVE pin is TDD against InMemoryMemoryStore, as it was for affect."
- "Embedding dimension matches Worldtree's PINNED_EMBEDDER_DIM, supplied as config (env RATATOSKR_MEMORY_EMBEDDING_DIM); the sqlite-vec virtual table is created at that fixed dim."
open_questions:
- "Whether the memory DB shares one SQLite file with affect or uses its own — default SEPARATE per plane (cleaner); revisit at the combined two-plane server (guide §7)."
- "metadata_filter richness: v1 advertises filterable_metadata_fields=[] (scope_filter only); add fields when a concrete Worldtree filter need lands."
- "Whether the memory DB shares one SQLite file with affect or its own — default SEPARATE per plane; the dev-shell entrypoint reads RATATOSKR_MEMORY_DB (analogous to RATATOSKR_AFFECT_DB). Revisit at the combined two-plane server (guide §7)."
external_invariants:
- source: ~/development/bifrost/bifrost/reference_server/memory.py
invariant_id: "InMemoryMemoryStore" # executable reference for the wire semantics we parity-prove against
invariant_id: "InMemoryMemoryStore"
- source: ~/development/bifrost/docs/implementing-a-consumer.md
invariant_id: "§5 memory plane"
revisions:
- version: "1.1"
at: 2026-06-15
summary: "Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005."
delta:
MODIFIED:
- "INV-001 byte-equal -> semantic round-trip; named structural field keys inline"
- "INV-002 idempotency_id = (\"default\", verb, actor, key) — reconciled with STEPS"
- "INV-005 search only (scan was deferred)"
- "search: top_k in-scope, scope_filter shape, recalled_view, metadata_filter-v1 reject"
- "delete_many atomicity; get_many clarified; transaction-term clarified"
---
## Context
@@ -37,65 +47,78 @@ deferred. We implement **bifrost's own** `MemoryDataStore` Protocol and hand it
The boundary (ADR-0001/0002/0009): **Worldtree owns intelligence — appraisal,
consolidation, trust; we own permanence.** But unlike affect (blind conduit),
memory is a **structural index**: we read the chunk's embedding vector + scope
keys + id/revision + origin/injection_source to serve search and enforce the
wire's rules. The semantic content/distillate + the inert fields (`trust_tier`,
`provenance`, `source_role`) are persisted verbatim and never interpreted.
memory is a **structural index**: we read a few fields of each chunk —
`record["embedding"]` (rank), `record["scope"]` (isolation), `record["id"]` +
revision (optimistic locking), and `origin`/`injection_source` (the consistency
rule). The semantic content, `record["distillate"]`, and inert fields
(`trust_tier`/`provenance`/`source_role`) are persisted verbatim and never
interpreted.
## Data flow
- **In:** Worldtree → `POST /bifrost/memory-call` → library validates envelope +
per-dispatch JWT → the verb on our store.
- **Chunk record (key fields we read; rest is opaque payload):** `id` (the chunk
id — reference falls back to `chunk_id`/`memory_id`), `embedding` (the vector —
fallback `vector`), `scope` (a `{axis: value}` dict — the isolation key),
`origin` + `injection_source` (consistency rule), `distillate` (the recall
view). Everything else (content, `metadata`, `trust_tier`, …) is stored verbatim.
- **At rest:** SQLite —
- `memory_chunks(chunk_id PK, record_json, revision, agent_id, end_user_id,
scope_json, origin, invalid, superseded, ...)` — the verbatim chunk + the
extracted structural columns for scope-filtering + lifecycle.
- a **sqlite-vec** virtual table `memory_vec(chunk_id, embedding[<dim>])` — the
embedding index for cosine search.
- `memory_chunks(chunk_id PK, record_json, revision, scope_json, origin, ...)`
the verbatim chunk + extracted columns (chunk_id, scope) for isolation.
- sqlite-vec virtual table `memory_vec(chunk_id, embedding[<dim>])` — the index.
- `memory_idempotency(idempotency_id PK, digest, expires_at)` — replay/conflict
cache (same shape as the affect plane).
- **Out:** verb-specific dicts mirroring the reference: `upsert_many` →
`{"upserted": N, "replayed": bool}`; `search` → list of
`{chunk, chunk_id, score, recalled_view, revision}`; `delete_many` →
`{"deleted": N}`; `get` → record + `revision`, or `None`.
cache (affect-parallel shape).
- **Out:** `upsert_many``{"upserted": N, "replayed": bool}`; `search` → list of
`{chunk, chunk_id, score, recalled_view, revision}` where **`recalled_view`** =
the chunk's `distillate` field, or the whole chunk if absent (per the reference);
`delete_many``{"deleted": N}`; `get` the verbatim record + a `revision` key,
or `None`; `get_many(ids)` → the list form of `get` (found records only).
## Invariants
- **INV-001** [hard]: **Persist verbatim; read only the structural surface.** The
whole chunk record is stored + round-tripped byte-equal (semantic round-trip).
The store reads ONLY: the embedding vector (search index), scope keys
(scope_filter), chunk id + revision (optimistic locking), and origin +
injection_source (the consistency rule). Content/distillate + inert fields
(`trust_tier`/`provenance`/`source_role`) are NOT interpreted.
- **INV-002** [hard]: **Idempotency = replay-or-conflict, actor-scoped** (same as
affect). `idempotency_id = (verb, _ctx_actor(ctx), idempotency_key)`. Same
digest → replay (`replayed: True`, no re-write); different digest → raise
- **INV-001** [hard]: **Persist verbatim (semantic round-trip); read only the
structural surface.** The whole chunk is stored and a read deserializes to a
Python object EQUAL to the input (`json.loads(record_json) == input`) — **not**
byte-equal (key order / formatting may differ); `get` additionally attaches a
`revision` key to the returned object. The store reads ONLY `record["embedding"]`,
`record["scope"]`, `record["id"]` + revision, and `origin`/`injection_source`.
Content / `distillate` / inert fields (`trust_tier`/`provenance`/`source_role`)
are NOT interpreted.
- **INV-002** [hard]: **Idempotency = replay-or-conflict, actor-scoped** (affect-
parallel). `idempotency_id = ("default", <verb>, _ctx_actor(ctx), idempotency_key)`
— the literal `"default"` class slot + the verb name, matching the reference
4-tuple (`idempotency_class` tunes only the cache TTL, not the id). Same digest →
replay (`replayed: True`, no re-write); different digest → raise
`IdempotencyConflict`. Actor from `ctx`, never from the record.
- **INV-003** [hard]: **Optimistic locking.** When `upsert_many` carries
`expected_revisions`, each record's stored revision must equal the expected;
any mismatch → raise `RevisionMismatch` and the whole batch rolls back. Each
successful upsert increments the chunk's revision.
- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + the
idempotency record in one transaction; on any error nothing is persisted
(no partial batch, no orphaned vec rows).
- **INV-005** [hard]: **Scope isolation.** `search`/`scan` results are filtered to
records matching `scope_filter`; a search never returns another scope's chunk.
- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement,
INV-007 upstream). `describe_store` advertises ONLY what v1 implements:
`relational_edges_supported=False`, `atomic_supersede_supported=False`,
`transaction_supported=False`, `optimistic_locking_supported=True`,
`filterable_metadata_fields=[]`. The client gates the gated verbs off these.
- **INV-007** [hard]: `origin == "injected_context"` requires `injection_source`;
a non-injected record carrying `injection_source` is rejected — both raise
`expected_revisions`, each record's stored revision must equal the expected; any
mismatch → raise `RevisionMismatch` and the whole batch rolls back. Each
successful upsert increments the chunk's revision (a first insert → revision 1).
- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + their
vec rows + the idempotency record in one transaction; on any error nothing is
persisted (no partial batch, no orphaned vec rows).
- **INV-005** [hard]: **Scope isolation.** `search` results are filtered to records
whose `record["scope"]` matches every axis in `scope_filter`; a search never
returns another scope's chunk.
- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement).
`describe_store` advertises ONLY what v1 implements: `relational_edges_supported=False`,
`atomic_supersede_supported=False`, `transaction_supported=False`,
`optimistic_locking_supported=True`, `filterable_metadata_fields=[]`.
(`transaction_supported` is the bifrost **wire-level** multi-op transaction
capability — NOT our internal SQLite transactions, which we use for atomic
batches.) The client gates the gated verbs off these.
- **INV-007** [hard]: `origin == "injected_context"` requires `injection_source`; a
non-injected record carrying `injection_source` is rejected — both raise
`InvalidArguments` (mirrors the reference).
- **INV-008** [hard]: The store is REQUIRED (`build_memory_app(store=None)`
raises); identity/scope/actor come from `ctx`, never call args (INV-006 affect-parallel).
- **INV-008** [hard]: The store is REQUIRED (`build_memory_app(store=None)` raises);
identity/scope/actor come from `ctx`, never call args.
## Concurrency
SQLite WAL (concurrent readers, single writer). `upsert_many`/`delete_many`
serialize on the writer; `search`/`get` are concurrent reads. sqlite-vec index
writes ride inside the upsert transaction.
writes ride inside the upsert/delete transaction.
## Division of labor (library vs store)
@@ -109,22 +132,22 @@ negotiation, routes). **This contract** owns the store (the basic verbs + SQLite
- `bifrost.consumer.build_memory_app(store, verifier, registration, maintenance_store=None, hooks=None)` → Starlette app.
- `bifrost.reference_server.JwtVerifier` + `bifrost.consumer.ConsumerRegistration`.
- `bifrost.memory.{StoreCapabilities, InvalidArguments, IdempotencyConflict, RevisionMismatch}` — typed surface.
- **Conformance (tests):** `bifrost.consumer.testing.InMemoryMemoryStore` + `bifrost.memory.dispatch_memory_call` (#195).
- `sqlite-vec` (the vector index extension) loaded into the SQLite connection.
- **Conformance (tests):** `bifrost.consumer.testing.InMemoryMemoryStore` + `bifrost.memory.dispatch_memory_call` (#195). The reference is the authoritative pin for exact field names + wire shapes.
- `sqlite-vec`the vector index extension loaded into the connection.
## Constraints
- **[security]** Never log chunk content/distillate. Index the vector + scope; don't interpret semantics.
- **[compatibility]** Implement bifrost's MemoryDataStore shape exactly; raise its typed exceptions; never fork the wire. Gated verbs are simply absent + advertised unsupported.
- **[correctness]** `search` ranking is cosine over the embedding; scope isolation (INV-005) is non-negotiable.
- **[security]** Never log chunk content / `distillate`. Index the vector + scope; don't interpret semantics.
- **[compatibility]** Implement bifrost's MemoryDataStore shape exactly; raise its typed exceptions; never fork the wire. Gated verbs are absent + advertised unsupported.
- **[correctness]** `search` ranks by cosine over `record["embedding"]`; scope isolation (INV-005) is non-negotiable; `top_k` counts IN-SCOPE results (see search STEPS).
## Out of scope (deferred — do NOT flag as drift)
- **Gated/maintenance verbs:** `upsert_edges`/`get_edges_for` (relational edges), `scan`, `mark_invalid`/`mark_superseded`, `patch_many`, `atomic_supersede`, the lease/checkpoint maintenance plane. All advertised-unsupported or absent in v1.
- **metadata_filter** beyond scope (advertise `filterable_metadata_fields=[]`).
- **Gated/maintenance verbs:** `upsert_edges`/`get_edges_for`, `scan`, `mark_invalid`/`mark_superseded`, `patch_many`, `atomic_supersede`, lease/checkpoint. Absent + advertised-unsupported.
- **metadata_filter beyond scope:** advertise `filterable_metadata_fields=[]`; a non-empty `metadata_filter` is unsupported in v1 (rejected — see search PRE).
- **The combined two-plane server** (guide §7) — separate memory + affect apps in v1.
- **Deployment** — runs as a dev-box background shell (`ratatoskr-memory-provider`), no systemd/infra.
- **idempotency-cache TTL pruning** — `expires_at` recorded, eviction deferred (affect-parallel INV-009).
- **Deployment** — dev-box background shell (`ratatoskr-memory-provider`), no systemd/infra.
- **idempotency-cache TTL pruning** — `expires_at` recorded, eviction deferred (affect-parallel).
```contract
FN open_memory_store(db_path: str, *, embedding_dim: int) -> RatatoskrMemoryStore
@@ -159,22 +182,22 @@ BRIEF: Persist chunks verbatim + index their vectors, atomically, replay-or-conf
PRE: [PRE-001 hard] idempotency_key non-empty str -- else InvalidArguments
PRE: [PRE-002 hard] each injected_context record has injection_source; non-injected has none -- else InvalidArguments (INV-007)
POST: [POST-001 return_value] {"upserted": len(records), "replayed": False} on persist; {"...","replayed": True} on replay (INV-002) -- assert
POST: [POST-002 state_change] each chunk stored verbatim + vector indexed + revision incremented; expected_revisions enforced (INV-003) -- assert
POST: [POST-002 state_change] each chunk stored verbatim + vector indexed + revision incremented (first insert → 1); expected_revisions enforced (INV-003) -- assert
POST: [POST-003 side_effect] on ANY error, nothing persisted (INV-004) -- rollback
ERROR_ROUTING:
InvalidArguments: { local_handling: raise on bad key / injection_source rule, flow_control: abort, state_recovery: none }
IdempotencyConflict: { local_handling: raise on key-reuse-different-digest, flow_control: abort, state_recovery: none }
RevisionMismatch: { local_handling: raise on stale expected_revision, flow_control: abort, state_recovery: full batch rollback }
STEPS:
1. [setup] validate idempotency_key; compute digest over {records, expected_revisions}; idempotency_id = ("default","upsert_many",_ctx_actor(ctx),key)
1. [setup] validate idempotency_key; digest over {records, expected_revisions}; idempotency_id = ("default", "upsert_many", _ctx_actor(ctx), idempotency_key) -- matches INV-002
2. [branch] idempotency lookup: same digest → RETURN replayed; different → RAISE IdempotencyConflict
3. [sequential, flexibility=prescriptive] BEGIN; IF expected_revisions: assert each stored revision matches else RAISE RevisionMismatch
4. [loop] FOR each record: validate origin/injection_source; UPSERT memory_chunks (record_json + extracted scope/origin cols, revision+1); UPSERT memory_vec(chunk_id, embedding)
5. [sequential] record idempotency (digest, expires_at); COMMIT
4. [loop] FOR each record: validate origin/injection_source; UPSERT memory_chunks (record_json + scope_json, revision+1); UPSERT memory_vec(record["id"], record["embedding"])
5. [sequential] record idempotency (digest, expires_at = now + ttl(idempotency_class)); COMMIT
6. [cleanup] RETURN {"upserted": len(records), "replayed": False}
TESTS:
basic_upsert [happy,tracer]: 2 records → {"upserted":2,"replayed":False}; get() round-trips each verbatim + revision=1
replay [happy]: same key+payload twice → second {"replayed":True}; one revision bump, not two
replay [happy]: same key+payload twice → first writes (revision 1), second {"replayed":True} with NO further write (revision stays 1)
conflict [adversarial]: same key, different records → IdempotencyConflict; first batch intact
optimistic_lock [adversarial]: expected_revisions stale → RevisionMismatch; nothing written
injection_rule [adversarial]: injected_context w/o injection_source → InvalidArguments; no write
@@ -183,38 +206,41 @@ TESTS:
```contract
FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, top-k.
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
POST: [POST-001 return_value] returns ≤ top_k results, each {chunk, chunk_id, score, recalled_view, revision}, ranked by similarity, scope-filtered (INV-005) -- assert
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS:
1. [setup] validate scope_filter shape
2. [sequential, flexibility=indicative] sqlite-vec KNN over memory_vec for the query vector, JOIN memory_chunks, FILTER by scope (INV-005), LIMIT top_k
3. [cleanup] RETURN result rows (chunk verbatim + score + revision)
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis])
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS:
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk (INV-005)
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
empty [boundary]: search empty store → []
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
```
```contract
FN get(self, chunk_id: str) -> dict | None
BRIEF: Point read; returns the verbatim chunk + current revision, or None.
POST: [POST-001 return_value] stored record (verbatim) + "revision" key, or None if absent (INV-001) -- assert
BRIEF: Point read; returns the verbatim chunk + current revision, or None. get_many(ids) is the list form (found records only).
POST: [POST-001 return_value] stored record (verbatim, json.loads) + "revision" key, or None if absent (INV-001) -- assert
STEPS:
1. [sequential] SELECT record_json, revision WHERE chunk_id; RETURN json.loads + revision, or None
TESTS:
get_hit [happy]: after upsert → record equal + revision present
get_absent [boundary]: unknown id → None
get_many [happy]: get_many([present, absent]) → [present record] only
```
```contract
FN delete_many(self, ids: list[str]) -> dict
BRIEF: Delete chunks (+ their vec rows) by id.
BRIEF: Delete chunks (+ their vec rows) by id, transactionally.
POST: [POST-001 return_value] {"deleted": N} where N = ids that existed -- assert
POST: [POST-002 state_change] deleted chunks gone from memory_chunks AND memory_vec -- no orphan vec rows
POST: [POST-002 state_change] in ONE transaction, deleted chunks gone from memory_chunks AND memory_vec; partial failure rolls back the whole batch (no orphan vec rows) -- assert
STEPS:
1. [loop] FOR each id present: DELETE from memory_chunks + memory_vec; count
1. [sequential, flexibility=prescriptive] BEGIN; FOR each id present: DELETE from memory_chunks + memory_vec; count; COMMIT
2. [cleanup] RETURN {"deleted": count}
TESTS:
delete_hit [happy]: delete 1 of 2 → {"deleted":1}; gone from chunks + vec; search won't surface it