vh/ratatoskr
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(provider): split memory search scope_filter → scope_all + scope_any (bifrost 0.8.0/wire v0.6)

Browse Source 
Repin bifrost 0.7.0→0.8.0 and reimplement the memory store's search scope
filter to the v0.6 split (#11): scope_all (AND/intersection) + scope_any
(OR/union over a list of conjunctive scopes), at parity with the v0.6
reference _matches_scope / _validate_scope. No-compat: scope_filter removed.

scope_any is the union-visibility primitive that resolves the #295/#297
silent-zero AND foot-gun — a subset-scoped chunk now recalls via an OR
member. End-to-end cold recall now gated only on Worldtree emitting
scope_any on its recall path (#297, upstream).

- store: search(scope_all, scope_any); _scope_subset + _matches_scope + _validate_scope
- contract v1.2: search FN sig, INV-005 recomposed, PRE-003 both fields, scope_any_union test
- tests: scope_any union, scope_all∧scope_any compose, both-empty match-all; parity vs real 0.8.0 dispatch (433 green)
- #17 contract: sync stale scope_filter/_scope_matches-AND refs to scope_all/scope_any
- runbook + persistent-memory updated; provider bounced onto 0.8.0 (fresh empty db)

v0.17.6
This commit is contained in:
vh
2026-06-16 23:09:31 -07:00
parent 43f2e148ad
commit 96d61a4bb1
8 changed files with 203 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contracts/bifrost_memory_provider.contract.md
+27 -12
View File
@@ -22,6 +22,16 @@ external_invariants:
- source: ~/development/bifrost/docs/implementing-a-consumer.md
invariant_id: "§5 memory plane"
revisions:
- version: "1.2"
at: 2026-06-16
summary: "Repin bifrost 0.7.0→0.8.0 (wire v0.5→v0.6, #11): search `scope_filter` split into `scope_all` (AND/intersection) + `scope_any` (OR/union over a list of conjunctive scopes). No-compat: `scope_filter` removed. Adds union-visibility recall in one call — the fix for the #295/#297 AND silent-zero foot-gun. Store at parity with the v0.6 reference `_matches_scope` / `_validate_scope`."
delta:
MODIFIED:
- "search signature: scope_filter -> scope_all + scope_any"
- "INV-005 scope isolation -> composed v0.6 (scope_all AND ∧ scope_any OR-union)"
- "PRE-003 validates axes in BOTH fields; STEP 1 = _validate_scope (shape + lattice)"
ADDED:
- "scope_any_union test (#295/#297 union capability); both-fields-empty match-all"
- version: "1.1"
at: 2026-06-15
summary: "Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005."
@@ -98,9 +108,13 @@ interpreted.
- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + their
vec rows + the idempotency record in one transaction; on any error nothing is
persisted (no partial batch, no orphaned vec rows).
- **INV-005** [hard]: **Scope isolation.** `search` results are filtered to records
whose `record["scope"]` matches every axis in `scope_filter`; a search never
returns another scope's chunk.
- **INV-005** [hard]: **Scope isolation (wire v0.6, #11).** `search` filters by two
explicit fields: `scope_all` (AND/intersection — record ⊇ every named axis) and
`scope_any` (OR/union over a LIST of conjunctive scope dicts — record ⊇ ≥1 element,
each element AND-matched as a whole). They compose by AND; both empty → no scope
constraint. A search never returns a chunk outside the composed filter. Byte-faithful
to the reference `_matches_scope`. (`scope_any` is the union-visibility primitive that
resolves the #295/#297 silent-zero — a subset-scoped chunk now recalls via an OR member.)
- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement).
`describe_store` advertises ONLY what v1 implements: `relational_edges_supported=False`,
`atomic_supersede_supported=False`, `transaction_supported=False`,
@@ -205,22 +219,23 @@ TESTS:
```
```contract
FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
FN search(self, vector: list[float], *, top_k: int, scope_all: dict | None = None, scope_any: list | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped by the v0.6 scope_all/scope_any filter, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
PRE: [PRE-003 hard] every scope_filter axis ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.5 lattice, matching the reference _validate_scope_filter (#10 made agent_self canonical)
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
PRE: [PRE-003 hard] scope_all is a flat dict and scope_any a list of flat dicts (else InvalidArguments); every axis in BOTH ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.6 lattice, matching the reference _validate_scope (#10 agent_self canonical, #11 scope split)
POST: [POST-001 return_value] returns the top_k highest-cosine records passing the composed v0.6 filter — `(scope_all empty OR record ⊇ scope_all) AND (scope_any empty OR record ⊇ ≥1 element)`; at most top_k, never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS:
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis]) AND every axis ∈ the v0.5 lattice {end_user, group, tenant, agent_self} (else InvalidFilter)
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
1. [setup] scope_all ← scope_all or {}; scope_any ← scope_any or []; validate via _validate_scope (flat-dict / list-of-dicts shape + every axis ∈ the v0.6 lattice, else InvalidArguments / InvalidFilter)
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only records passing _matches_scope(scope_all, scope_any) (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS:
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
empty [boundary]: search empty store → []
scope_isolation [adversarial]: two scopes, scope_all one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
scope_any_union [scenario]: scope_any=[{end_user:u},{agent_self:a}] recalls BOTH a subject-scoped and a self-scoped chunk in one call (#295/#297 union capability); scope_all+scope_any compose by AND
empty [boundary]: search empty store → []; both fields empty → match all
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
lattice_axes [adversarial]: out-of-lattice scope axis → InvalidFilter; agent_self admitted (wire v0.5, #10)
lattice_axes [adversarial]: out-of-lattice axis in scope_all OR scope_any → InvalidFilter; non-list scope_any → InvalidArguments; agent_self admitted (wire v0.5, #10)
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
```
docs/contracts/issues/17.contract.md
+7 -7
View File
@@ -21,7 +21,7 @@ assumptions:
- "PROVEN: the session-create BEARER is the identity Worldtree signs the Bifrost handshake JWT with (HS256 shared-secret). Bearer = the canary key (WORLDTREE_API_KEY) → handshake 401 `bifrost.auth_rejected` → 502 to the client. Bearer = the consumer Heimdall key (== the provider's RATATOSKR_HEIMDALL_KEY string) → handshake 200. So a BOUND session-create MUST authenticate with the consumer key, NOT the canary key. These are two distinct ratatoskr identities."
- "PROVEN: dev HTTP is accepted (spec wants HTTPS) because the provider host:port is on Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist — a Worldtree-side, infra-ops-owned config. The endpoint_url must be the WORLDTREE-VISIBLE base URL (e.g. `http://10.100.10.50:8391`), not the client's loopback. Provider routes live at `/bifrost/handshake` + `/bifrost/memory-call` (memory) and `/bifrost/affect-call` (affect) under that base."
- "PROVEN (bifrost source, memory.py:244 vs 262): `dispatch_memory_call` passes `ctx` to `upsert_many` but NOT to `search`/`get`/`delete`. So the recall verb's store method has no session_id; correlation identity must be captured at the dispatch/ASGI layer (JWT ctx), not inside the store method. turn_id (finer than session_id) availability is UNVERIFIED — a contract-stage JWT-claims/envelope dump resolves it; design the op-feed to accept a turn_id later without a schema break."
- "Provider stores MUST NOT change scope semantics. `_scope_matches` stays AND (byte-faithful to bifrost reference `reference_server/memory.py:398`); the missing axis-lattice validation (my store admits a wider axis set than the reference) is a SEPARATE parity item, GATED on the `agent_self`-canonicity resolution, and OUT OF SCOPE for #17."
- "Provider stores MUST NOT change scope semantics. `_matches_scope` stays the v0.6 composed filter — `scope_all` (AND/intersection) ∧ `scope_any` (OR/union over conjunctive scopes) — byte-faithful to bifrost reference `reference_server/memory.py` (wire v0.6, #11); the 4-axis lattice validation (`_validate_scope`) is in place and at parity. Scope semantics are settled (the v0.6 scope split shipped, bifrost 0.8.0) and OUT OF SCOPE for #17 — observe is read-only over them."
- "The existing `create_session(client, agent_id, *, end_user_id=None)` (sessions.py:179) is extended, not replaced (pre-v1, no compat shim). The httpx client carries the canary key as its default Authorization; the bound create overrides the bearer per-request with the consumer key."
- "Tests use `respx` for the bind unit tests (mirroring tests/test_sessions.py) + the in-process op-feed; the live-smoke acceptance is manual (per the repo's load-bearing-smoke posture), captured as a documented runbook step, not a unit test. `docs/bifrost-self-test.md` is the manual procedure this feature productizes."
- "v1 ships the CLI + TUI + web trigger surfaces in lockstep (the repo's BOTH-presenters-in-lockstep rule); the op-feed is read by the debug surface as structured lines for now (pane-correlated rendering is PARKED)."
@@ -151,9 +151,9 @@ class BifrostConsumerKeyMissing(Exception):
one plane. v1 documents this limit explicitly; binding both planes for one turn
is the PARKED composite-endpoint feature, not a v1 path.
- **INV-004 (no scope-semantics change).** The observe instrumentation is
READ-ONLY over the dispatch path; it MUST NOT alter `_scope_matches`, the
AND-parity, or any store behavior. The op-feed reports the effective scope used
per op; it never rewrites scope client-side.
READ-ONLY over the dispatch path; it MUST NOT alter `_matches_scope`, the v0.6
`scope_all`/`scope_any` semantics, or any store behavior. The op-feed reports the
effective scope used per op; it never rewrites scope client-side.
- **INV-005 (correlation key at the dispatch layer).** session_id is captured from
the JWT ctx at the dispatch/ASGI layer — present for ALL JWT-carrying verbs,
INCLUDING search/get/delete (bifrost withholds ctx from those STORE methods, but
@@ -239,7 +239,7 @@ POST: [POST-004 return_value] store scope semantics untouched; read-only over di
STEPS:
1. wrap the dispatch/ASGI layer so each inbound bifrost-call yields one OpEvent
2. read session_id off the JWT ctx if present; else None
3. summarise req (scope_filter/top_k for search; record-count+scopes for upsert) + resp (hit-count+ids/scores | upserted+replayed | error code) — NEVER verbatim content
3. summarise req (scope_all/scope_any/top_k for search; record-count+scopes for upsert) + resp (hit-count+ids/scores | upserted+replayed | error code) — NEVER verbatim content
4. emit to sink; swallow sink errors
```
@@ -279,7 +279,7 @@ let an implementer comply while violating intent.
- **Op-feed summary shapes (per verb).** `req_summary` / `resp_summary` are
scope-only, never verbatim content:
- `search` → req `{scope_filter, top_k}`; resp `{hit_count, [{chunk_id, score}]}`
- `search` → req `{scope_all, scope_any, top_k}`; resp `{hit_count, [{chunk_id, score}]}`
- `upsert_many` → req `{record_count, [scope]}`; resp `{upserted, replayed}`
- `get` / `get_many` → req `{ids}`; resp `{found_count}`
- `delete_many` → req `{ids}`; resp `{deleted}`
@@ -309,4 +309,4 @@ let an implementer comply while violating intent.
- Turn-correlated debug-pane UI — needs turn_id (open question) + grace-window buffering + client read channel.
- Key identity unification (Heimdall) — operator's call, crosses service boundary.
- Auto-bind on Tier-3 agents — operator's call; v1 is explicit opt-in only.
- Provider axis-lattice validation (the agent_self parity item) — gated on the `agent_self`-canonicity cross-repo resolution; tracked separately, NOT #17.
- Provider axis-lattice validation + the v0.6 scope split — RESOLVED (shipped: bifrost 0.8.0/wire v0.6, `_validate_scope` 4-axis + `scope_all`/`scope_any`); no longer a #17 concern.