feat(provider): split memory search scope_filter → scope_all + scope_any (bifrost 0.8.0/wire v0.6)

Repin bifrost 0.7.0→0.8.0 and reimplement the memory store's search scope
filter to the v0.6 split (#11): scope_all (AND/intersection) + scope_any
(OR/union over a list of conjunctive scopes), at parity with the v0.6
reference _matches_scope / _validate_scope. No-compat: scope_filter removed.

scope_any is the union-visibility primitive that resolves the #295/#297
silent-zero AND foot-gun — a subset-scoped chunk now recalls via an OR
member. End-to-end cold recall now gated only on Worldtree emitting
scope_any on its recall path (#297, upstream).

- store: search(scope_all, scope_any); _scope_subset + _matches_scope + _validate_scope
- contract v1.2: search FN sig, INV-005 recomposed, PRE-003 both fields, scope_any_union test
- tests: scope_any union, scope_all∧scope_any compose, both-empty match-all; parity vs real 0.8.0 dispatch (433 green)
- #17 contract: sync stale scope_filter/_scope_matches-AND refs to scope_all/scope_any
- runbook + persistent-memory updated; provider bounced onto 0.8.0 (fresh empty db)

v0.17.6
This commit is contained in:
vh
2026-06-16 23:09:31 -07:00
parent 43f2e148ad
commit 96d61a4bb1
8 changed files with 203 additions and 63 deletions
@@ -22,6 +22,16 @@ external_invariants:
- source: ~/development/bifrost/docs/implementing-a-consumer.md
invariant_id: "§5 memory plane"
revisions:
- version: "1.2"
at: 2026-06-16
summary: "Repin bifrost 0.7.0→0.8.0 (wire v0.5→v0.6, #11): search `scope_filter` split into `scope_all` (AND/intersection) + `scope_any` (OR/union over a list of conjunctive scopes). No-compat: `scope_filter` removed. Adds union-visibility recall in one call — the fix for the #295/#297 AND silent-zero foot-gun. Store at parity with the v0.6 reference `_matches_scope` / `_validate_scope`."
delta:
MODIFIED:
- "search signature: scope_filter -> scope_all + scope_any"
- "INV-005 scope isolation -> composed v0.6 (scope_all AND ∧ scope_any OR-union)"
- "PRE-003 validates axes in BOTH fields; STEP 1 = _validate_scope (shape + lattice)"
ADDED:
- "scope_any_union test (#295/#297 union capability); both-fields-empty match-all"
- version: "1.1"
at: 2026-06-15
summary: "Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005."
@@ -98,9 +108,13 @@ interpreted.
- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + their
vec rows + the idempotency record in one transaction; on any error nothing is
persisted (no partial batch, no orphaned vec rows).
- **INV-005** [hard]: **Scope isolation.** `search` results are filtered to records
whose `record["scope"]` matches every axis in `scope_filter`; a search never
returns another scope's chunk.
- **INV-005** [hard]: **Scope isolation (wire v0.6, #11).** `search` filters by two
explicit fields: `scope_all` (AND/intersection — record ⊇ every named axis) and
`scope_any` (OR/union over a LIST of conjunctive scope dicts — record ⊇ ≥1 element,
each element AND-matched as a whole). They compose by AND; both empty → no scope
constraint. A search never returns a chunk outside the composed filter. Byte-faithful
to the reference `_matches_scope`. (`scope_any` is the union-visibility primitive that
resolves the #295/#297 silent-zero — a subset-scoped chunk now recalls via an OR member.)
- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement).
`describe_store` advertises ONLY what v1 implements: `relational_edges_supported=False`,
`atomic_supersede_supported=False`, `transaction_supported=False`,
@@ -205,22 +219,23 @@ TESTS:
```
```contract
FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
FN search(self, vector: list[float], *, top_k: int, scope_all: dict | None = None, scope_any: list | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped by the v0.6 scope_all/scope_any filter, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
PRE: [PRE-003 hard] every scope_filter axis ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.5 lattice, matching the reference _validate_scope_filter (#10 made agent_self canonical)
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
PRE: [PRE-003 hard] scope_all is a flat dict and scope_any a list of flat dicts (else InvalidArguments); every axis in BOTH ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.6 lattice, matching the reference _validate_scope (#10 agent_self canonical, #11 scope split)
POST: [POST-001 return_value] returns the top_k highest-cosine records passing the composed v0.6 filter — `(scope_all empty OR record ⊇ scope_all) AND (scope_any empty OR record ⊇ ≥1 element)`; at most top_k, never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS:
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis]) AND every axis ∈ the v0.5 lattice {end_user, group, tenant, agent_self} (else InvalidFilter)
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
1. [setup] scope_all ← scope_all or {}; scope_any ← scope_any or []; validate via _validate_scope (flat-dict / list-of-dicts shape + every axis ∈ the v0.6 lattice, else InvalidArguments / InvalidFilter)
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only records passing _matches_scope(scope_all, scope_any) (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS:
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
empty [boundary]: search empty store → []
scope_isolation [adversarial]: two scopes, scope_all one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
scope_any_union [scenario]: scope_any=[{end_user:u},{agent_self:a}] recalls BOTH a subject-scoped and a self-scoped chunk in one call (#295/#297 union capability); scope_all+scope_any compose by AND
empty [boundary]: search empty store → []; both fields empty → match all
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
lattice_axes [adversarial]: out-of-lattice scope axis → InvalidFilter; agent_self admitted (wire v0.5, #10)
lattice_axes [adversarial]: out-of-lattice axis in scope_all OR scope_any → InvalidFilter; non-list scope_any → InvalidArguments; agent_self admitted (wire v0.5, #10)
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
```