feat(provider): split memory search scope_filter → scope_all + scope_any (bifrost 0.8.0/wire v0.6)
Repin bifrost 0.7.0→0.8.0 and reimplement the memory store's search scope filter to the v0.6 split (#11): scope_all (AND/intersection) + scope_any (OR/union over a list of conjunctive scopes), at parity with the v0.6 reference _matches_scope / _validate_scope. No-compat: scope_filter removed. scope_any is the union-visibility primitive that resolves the #295/#297 silent-zero AND foot-gun — a subset-scoped chunk now recalls via an OR member. End-to-end cold recall now gated only on Worldtree emitting scope_any on its recall path (#297, upstream). - store: search(scope_all, scope_any); _scope_subset + _matches_scope + _validate_scope - contract v1.2: search FN sig, INV-005 recomposed, PRE-003 both fields, scope_any_union test - tests: scope_any union, scope_all∧scope_any compose, both-empty match-all; parity vs real 0.8.0 dispatch (433 green) - #17 contract: sync stale scope_filter/_scope_matches-AND refs to scope_all/scope_any - runbook + persistent-memory updated; provider bounced onto 0.8.0 (fresh empty db) v0.17.6
This commit is contained in:
@@ -107,7 +107,7 @@ PY
|
||||
For a background-shell provider, that is the task output file; tail it:
|
||||
|
||||
```
|
||||
[memory-provider] memory-call search REQUEST: scope_filter={...} top_k=... vec_dim=1024
|
||||
[memory-provider] memory-call search REQUEST: scope_all={...} scope_any=[...] top_k=... vec_dim=1024
|
||||
[memory-provider] memory-call search RESPONSE: N hit(s) [{'chunk_id':..., 'score':..., 'scope':...}]
|
||||
```
|
||||
|
||||
@@ -117,14 +117,15 @@ delete — promoted failure-surfaces may be wanted corpus.
|
||||
|
||||
## Reading the result
|
||||
|
||||
The `search REQUEST` `scope_filter` vs the `search RESPONSE` hit count is the
|
||||
whole diagnosis surface:
|
||||
The `search REQUEST` `scope_all`/`scope_any` vs the `search RESPONSE` hit count is
|
||||
the whole diagnosis surface:
|
||||
|
||||
- **0 hits** → the filter didn't match any stored chunk. Compare the filter's
|
||||
axes against the stored `scope`. INV-005 requires **every** filter axis to
|
||||
match, so an extra axis on the filter that the chunks don't carry (e.g.
|
||||
`agent_self`) zeroes the result even when `end_user` matches. That's a
|
||||
**scope-build / persist-recall-symmetry** question (Worldtree-side).
|
||||
- **0 hits** → the composed v0.6 filter matched no stored chunk. `scope_all` axes
|
||||
are AND-matched — an extra axis the chunks don't carry (e.g. `agent_self`) zeroes
|
||||
the result even when `end_user` matches. `scope_any` is the OR/union escape hatch:
|
||||
a subset-scoped chunk recalls if its scope ⊇ **any one** element. So 0 hits with a
|
||||
populated store now means Worldtree sent an over-specified `scope_all` instead of a
|
||||
`scope_any` union — a **scope-build** question (Worldtree-side, post-v0.6).
|
||||
- **Hit present but the model says "no memory"** → we returned it; Worldtree
|
||||
dropped it downstream of search → **recall-assembly / injection**
|
||||
(Worldtree-side).
|
||||
@@ -144,10 +145,19 @@ RESPONSE: 0 hit(s)
|
||||
ANSWER: "I don't have access to your past preferences..."
|
||||
```
|
||||
|
||||
(That capture is the **pre-v0.6 wire** — a single AND-only `scope_filter`.)
|
||||
|
||||
Root cause: the recall filter carried `agent_self` but the stored chunks are
|
||||
`{end_user: smoke-user}` only → the `agent_self` axis excluded all of them.
|
||||
Branch (a), scope asymmetry — fed to #297.
|
||||
|
||||
**Resolution (bifrost 0.8.0 / wire v0.6, #11):** the single `scope_filter` is split
|
||||
into `scope_all` (AND) + `scope_any` (OR/union). Worldtree can now send the visible
|
||||
scopes as a `scope_any` union (e.g. `[{end_user: smoke-user}, {end_user: smoke-user,
|
||||
agent_self: ...}]`), so the subset-scoped chunk recalls via the matching OR member.
|
||||
Our store implements this at parity with the v0.6 reference; **end-to-end cold recall
|
||||
now waits only on Worldtree emitting `scope_any`** on the recall path (#297).
|
||||
|
||||
## Notes / foot-guns
|
||||
|
||||
- **HTTP, not HTTPS.** The spec requires `endpoint_url` be HTTPS; dev is relaxed
|
||||
|
||||
@@ -22,6 +22,16 @@ external_invariants:
|
||||
- source: ~/development/bifrost/docs/implementing-a-consumer.md
|
||||
invariant_id: "§5 memory plane"
|
||||
revisions:
|
||||
- version: "1.2"
|
||||
at: 2026-06-16
|
||||
summary: "Repin bifrost 0.7.0→0.8.0 (wire v0.5→v0.6, #11): search `scope_filter` split into `scope_all` (AND/intersection) + `scope_any` (OR/union over a list of conjunctive scopes). No-compat: `scope_filter` removed. Adds union-visibility recall in one call — the fix for the #295/#297 AND silent-zero foot-gun. Store at parity with the v0.6 reference `_matches_scope` / `_validate_scope`."
|
||||
delta:
|
||||
MODIFIED:
|
||||
- "search signature: scope_filter -> scope_all + scope_any"
|
||||
- "INV-005 scope isolation -> composed v0.6 (scope_all AND ∧ scope_any OR-union)"
|
||||
- "PRE-003 validates axes in BOTH fields; STEP 1 = _validate_scope (shape + lattice)"
|
||||
ADDED:
|
||||
- "scope_any_union test (#295/#297 union capability); both-fields-empty match-all"
|
||||
- version: "1.1"
|
||||
at: 2026-06-15
|
||||
summary: "Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005."
|
||||
@@ -98,9 +108,13 @@ interpreted.
|
||||
- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + their
|
||||
vec rows + the idempotency record in one transaction; on any error nothing is
|
||||
persisted (no partial batch, no orphaned vec rows).
|
||||
- **INV-005** [hard]: **Scope isolation.** `search` results are filtered to records
|
||||
whose `record["scope"]` matches every axis in `scope_filter`; a search never
|
||||
returns another scope's chunk.
|
||||
- **INV-005** [hard]: **Scope isolation (wire v0.6, #11).** `search` filters by two
|
||||
explicit fields: `scope_all` (AND/intersection — record ⊇ every named axis) and
|
||||
`scope_any` (OR/union over a LIST of conjunctive scope dicts — record ⊇ ≥1 element,
|
||||
each element AND-matched as a whole). They compose by AND; both empty → no scope
|
||||
constraint. A search never returns a chunk outside the composed filter. Byte-faithful
|
||||
to the reference `_matches_scope`. (`scope_any` is the union-visibility primitive that
|
||||
resolves the #295/#297 silent-zero — a subset-scoped chunk now recalls via an OR member.)
|
||||
- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement).
|
||||
`describe_store` advertises ONLY what v1 implements: `relational_edges_supported=False`,
|
||||
`atomic_supersede_supported=False`, `transaction_supported=False`,
|
||||
@@ -205,22 +219,23 @@ TESTS:
|
||||
```
|
||||
|
||||
```contract
|
||||
FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
|
||||
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
|
||||
FN search(self, vector: list[float], *, top_k: int, scope_all: dict | None = None, scope_any: list | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
|
||||
BRIEF: Vector (cosine) recall over sqlite-vec, scoped by the v0.6 scope_all/scope_any filter, returning the top_k IN-SCOPE chunks.
|
||||
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
|
||||
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
|
||||
PRE: [PRE-003 hard] every scope_filter axis ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.5 lattice, matching the reference _validate_scope_filter (#10 made agent_self canonical)
|
||||
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
|
||||
PRE: [PRE-003 hard] scope_all is a flat dict and scope_any a list of flat dicts (else InvalidArguments); every axis in BOTH ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.6 lattice, matching the reference _validate_scope (#10 agent_self canonical, #11 scope split)
|
||||
POST: [POST-001 return_value] returns the top_k highest-cosine records passing the composed v0.6 filter — `(scope_all empty OR record ⊇ scope_all) AND (scope_any empty OR record ⊇ ≥1 element)`; at most top_k, never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
|
||||
STEPS:
|
||||
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis]) AND every axis ∈ the v0.5 lattice {end_user, group, tenant, agent_self} (else InvalidFilter)
|
||||
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
|
||||
1. [setup] scope_all ← scope_all or {}; scope_any ← scope_any or []; validate via _validate_scope (flat-dict / list-of-dicts shape + every axis ∈ the v0.6 lattice, else InvalidArguments / InvalidFilter)
|
||||
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only records passing _matches_scope(scope_all, scope_any) (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
|
||||
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
|
||||
TESTS:
|
||||
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
|
||||
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
|
||||
empty [boundary]: search empty store → []
|
||||
scope_isolation [adversarial]: two scopes, scope_all one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
|
||||
scope_any_union [scenario]: scope_any=[{end_user:u},{agent_self:a}] recalls BOTH a subject-scoped and a self-scoped chunk in one call (#295/#297 union capability); scope_all+scope_any compose by AND
|
||||
empty [boundary]: search empty store → []; both fields empty → match all
|
||||
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
|
||||
lattice_axes [adversarial]: out-of-lattice scope axis → InvalidFilter; agent_self admitted (wire v0.5, #10)
|
||||
lattice_axes [adversarial]: out-of-lattice axis in scope_all OR scope_any → InvalidFilter; non-list scope_any → InvalidArguments; agent_self admitted (wire v0.5, #10)
|
||||
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
|
||||
```
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ assumptions:
|
||||
- "PROVEN: the session-create BEARER is the identity Worldtree signs the Bifrost handshake JWT with (HS256 shared-secret). Bearer = the canary key (WORLDTREE_API_KEY) → handshake 401 `bifrost.auth_rejected` → 502 to the client. Bearer = the consumer Heimdall key (== the provider's RATATOSKR_HEIMDALL_KEY string) → handshake 200. So a BOUND session-create MUST authenticate with the consumer key, NOT the canary key. These are two distinct ratatoskr identities."
|
||||
- "PROVEN: dev HTTP is accepted (spec wants HTTPS) because the provider host:port is on Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist — a Worldtree-side, infra-ops-owned config. The endpoint_url must be the WORLDTREE-VISIBLE base URL (e.g. `http://10.100.10.50:8391`), not the client's loopback. Provider routes live at `/bifrost/handshake` + `/bifrost/memory-call` (memory) and `/bifrost/affect-call` (affect) under that base."
|
||||
- "PROVEN (bifrost source, memory.py:244 vs 262): `dispatch_memory_call` passes `ctx` to `upsert_many` but NOT to `search`/`get`/`delete`. So the recall verb's store method has no session_id; correlation identity must be captured at the dispatch/ASGI layer (JWT ctx), not inside the store method. turn_id (finer than session_id) availability is UNVERIFIED — a contract-stage JWT-claims/envelope dump resolves it; design the op-feed to accept a turn_id later without a schema break."
|
||||
- "Provider stores MUST NOT change scope semantics. `_scope_matches` stays AND (byte-faithful to bifrost reference `reference_server/memory.py:398`); the missing axis-lattice validation (my store admits a wider axis set than the reference) is a SEPARATE parity item, GATED on the `agent_self`-canonicity resolution, and OUT OF SCOPE for #17."
|
||||
- "Provider stores MUST NOT change scope semantics. `_matches_scope` stays the v0.6 composed filter — `scope_all` (AND/intersection) ∧ `scope_any` (OR/union over conjunctive scopes) — byte-faithful to bifrost reference `reference_server/memory.py` (wire v0.6, #11); the 4-axis lattice validation (`_validate_scope`) is in place and at parity. Scope semantics are settled (the v0.6 scope split shipped, bifrost 0.8.0) and OUT OF SCOPE for #17 — observe is read-only over them."
|
||||
- "The existing `create_session(client, agent_id, *, end_user_id=None)` (sessions.py:179) is extended, not replaced (pre-v1, no compat shim). The httpx client carries the canary key as its default Authorization; the bound create overrides the bearer per-request with the consumer key."
|
||||
- "Tests use `respx` for the bind unit tests (mirroring tests/test_sessions.py) + the in-process op-feed; the live-smoke acceptance is manual (per the repo's load-bearing-smoke posture), captured as a documented runbook step, not a unit test. `docs/bifrost-self-test.md` is the manual procedure this feature productizes."
|
||||
- "v1 ships the CLI + TUI + web trigger surfaces in lockstep (the repo's BOTH-presenters-in-lockstep rule); the op-feed is read by the debug surface as structured lines for now (pane-correlated rendering is PARKED)."
|
||||
@@ -151,9 +151,9 @@ class BifrostConsumerKeyMissing(Exception):
|
||||
one plane. v1 documents this limit explicitly; binding both planes for one turn
|
||||
is the PARKED composite-endpoint feature, not a v1 path.
|
||||
- **INV-004 (no scope-semantics change).** The observe instrumentation is
|
||||
READ-ONLY over the dispatch path; it MUST NOT alter `_scope_matches`, the
|
||||
AND-parity, or any store behavior. The op-feed reports the effective scope used
|
||||
per op; it never rewrites scope client-side.
|
||||
READ-ONLY over the dispatch path; it MUST NOT alter `_matches_scope`, the v0.6
|
||||
`scope_all`/`scope_any` semantics, or any store behavior. The op-feed reports the
|
||||
effective scope used per op; it never rewrites scope client-side.
|
||||
- **INV-005 (correlation key at the dispatch layer).** session_id is captured from
|
||||
the JWT ctx at the dispatch/ASGI layer — present for ALL JWT-carrying verbs,
|
||||
INCLUDING search/get/delete (bifrost withholds ctx from those STORE methods, but
|
||||
@@ -239,7 +239,7 @@ POST: [POST-004 return_value] store scope semantics untouched; read-only over di
|
||||
STEPS:
|
||||
1. wrap the dispatch/ASGI layer so each inbound bifrost-call yields one OpEvent
|
||||
2. read session_id off the JWT ctx if present; else None
|
||||
3. summarise req (scope_filter/top_k for search; record-count+scopes for upsert) + resp (hit-count+ids/scores | upserted+replayed | error code) — NEVER verbatim content
|
||||
3. summarise req (scope_all/scope_any/top_k for search; record-count+scopes for upsert) + resp (hit-count+ids/scores | upserted+replayed | error code) — NEVER verbatim content
|
||||
4. emit to sink; swallow sink errors
|
||||
```
|
||||
|
||||
@@ -279,7 +279,7 @@ let an implementer comply while violating intent.
|
||||
|
||||
- **Op-feed summary shapes (per verb).** `req_summary` / `resp_summary` are
|
||||
scope-only, never verbatim content:
|
||||
- `search` → req `{scope_filter, top_k}`; resp `{hit_count, [{chunk_id, score}]}`
|
||||
- `search` → req `{scope_all, scope_any, top_k}`; resp `{hit_count, [{chunk_id, score}]}`
|
||||
- `upsert_many` → req `{record_count, [scope]}`; resp `{upserted, replayed}`
|
||||
- `get` / `get_many` → req `{ids}`; resp `{found_count}`
|
||||
- `delete_many` → req `{ids}`; resp `{deleted}`
|
||||
@@ -309,4 +309,4 @@ let an implementer comply while violating intent.
|
||||
- Turn-correlated debug-pane UI — needs turn_id (open question) + grace-window buffering + client read channel.
|
||||
- Key identity unification (Heimdall) — operator's call, crosses service boundary.
|
||||
- Auto-bind on Tier-3 agents — operator's call; v1 is explicit opt-in only.
|
||||
- Provider axis-lattice validation (the agent_self parity item) — gated on the `agent_self`-canonicity cross-repo resolution; tracked separately, NOT #17.
|
||||
- Provider axis-lattice validation + the v0.6 scope split — RESOLVED (shipped: bifrost 0.8.0/wire v0.6, `_validate_scope` 4-axis + `scope_all`/`scope_any`); no longer a #17 concern.
|
||||
|
||||
@@ -43,6 +43,22 @@ model output is untrusted); upstream API key stays server-side (INV-003).
|
||||
|
||||
_As of 2026-06-16:_
|
||||
|
||||
**LATEST (2026-06-16 PM) — BIFROST REPINNED 0.7.0→0.8.0 (wire v0.5→v0.6).** The
|
||||
memory `search` scope filter was split into `scope_all` (AND/intersection) +
|
||||
`scope_any` (OR/union over a LIST of conjunctive scopes) — bifrost #11, the canonical
|
||||
fix for the #295/#297 silent-zero AND foot-gun. Our store + contract (v1.2) + tests
|
||||
reimplemented to parity with the v0.6 reference `_matches_scope`/`_validate_scope`
|
||||
(no-compat: `scope_filter` REMOVED). 433 tests green incl. the new `scope_any` union
|
||||
test + the parity-vs-reference test through the real 0.8.0 `dispatch_memory_call`.
|
||||
Memory provider BOUNCED onto 0.8.0 (`:8391`, fresh empty `memory.db` — the prior
|
||||
5-chunk #296 corpus was WIPED, operator confirmed "nothing of value", SUPERSEDES the
|
||||
"KEEP PINNED" note below). **End-to-end cold recall now waits only on Worldtree
|
||||
EMITTING `scope_any` on its recall path (#297, upstream).** Affect plane untouched
|
||||
(split is memory-only); affect provider still on its 0.7.0-loaded process (bounce
|
||||
optional — affect wire unchanged at 0.8.0). NOT yet committed; patch bump v0.17.6
|
||||
pending operator commit approval. **#17 contract carries stale `scope_filter` /
|
||||
`_scope_matches`-AND references — fix when #17 TDD starts.**
|
||||
|
||||
**Ratatoskr now has a SECOND identity: the v1 Bifrost Tier-3 consumer** — the
|
||||
durable persistence provider Worldtree writes Tier-3 agent affect/persona +
|
||||
memory into — alongside the original debug-observability TUI/web. The
|
||||
@@ -193,6 +209,8 @@ decision. Captures rationale that won't be obvious from code alone.
|
||||
- `[2026-06-16]` **agent_self lattice SHIPPED both sides → our axis-validation gap CLOSED (v0.17.5).** bifrost 0.7.0 / wire v0.5 adds agent_self to the scope lattice `{end_user,group,tenant,agent_self}` (#10, driven by our foot-gun flag via bifrost-dev); Worldtree pinned 0.7.0 + canonical-synced the v0.5 spec (v0.35.11, `c860fb0`). **SUPERSEDES the prior "HELD, do NOT add axis-validation" note** — we DID add `_validate_scope_filter` (4-axis) to match the reference (bifrost-dev's recommendation, purely additive; out-of-lattice → InvalidFilter). #297 union build unblocked. Memory provider restarted on 0.7.0.
|
||||
- `[2026-06-16]` **#17 contract reviewed + the debug-assist arc fully closed.** `/heid` design consult + `/heid-contract-review` panel both run on `docs/contracts/issues/17.contract.md` (validates OK). The #295 debug-assist that opened the session is closed end-to-end: root cause (scope-axis asymmetry) → #296/#297 research issues + corpus → a shipped bifrost protocol change (agent_self canonical) → our store at parity. NEXT durable step: #17 TDD (tracked: Gitea #17 + the contract).
|
||||
|
||||
- `[2026-06-16]` **Repinned bifrost 0.7.0→0.8.0 + reimplemented memory `search` to the v0.6 scope split (operator-directed).** `scope_filter` → `scope_all` (AND) + `scope_any` (OR/union over a list of conjunctive scopes), bifrost #11 — the canonical resolution of the #295/#297 silent-zero. **SUPERSEDES the "Provider stores byte-faithful to AND reference / Do NOT flip `_scope_matches` to OR" entry above**: the reference itself now does OR via `scope_any` (a NEW field — `scope_all` keeps the old AND semantics; this is an additive split, not a flip of the AND predicate). Store / contract (v1.2) / tests at parity with the v0.6 reference; provider bounced onto 0.8.0 with a wiped DB (operator: "nothing of value"). Cold recall now gated only on Worldtree emitting `scope_any` (#297). **#17's contract has stale `scope_filter`/`_scope_matches`-AND references (its `assumptions`, INV-004, and the op-feed `search → req {scope_filter}` summary shape) — update those to `scope_all`/`scope_any` when #17 TDD starts; INV-004's intent (observe must not alter scope semantics) still holds.**
|
||||
|
||||
_For per-issue TDD implementation notes, Volva findings, and contract amendments, see the git log — every per-issue commit carries a structured message capturing the trail._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "ratatoskr"
|
||||
version = "0.17.5"
|
||||
version = "0.17.6"
|
||||
description = "Worldtree Conversation API debug TUI — multi-pane observability dashboard"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
@@ -30,7 +30,7 @@ web = [
|
||||
# from the debug TUI. Recipe: bifrost/docs/implementing-a-consumer.md.
|
||||
provider = [
|
||||
"ratatoskr[web]", # reuse the starlette + uvicorn ASGI stack
|
||||
"bifrost>=0.7.0", # consumer engines + library (0.7.0/wire-v0.5: agent_self canonical in the scope lattice; 0.6.0 yanked: circular import)
|
||||
"bifrost>=0.8.0", # consumer engines + library (0.8.0/wire-v0.6: scope_filter split into scope_all (AND) + scope_any (OR/union, #11); 0.7.0/v0.5 added agent_self)
|
||||
"jsonschema>=4", # bifrost runtime dep — envelope validation
|
||||
"sqlite-vec>=0.1.6", # vector index for the memory plane (vec0 virtual table)
|
||||
]
|
||||
|
||||
@@ -34,9 +34,9 @@ from bifrost.reference_server import JwtVerifier
|
||||
_SHORT_RETRY_TTL_SECONDS = 300
|
||||
_DURABLE_JOB_TTL_SECONDS = 24 * 60 * 60
|
||||
|
||||
# bifrost wire v0.5 scope lattice: three subject axes + agent_self (the #248
|
||||
# agent-identity axis, made canonical in #10). An axis outside it is InvalidFilter
|
||||
# (-> memory.invalid_filter 400), matching bifrost's reference _validate_scope_filter.
|
||||
# bifrost wire v0.6 scope lattice: three subject axes + agent_self (the #248
|
||||
# agent-identity axis, canonical since #10/v0.5). An axis outside it is InvalidFilter
|
||||
# (-> memory.invalid_filter 400), matching bifrost's reference _validate_scope.
|
||||
_SCOPE_LATTICE = {"end_user", "group", "tenant", "agent_self"}
|
||||
|
||||
# Inbound memory-call observability (#17 observe brick). A self-contained
|
||||
@@ -87,11 +87,37 @@ def _record_vector(record: dict) -> list[float]:
|
||||
return [float(v) for v in value] if isinstance(value, list) else []
|
||||
|
||||
|
||||
def _scope_matches(record_scope: Any, scope_filter: dict) -> bool:
|
||||
"""INV-005: record is in-scope iff every scope_filter axis matches record["scope"]."""
|
||||
if not isinstance(record_scope, dict):
|
||||
def _scope_subset(record_scope: dict, filter_dict: dict) -> bool:
|
||||
"""True iff record_scope has EVERY axis of filter_dict (AND). Empty filter = match."""
|
||||
return all(record_scope.get(axis) == value for axis, value in filter_dict.items())
|
||||
|
||||
|
||||
def _matches_scope(record_scope: Any, scope_all: dict, scope_any: list) -> bool:
|
||||
"""INV-005 (wire v0.6, #11): a record passes iff
|
||||
`(scope_all empty OR record ⊇ scope_all) AND (scope_any empty OR it matches ≥1 element)`.
|
||||
scope_any is OR/union over a LIST of conjunctive scope dicts (each AND-matched as a whole),
|
||||
never single axes flattened together (the Worldtree #297 over-broadening foot-gun).
|
||||
Byte-faithful to bifrost reference `reference_server/memory.py:_matches_scope`.
|
||||
"""
|
||||
scope = record_scope if isinstance(record_scope, dict) else {}
|
||||
if not _scope_subset(scope, scope_all):
|
||||
return False
|
||||
return all(record_scope.get(axis) == value for axis, value in scope_filter.items())
|
||||
if scope_any and not any(_scope_subset(scope, element) for element in scope_any):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _validate_scope(scope_all: dict, scope_any: list) -> None:
|
||||
"""STEP 1: scope_all is a flat dict, scope_any a list of flat dicts; every axis in BOTH
|
||||
fields ∈ the v0.6 lattice {end_user, group, tenant, agent_self} (else InvalidFilter),
|
||||
matching the reference `_validate_scope`."""
|
||||
if not isinstance(scope_all, dict):
|
||||
raise InvalidArguments("scope_all must be a flat {axis: value} dict")
|
||||
if not isinstance(scope_any, list) or any(not isinstance(e, dict) for e in scope_any):
|
||||
raise InvalidArguments("scope_any must be a list of {axis: value} dicts")
|
||||
for scope in (scope_all, *scope_any):
|
||||
if any(axis not in _SCOPE_LATTICE for axis in scope):
|
||||
raise InvalidFilter("scope_filter contains unsupported axis")
|
||||
|
||||
|
||||
def _validate_injection(record: dict) -> None:
|
||||
@@ -200,7 +226,8 @@ class RatatoskrMemoryStore:
|
||||
vector: list[float],
|
||||
*,
|
||||
top_k: int,
|
||||
scope_filter: dict | None = None,
|
||||
scope_all: dict | None = None,
|
||||
scope_any: list | None = None,
|
||||
metadata_filter: dict | None = None,
|
||||
include: dict | None = None,
|
||||
fidelity_target: Any = None,
|
||||
@@ -209,13 +236,12 @@ class RatatoskrMemoryStore:
|
||||
raise InvalidArguments(f"vector length {len(vector)} != embedding_dim {self._dim}")
|
||||
if metadata_filter: # PRE-002: v1 advertises no filterable metadata fields
|
||||
raise InvalidArguments("metadata_filter is unsupported in v1")
|
||||
if scope_filter is not None and not isinstance(scope_filter, dict): # STEP 1
|
||||
raise InvalidArguments("scope_filter must be a flat {axis: value} dict")
|
||||
if scope_filter and any(axis not in _SCOPE_LATTICE for axis in scope_filter):
|
||||
raise InvalidFilter("scope_filter contains unsupported axis")
|
||||
scope_all = scope_all or {}
|
||||
scope_any = scope_any or []
|
||||
_validate_scope(scope_all, scope_any) # STEP 1 (raises InvalidArguments / InvalidFilter)
|
||||
_log.info(
|
||||
"memory-call search REQUEST: scope_filter=%r top_k=%s metadata_filter=%r vec_dim=%d",
|
||||
scope_filter, top_k, metadata_filter, len(vector),
|
||||
"memory-call search REQUEST: scope_all=%r scope_any=%r top_k=%s vec_dim=%d",
|
||||
scope_all, scope_any, top_k, len(vector),
|
||||
)
|
||||
|
||||
def _emit(rs: list[dict]) -> list[dict]:
|
||||
@@ -249,7 +275,7 @@ class RatatoskrMemoryStore:
|
||||
results: list[dict] = []
|
||||
for chunk_id, distance, record_json, revision in rows:
|
||||
record = json.loads(record_json)
|
||||
if scope_filter and not _scope_matches(record.get("scope"), scope_filter):
|
||||
if not _matches_scope(record.get("scope"), scope_all, scope_any):
|
||||
continue
|
||||
results.append(
|
||||
{
|
||||
|
||||
@@ -176,7 +176,7 @@ async def test_basic_search_ranks_by_cosine_with_recalled_view():
|
||||
idempotency_key="k1",
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_filter=scope)
|
||||
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_all=scope)
|
||||
assert [r["chunk_id"] for r in results] == ["c1", "c3"] # nearest to [1,0] by cosine
|
||||
top = results[0]
|
||||
assert top["chunk"] == c1 # verbatim chunk, no revision attached
|
||||
@@ -196,7 +196,7 @@ async def test_scope_isolation_excludes_other_scope_even_if_closer():
|
||||
idempotency_key="k1",
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_filter={"end_user": "u1"})
|
||||
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_all={"end_user": "u1"})
|
||||
assert [r["chunk_id"] for r in results] == ["u1-far"] # u2-near excluded despite ranking first
|
||||
|
||||
|
||||
@@ -219,19 +219,28 @@ async def test_search_wrong_vector_dim_rejected():
|
||||
await store.search([1.0, 0.0], top_k=5)
|
||||
|
||||
|
||||
async def test_search_non_dict_scope_filter_rejected():
|
||||
# search STEP 1: scope_filter must be a flat {axis: value} dict
|
||||
async def test_search_non_dict_scope_all_rejected():
|
||||
# search STEP 1: scope_all must be a flat {axis: value} dict
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
with pytest.raises(InvalidArguments):
|
||||
await store.search(_vec(1.0), top_k=5, scope_filter="u1")
|
||||
await store.search(_vec(1.0), top_k=5, scope_all="u1")
|
||||
|
||||
|
||||
async def test_search_non_list_scope_any_rejected():
|
||||
# search STEP 1: scope_any must be a LIST of {axis: value} dicts (#11)
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
with pytest.raises(InvalidArguments):
|
||||
await store.search(_vec(1.0), top_k=5, scope_any={"end_user": "u1"})
|
||||
|
||||
|
||||
async def test_search_out_of_lattice_scope_axis_rejected():
|
||||
# v0.5 scope lattice = {end_user, group, tenant, agent_self}; an axis outside
|
||||
# it is InvalidFilter (-> memory.invalid_filter 400), matching bifrost's reference.
|
||||
# v0.6 scope lattice = {end_user, group, tenant, agent_self}; an axis outside
|
||||
# it is InvalidFilter (-> memory.invalid_filter 400) in EITHER field, matching the reference.
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
with pytest.raises(InvalidFilter):
|
||||
await store.search(_vec(1.0), top_k=5, scope_filter={"bogus_axis": "x"})
|
||||
await store.search(_vec(1.0), top_k=5, scope_all={"bogus_axis": "x"})
|
||||
with pytest.raises(InvalidFilter):
|
||||
await store.search(_vec(1.0), top_k=5, scope_any=[{"bogus_axis": "x"}])
|
||||
|
||||
|
||||
async def test_search_agent_self_axis_accepted():
|
||||
@@ -243,7 +252,7 @@ async def test_search_agent_self_axis_accepted():
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(
|
||||
_vec(1.0), top_k=5, scope_filter={"agent_self": "ratatoskr:smoke"}
|
||||
_vec(1.0), top_k=5, scope_all={"agent_self": "ratatoskr:smoke"}
|
||||
)
|
||||
assert [r["chunk_id"] for r in results] == ["a1"]
|
||||
|
||||
@@ -252,7 +261,67 @@ async def test_search_top_k_zero_returns_empty():
|
||||
# POST-001: at most top_k — zero means zero
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
await store.upsert_many([_chunk("c1")], idempotency_key="k1", ctx=_ctx())
|
||||
assert await store.search(_vec(1.0), top_k=0, scope_filter={"end_user": "u1"}) == []
|
||||
assert await store.search(_vec(1.0), top_k=0, scope_all={"end_user": "u1"}) == []
|
||||
|
||||
|
||||
async def test_search_no_scope_matches_all():
|
||||
# v0.6: both fields empty -> no scope constraint (match all, within top_k).
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
await store.upsert_many(
|
||||
[
|
||||
_chunk("u1", scope={"end_user": "u1"}),
|
||||
_chunk("u2", scope={"end_user": "u2"}),
|
||||
],
|
||||
idempotency_key="k1",
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(_vec(1.0), top_k=10)
|
||||
assert {r["chunk_id"] for r in results} == {"u1", "u2"}
|
||||
|
||||
|
||||
async def test_search_scope_any_unions_across_scopes():
|
||||
# v0.6 (#11): scope_any is OR/union over a LIST of conjunctive scopes. A {end_user:u1}
|
||||
# chunk AND an {agent_self:a} chunk are BOTH recalled in ONE call — the capability
|
||||
# that resolves the #295/#297 silent-zero AND foot-gun (subset-scoped chunks now recall).
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
await store.upsert_many(
|
||||
[
|
||||
_chunk("subj", embedding=_vec(1.0, 0.0), scope={"end_user": "u1"}),
|
||||
_chunk("self", embedding=_vec(0.9, 0.1), scope={"agent_self": "ratatoskr:sindra"}),
|
||||
_chunk("other", embedding=_vec(0.8, 0.2), scope={"end_user": "u9"}),
|
||||
],
|
||||
idempotency_key="k1",
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(
|
||||
_vec(1.0, 0.0),
|
||||
top_k=10,
|
||||
scope_any=[{"end_user": "u1"}, {"agent_self": "ratatoskr:sindra"}],
|
||||
)
|
||||
assert {r["chunk_id"] for r in results} == {"subj", "self"} # union; u9 excluded
|
||||
|
||||
|
||||
async def test_search_scope_all_and_scope_any_compose_by_and():
|
||||
# v0.6: a record passes iff (record ⊇ scope_all) AND (matches ≥1 scope_any element).
|
||||
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
|
||||
await store.upsert_many(
|
||||
[
|
||||
# tenant t1 AND (end_user u1 OR u2) — only these pass
|
||||
_chunk("t1u1", embedding=_vec(1.0, 0.0), scope={"tenant": "t1", "end_user": "u1"}),
|
||||
_chunk("t1u2", embedding=_vec(0.9, 0.1), scope={"tenant": "t1", "end_user": "u2"}),
|
||||
_chunk("t1u9", embedding=_vec(0.8, 0.2), scope={"tenant": "t1", "end_user": "u9"}),
|
||||
_chunk("t2u1", embedding=_vec(0.7, 0.3), scope={"tenant": "t2", "end_user": "u1"}),
|
||||
],
|
||||
idempotency_key="k1",
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(
|
||||
_vec(1.0, 0.0),
|
||||
top_k=10,
|
||||
scope_all={"tenant": "t1"},
|
||||
scope_any=[{"end_user": "u1"}, {"end_user": "u2"}],
|
||||
)
|
||||
assert {r["chunk_id"] for r in results} == {"t1u1", "t1u2"} # t1u9 fails any; t2u1 fails all
|
||||
|
||||
|
||||
async def test_scope_isolation_fills_top_k_from_in_scope_past_higher_out_of_scope():
|
||||
@@ -269,7 +338,7 @@ async def test_scope_isolation_fills_top_k_from_in_scope_past_higher_out_of_scop
|
||||
idempotency_key="k1",
|
||||
ctx=_ctx(),
|
||||
)
|
||||
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_filter={"end_user": "u1"})
|
||||
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_all={"end_user": "u1"})
|
||||
# exactly top_k in-scope (the 2 nearest u1 chunks); the higher-ranked u2 chunk is excluded
|
||||
assert [r["chunk_id"] for r in results] == ["u1-near", "u1-mid"]
|
||||
|
||||
@@ -298,7 +367,7 @@ async def test_delete_hit_removes_chunk_and_vec_row():
|
||||
assert _row_count(store, "memory_chunks") == 1
|
||||
assert _row_count(store, "memory_vec") == 1 # c1's vec row gone too (no orphan)
|
||||
# delete_hit: search no longer surfaces it (vec/chunk coupling held)
|
||||
hits = await store.search(_vec(1.0), top_k=5, scope_filter={"end_user": "u1"})
|
||||
hits = await store.search(_vec(1.0), top_k=5, scope_all={"end_user": "u1"})
|
||||
assert all(r["chunk_id"] != "c1" for r in hits)
|
||||
|
||||
|
||||
@@ -388,7 +457,7 @@ async def test_parity_search_ranked_ids_vs_reference_through_dispatch():
|
||||
await dispatch_memory_call(up, wctx, mine)
|
||||
search_env = {
|
||||
"operation": "search",
|
||||
"args": {"vector": [1.0, 0.0], "top_k": 2, "scope_filter": {"end_user": "u1"}},
|
||||
"args": {"vector": [1.0, 0.0], "top_k": 2, "scope_all": {"end_user": "u1"}},
|
||||
}
|
||||
rstatus, rbody = await dispatch_memory_call(search_env, rctx, ref)
|
||||
mstatus, mbody = await dispatch_memory_call(search_env, rctx, mine)
|
||||
@@ -415,7 +484,9 @@ async def test_parity_expected_revisions_vs_reference_through_dispatch():
|
||||
"args": {"records": [_ref_record("a", [1.0, 0.0]), _ref_record("b", [0.0, 1.0])]},
|
||||
"idempotency_key": "seed",
|
||||
}
|
||||
assert await dispatch_memory_call(seed, wctx, ref) == await dispatch_memory_call(seed, wctx, mine)
|
||||
ref_seed = await dispatch_memory_call(seed, wctx, ref)
|
||||
mine_seed = await dispatch_memory_call(seed, wctx, mine)
|
||||
assert ref_seed == mine_seed
|
||||
|
||||
# partial map: only "a" is locked (revision 1); "b" is omitted from expected_revisions
|
||||
partial = {
|
||||
|
||||
@@ -190,14 +190,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "bifrost"
|
||||
version = "0.7.0"
|
||||
version = "0.8.0"
|
||||
source = { registry = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" }
|
||||
dependencies = [
|
||||
{ name = "jsonschema" },
|
||||
]
|
||||
sdist = { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.7.0/bifrost-0.7.0.tar.gz", hash = "sha256:27d5c2c3052ad48510a4a73973141f2177b9e1f66576ab3d8976d5c4deb5bfac" }
|
||||
sdist = { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.8.0/bifrost-0.8.0.tar.gz", hash = "sha256:28194877c81a056a0803b052e86902c092e965d4ce63a5623d7a31240cedb645" }
|
||||
wheels = [
|
||||
{ url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.7.0/bifrost-0.7.0-py3-none-any.whl", hash = "sha256:9704c50430f350f6dde5428f79a8719e044021ce717a5eac60a13424b1ee4623" },
|
||||
{ url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.8.0/bifrost-0.8.0-py3-none-any.whl", hash = "sha256:2aac5e4a7828d718389748a78dae6baeb5e9ee4a801a10c427c06c5cc7ed6597" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1052,7 +1052,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "ratatoskr"
|
||||
version = "0.17.5"
|
||||
version = "0.17.6"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "httpx" },
|
||||
@@ -1086,7 +1086,7 @@ web = [
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "bifrost", marker = "extra == 'provider'", specifier = ">=0.7.0", index = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" },
|
||||
{ name = "bifrost", marker = "extra == 'provider'", specifier = ">=0.8.0", index = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" },
|
||||
{ name = "httpx", specifier = ">=0.27" },
|
||||
{ name = "httpx-sse", specifier = ">=0.4" },
|
||||
{ name = "jsonschema", marker = "extra == 'provider'", specifier = ">=4" },
|
||||
|
||||
Reference in New Issue
Block a user