feat(provider): validate scope_filter against the v0.5 4-axis lattice

bifrost 0.7.0 (wire v0.5) makes agent_self canonical: the scope lattice is
now {end_user, group, tenant, agent_self}. Our store was MORE permissive than
bifrost's reference (no _validate_scope_filter), which silently 0-zeroed the
#295 cold recall instead of a loud 400. Now matched: search rejects an
out-of-lattice axis with InvalidFilter (-> memory.invalid_filter 400), agent_self
admitted. Purely additive — everything that validated before still validates.

Closes the parity gap our own foot-gun flag opened (bifrost-dev shipped the
lattice add #10 off it). Pin bumped bifrost>=0.6.1 -> >=0.7.0. Contract
search PRE-003 + lattice_axes test; 2 new store tests; full suite 429 green.
This commit is contained in:
vh
2026-06-16 01:24:01 -07:00
parent ca02c70b7c
commit aac4353933
5 changed files with 46 additions and 9 deletions
@@ -209,9 +209,10 @@ FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None =
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks. BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
PRE: [PRE-003 hard] every scope_filter axis ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.5 lattice, matching the reference _validate_scope_filter (#10 made agent_self canonical)
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS: STEPS:
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis]) 1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis]) AND every axis ∈ the v0.5 lattice {end_user, group, tenant, agent_self} (else InvalidFilter)
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed) 2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision) 3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS: TESTS:
@@ -219,6 +220,7 @@ TESTS:
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005) scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
empty [boundary]: search empty store → [] empty [boundary]: search empty store → []
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
lattice_axes [adversarial]: out-of-lattice scope axis → InvalidFilter; agent_self admitted (wire v0.5, #10)
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195) parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
``` ```
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "ratatoskr" name = "ratatoskr"
version = "0.17.4" version = "0.17.5"
description = "Worldtree Conversation API debug TUI — multi-pane observability dashboard" description = "Worldtree Conversation API debug TUI — multi-pane observability dashboard"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
@@ -30,7 +30,7 @@ web = [
# from the debug TUI. Recipe: bifrost/docs/implementing-a-consumer.md. # from the debug TUI. Recipe: bifrost/docs/implementing-a-consumer.md.
provider = [ provider = [
"ratatoskr[web]", # reuse the starlette + uvicorn ASGI stack "ratatoskr[web]", # reuse the starlette + uvicorn ASGI stack
"bifrost>=0.6.1", # consumer engines + library (0.6.0 yanked: circular import) "bifrost>=0.7.0", # consumer engines + library (0.7.0/wire-v0.5: agent_self canonical in the scope lattice; 0.6.0 yanked: circular import)
"jsonschema>=4", # bifrost runtime dep — envelope validation "jsonschema>=4", # bifrost runtime dep — envelope validation
"sqlite-vec>=0.1.6", # vector index for the memory plane (vec0 virtual table) "sqlite-vec>=0.1.6", # vector index for the memory plane (vec0 virtual table)
] ]
+8
View File
@@ -25,6 +25,7 @@ from bifrost.consumer import ConsumerRegistration, build_memory_app
from bifrost.memory import ( from bifrost.memory import (
IdempotencyConflict, IdempotencyConflict,
InvalidArguments, InvalidArguments,
InvalidFilter,
RevisionMismatch, RevisionMismatch,
StoreCapabilities, StoreCapabilities,
) )
@@ -33,6 +34,11 @@ from bifrost.reference_server import JwtVerifier
_SHORT_RETRY_TTL_SECONDS = 300 _SHORT_RETRY_TTL_SECONDS = 300
_DURABLE_JOB_TTL_SECONDS = 24 * 60 * 60 _DURABLE_JOB_TTL_SECONDS = 24 * 60 * 60
# bifrost wire v0.5 scope lattice: three subject axes + agent_self (the #248
# agent-identity axis, made canonical in #10). An axis outside it is InvalidFilter
# (-> memory.invalid_filter 400), matching bifrost's reference _validate_scope_filter.
_SCOPE_LATTICE = {"end_user", "group", "tenant", "agent_self"}
# Inbound memory-call observability (#17 observe brick). A self-contained # Inbound memory-call observability (#17 observe brick). A self-contained
# stdout handler so the lines reliably reach the provider's stdout regardless # stdout handler so the lines reliably reach the provider's stdout regardless
# of uvicorn's logging config. INFO-level, no propagation to root. # of uvicorn's logging config. INFO-level, no propagation to root.
@@ -205,6 +211,8 @@ class RatatoskrMemoryStore:
raise InvalidArguments("metadata_filter is unsupported in v1") raise InvalidArguments("metadata_filter is unsupported in v1")
if scope_filter is not None and not isinstance(scope_filter, dict): # STEP 1 if scope_filter is not None and not isinstance(scope_filter, dict): # STEP 1
raise InvalidArguments("scope_filter must be a flat {axis: value} dict") raise InvalidArguments("scope_filter must be a flat {axis: value} dict")
if scope_filter and any(axis not in _SCOPE_LATTICE for axis in scope_filter):
raise InvalidFilter("scope_filter contains unsupported axis")
_log.info( _log.info(
"memory-call search REQUEST: scope_filter=%r top_k=%s metadata_filter=%r vec_dim=%d", "memory-call search REQUEST: scope_filter=%r top_k=%s metadata_filter=%r vec_dim=%d",
scope_filter, top_k, metadata_filter, len(vector), scope_filter, top_k, metadata_filter, len(vector),
+28 -1
View File
@@ -10,7 +10,12 @@ from __future__ import annotations
import types import types
import pytest import pytest
from bifrost.memory import IdempotencyConflict, InvalidArguments, RevisionMismatch from bifrost.memory import (
IdempotencyConflict,
InvalidArguments,
InvalidFilter,
RevisionMismatch,
)
from ratatoskr.provider.memory_store import ( from ratatoskr.provider.memory_store import (
build_memory_provider_app, build_memory_provider_app,
@@ -221,6 +226,28 @@ async def test_search_non_dict_scope_filter_rejected():
await store.search(_vec(1.0), top_k=5, scope_filter="u1") await store.search(_vec(1.0), top_k=5, scope_filter="u1")
async def test_search_out_of_lattice_scope_axis_rejected():
# v0.5 scope lattice = {end_user, group, tenant, agent_self}; an axis outside
# it is InvalidFilter (-> memory.invalid_filter 400), matching bifrost's reference.
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
with pytest.raises(InvalidFilter):
await store.search(_vec(1.0), top_k=5, scope_filter={"bogus_axis": "x"})
async def test_search_agent_self_axis_accepted():
# agent_self became canonical at wire v0.5 (#10) — admitted, not rejected.
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
await store.upsert_many(
[_chunk("a1", scope={"agent_self": "ratatoskr:smoke"})],
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(
_vec(1.0), top_k=5, scope_filter={"agent_self": "ratatoskr:smoke"}
)
assert [r["chunk_id"] for r in results] == ["a1"]
async def test_search_top_k_zero_returns_empty(): async def test_search_top_k_zero_returns_empty():
# POST-001: at most top_k — zero means zero # POST-001: at most top_k — zero means zero
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM) store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
Generated
+5 -5
View File
@@ -190,14 +190,14 @@ wheels = [
[[package]] [[package]]
name = "bifrost" name = "bifrost"
version = "0.6.1" version = "0.7.0"
source = { registry = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" } source = { registry = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" }
dependencies = [ dependencies = [
{ name = "jsonschema" }, { name = "jsonschema" },
] ]
sdist = { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.6.1/bifrost-0.6.1.tar.gz", hash = "sha256:2eaf93c6da91faa6faa80a4c9a8d0c66161f4a7cc31ff041b0ae64daf3c161ea" } sdist = { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.7.0/bifrost-0.7.0.tar.gz", hash = "sha256:27d5c2c3052ad48510a4a73973141f2177b9e1f66576ab3d8976d5c4deb5bfac" }
wheels = [ wheels = [
{ url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.6.1/bifrost-0.6.1-py3-none-any.whl", hash = "sha256:ed505d2c08cf4cdd0a84c68ec42f8732b4c1baf7d72befe5eacf75d88381d5ce" }, { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.7.0/bifrost-0.7.0-py3-none-any.whl", hash = "sha256:9704c50430f350f6dde5428f79a8719e044021ce717a5eac60a13424b1ee4623" },
] ]
[[package]] [[package]]
@@ -1052,7 +1052,7 @@ wheels = [
[[package]] [[package]]
name = "ratatoskr" name = "ratatoskr"
version = "0.17.4" version = "0.17.5"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "httpx" }, { name = "httpx" },
@@ -1086,7 +1086,7 @@ web = [
[package.metadata] [package.metadata]
requires-dist = [ requires-dist = [
{ name = "bifrost", marker = "extra == 'provider'", specifier = ">=0.6.1", index = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" }, { name = "bifrost", marker = "extra == 'provider'", specifier = ">=0.7.0", index = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" },
{ name = "httpx", specifier = ">=0.27" }, { name = "httpx", specifier = ">=0.27" },
{ name = "httpx-sse", specifier = ">=0.4" }, { name = "httpx-sse", specifier = ">=0.4" },
{ name = "jsonschema", marker = "extra == 'provider'", specifier = ">=4" }, { name = "jsonschema", marker = "extra == 'provider'", specifier = ">=4" },