Compare commits

...

12 Commits

Author SHA1 Message Date
vh 37cdef511f fix(web): de-ugly the Tier-3 persona pane — clear message instead of bare HTTP 404
persona_state hard-404s every Tier-3 (colon-id) agent by design upstream
(WT api.py:1220, "Phase 2.0 has no Tier 3 persona") — so the Persona pane
showed "persona not available (HTTP 404)" for consumer-defined characters.
loadPersona now reads error_code + renders a clear Tier-3-aware message
(she still responds in character; only the affect/OCEAN readout is gated),
with distinct text for persona_not_configured / 403 / other.

Also (snapshot): sindra switched to thoughtful-character role
(mistral-small-4-reasoning); worldtree-dev pinged re Tier-3 persona_state
roadmap (thread 01KVCR6P); #17 (bifrost-binding the chat client) teed up as
the next-context target.

v0.17.7
2026-06-18 00:00:41 -07:00
vh 835375d22b memory: snapshot — FULL COVERAGE proven (verbose persona too): sindra-probe theatrical turn promoted the user fact cleanly under Stage 2/v0.36.0 + cold-recalled @0.694; :8081 confirmed on v0.36.0; closes the verbose-persona caveat end-to-end. Operator session: :8391 wiped, ratatoskr-web up :8765 (consumer key, sindra in picker) — persona+debug only, web client does NOT bind :8391 (#17 unbuilt = no memory persistence in web chat) 2026-06-17 23:45:08 -07:00
vh 7666203722 memory: snapshot — Tier-3 memory PROVEN end-to-end live (terse-probe cold recall @0.6994, fresh history-free session); #296 arc closed: Stage 1 (v0.35.19) recallability gate validated live + bisect localized residual to verbose-persona volume, Stage 2 (v0.36.0) MERGED at worldtree-codex (user-only per-turn extraction), live-validated eval fixture pair -> #305; root-cause chain v0.35.16 emit-2-meta -> v0.35.19 emit-then-reject -> v0.36.0 fix; foot-gun: :8391 store-wipe != WT promotion-dedup reset (clean promotion smoke needs a fresh agent+end_user) 2026-06-17 21:27:17 -07:00
vh 84d8c3f65f memory: snapshot — cold-recall arc PROVEN live e2e (#297/#298 union recall; WT v0.35.16 emits scope_any into our v0.17.6 store); #296 extraction quality the isolated upstream gap (triage→worldtree-dev, both symptoms localized in-code: empty _EXTRACTOR_SYSTEM + both-roles prefilter); sindra restored (DELETE+redefine, role:character→mistral-small-4, memory:{}); learnings: Tier-3 owner-scoped, define-takes-role, promotion 4-trigger hybrid, DELETE≠drain 2026-06-17 11:15:06 -07:00
vh 4eee7c89b2 pin: bump Worldtree spec to f1b59f8 (v0.35.16) — cold recall closes end-to-end
Worldtree shipped its half of the union-recall fix: #297 (client-side
per-scope-value union recall) + #298/#299 (adopt the bifrost v0.6
scope_any/scope_all wire, v0.35.16). It now emits scope_any on the recall
path, pairing with our v0.17.6 provider — cold cross-session recall is
closed end-to-end (pending a live re-smoke against a v0.35.16 instance).

Re-vendored conversation-api-spec.md + conversation_api.contract.md;
285-commit catch-up (v0.29.0 -> v0.35.16). Diff-reviewed: no client-facing
breaking changes for our consumer.

- #211 agent-slug rename (saga->echo, actor->mask) — slugs only, we pass --agent
- #245 end_user_id persistence + memory-scope resolver (additive)
- #187/#188/#219 Tier-3 define/PATCH policy (additive); error codes stable
- bifrost binding field + ephemeral_does_not_accept_bifrost 422 now documented (#17 surface)
- docs: SPEC-PIN.md pin table + history; bifrost-self-test recall status; persistent-memory

No package version bump (docs/pin-only, no ratatoskr code change).
2026-06-17 08:24:27 -07:00
vh 96d61a4bb1 feat(provider): split memory search scope_filter → scope_all + scope_any (bifrost 0.8.0/wire v0.6)
Repin bifrost 0.7.0→0.8.0 and reimplement the memory store's search scope
filter to the v0.6 split (#11): scope_all (AND/intersection) + scope_any
(OR/union over a list of conjunctive scopes), at parity with the v0.6
reference _matches_scope / _validate_scope. No-compat: scope_filter removed.

scope_any is the union-visibility primitive that resolves the #295/#297
silent-zero AND foot-gun — a subset-scoped chunk now recalls via an OR
member. End-to-end cold recall now gated only on Worldtree emitting
scope_any on its recall path (#297, upstream).

- store: search(scope_all, scope_any); _scope_subset + _matches_scope + _validate_scope
- contract v1.2: search FN sig, INV-005 recomposed, PRE-003 both fields, scope_any_union test
- tests: scope_any union, scope_all∧scope_any compose, both-empty match-all; parity vs real 0.8.0 dispatch (433 green)
- #17 contract: sync stale scope_filter/_scope_matches-AND refs to scope_all/scope_any
- runbook + persistent-memory updated; provider bounced onto 0.8.0 (fresh empty db)

v0.17.6
2026-06-16 23:09:31 -07:00
vh 43f2e148ad memory: snapshot — observe brick + self-drive proven; #295 root-caused (upstream); agent_self canonical shipped both sides + 4-axis parity (v0.17.5); #17 contract reviewed, TDD next 2026-06-16 18:26:36 -07:00
vh aac4353933 feat(provider): validate scope_filter against the v0.5 4-axis lattice
bifrost 0.7.0 (wire v0.5) makes agent_self canonical: the scope lattice is
now {end_user, group, tenant, agent_self}. Our store was MORE permissive than
bifrost's reference (no _validate_scope_filter), which silently 0-zeroed the
#295 cold recall instead of a loud 400. Now matched: search rejects an
out-of-lattice axis with InvalidFilter (-> memory.invalid_filter 400), agent_self
admitted. Purely additive — everything that validated before still validates.

Closes the parity gap our own foot-gun flag opened (bifrost-dev shipped the
lattice add #10 off it). Pin bumped bifrost>=0.6.1 -> >=0.7.0. Contract
search PRE-003 + lattice_axes test; 2 new store tests; full suite 429 green.
2026-06-16 01:24:01 -07:00
vh ca02c70b7c docs(#17): self-drive+observe contract, bifrost self-test runbook + snapshot
- docs/contracts/issues/17.contract.md — issue-scoped v2.1 contract for #17
  (Bifrost-binding the chat client). v1 scope = single-plane bind +
  dispatch-layer op-feed (composite endpoint + turn-pane UI parked). Design
  consulted via /heid, paraphrase-gated via /heid-contract-review panel; two
  internal inconsistencies fixed (OpEvent turn_id reservation made literal;
  session_id-for-all-verbs correction). Validates OK, prd drift-clean.
- docs/bifrost-self-test.md — reusable runbook for driving + observing the
  full Bifrost round-trip against our own provider (the manual form of #17;
  pins the consumer-key-as-bearer tripwire).
- persistent-memory.md — snapshot: observe brick shipped, self-drive proven,
  #295 root-caused (upstream, scope-axis asymmetry) -> #296/#297, agent_self
  -> canonical decided.
2026-06-16 01:15:28 -07:00
vh 2fef6e39f9 feat(provider): inbound memory-call observe log (#17 observe brick)
Structured [memory-provider] request/response logging on the memory-call
path: search REQUEST (scope_filter, top_k) + RESPONSE (chunk_ids, scores,
scope), plus concise entry lines on upsert_many/delete_many. Self-contained
stdout handler so the lines reach the provider stdout under uvicorn.

Additive observability only — no search-semantics change (AND-parity with
bifrost's reference store holds). This is the first concrete brick of #17's
observe half, and the lens that root-caused #295's cold-recall miss (the
persist/recall scope-axis asymmetry) from the provider side.
2026-06-16 01:15:16 -07:00
vh 2b47dcff5a memory: snapshot — memory provider live-proven (persist/dispatch/search); recall-injection upstream; #17 filed 2026-06-15 23:44:33 -07:00
vh e57b054054 memory: snapshot — memory plane shipped (v0.17.3), canonical sync + #3/#4 refresh 2026-06-15 21:46:07 -07:00
12 changed files with 1340 additions and 96 deletions
+6 -6
View File
@@ -7,17 +7,17 @@ documents the pin, the vendored artifacts, and the bump procedure.
| Field | Value |
|---|---|
| Worldtree git SHA | `562001af28d752c3a60d449c7ddd09f44fa9dc9a` |
| Worldtree HEAD message | `feat(#201): v0.29.0 — awaiting_llm_first_token SSE heartbeat` |
| Pinned on | 2026-05-26 |
| Pinned by | ratatoskr-dev (bump for #201 awaiting_llm_first_token SSE) |
| Worldtree version at pin | `v0.29.0` |
| Worldtree git SHA | `f1b59f8cd6fe41e497d0be9dad9d3110451f0d9a` |
| Worldtree HEAD message | `Merge #299: adopt bifrost v0.6 memory scope wire (scope_any/scope_all)` |
| Pinned on | 2026-06-17 |
| Pinned by | ratatoskr-dev (bump for #297/#298 — cold recall closed end-to-end) |
| Worldtree version at pin | `v0.35.16` |
## Pin history
| Date | SHA | Version | Notable deltas consumed |
|---|---|---|---|
| 2026-05-26 | `562001a` | v0.29.0 | #201 — new SSE event `awaiting_llm_first_token` (heartbeat during BuildingPrompt → CallingLLM gap, default 5s interval) |
| 2026-06-17 | `f1b59f8` | v0.35.16 | **#297 + #298/#299 — Worldtree adopts the bifrost v0.6 scope wire (emits `scope_any`/`scope_all`) + client-side per-scope-value union recall. With our v0.17.6 provider this closes cold cross-session recall end-to-end.** Catch-up bump (v0.29.0→v0.35.16). Intervening client-facing deltas reviewed, none break our consumer: #211 agent rename (`saga``echo`, `actor``mask` — slugs only); #245 `end_user_id` persistence + memory-scope resolver; #187/#188/#219 Tier-3 define/PATCH policy (additive); `bifrost` binding field + `ephemeral_does_not_accept_bifrost` 422 now documented (the #17 surface). Error codes stable; no ratatoskr code change required. |
| 2026-05-25 | `da93ca7` | v0.28.0 | #204 — new SSE event `affect_update` (current/scheduled), new endpoint `GET /agents/{id}/persona_state`, auth-model doc edits |
| 2026-05-20 | `55101e9` | v0.19.0 | initial scaffold pin |
+175
View File
@@ -0,0 +1,175 @@
# Bifrost round-trip self-test
How to drive **and** observe a full Tier-3 Bifrost round-trip against
ratatoskr's *own* provider — bind a Worldtree session to our affect/memory
store, fire a turn, and read exactly what Worldtree dispatched to us,
correlated with the turn that triggered it.
This is the **manual form of issue #17** (self-drive + correlated-log
affect/memory ops). Until #17 ships that capability inside the TUI/web/CLI,
this runbook is the reproducible loop — and it is the substrate worldtree-dev's
#296 (salience-algorithm research) and #297 (recall scope/assembly research)
diagnose against.
> First proven by hand 2026-06-16 while isolating #295's recall leg.
## The load-bearing tripwire: which key signs the bind
A bound session-create authenticates as the **Bifrost consumer**, not the
canary client. Worldtree signs the Bifrost handshake JWT with the
**session-create bearer token** (HS256 shared-secret model); our provider
verifies it against `RATATOSKR_HEIMDALL_KEY`.
So the bearer on `POST /sessions` **must be the consumer Heimdall key**
(`RATATOSKR_HEIMDALL_KEY`, in `~/.config/ratatoskr/provider.env`) — **not**
`WORLDTREE_API_KEY` (the mimir/foundational TUI key in `env.sh`). They are two
different keys for two identities of ratatoskr.
| Bearer used on `POST /sessions` | Handshake at our provider |
|---|---|
| `WORLDTREE_API_KEY` (mimir/TUI key) | **401** `bifrost.auth_rejected` → 502 to client |
| `RATATOSKR_HEIMDALL_KEY` (consumer key) | **200 OK** → session bound |
ratatoskr is two identities: the conversation-API **canary client**
(`WORLDTREE_API_KEY`) and the Bifrost **consumer/provider**
(`RATATOSKR_HEIMDALL_KEY`). Self-driving a bound session crosses into the
consumer identity, so it uses the consumer key. #17's Bind half has to carry
this distinction.
## Prereqs
- Provider(s) running on this box (nh3-dev, `10.100.10.50`):
- memory → `ratatoskr-memory-provider` on `:8391`
- affect → `ratatoskr-provider` on `:8390`
- Each is a dev background shell, env-sourced from `provider.env`. The memory
provider's stdout carries the inbound observe log (`[memory-provider]` lines
added in `memory_store.py`).
- The provider endpoint is reachable + allowlisted from Worldtree
(`10.250.50.152`): `http://10.100.10.50:8391`. The allowlist
(`BIFROST_CLIENT_ALLOWED_HOSTS`) is **Worldtree-side, infra-ops-owned** — if a
bind 502s with a route/allowlist error, that entry is the thing to check.
- A memory-enabled Tier-3 agent defined on the instance. `ratatoskr:smoke`
(scope `end_user:smoke-user`) is the standing fixture; it is hidden from
`GET /agents` (Tier-3 agents are not in the public roster) but resolves on
session-create.
## Steps
```bash
cd ~/development/ratatoskr
set -a && . ~/.config/ratatoskr/provider.env && set +a # RATATOSKR_HEIMDALL_KEY etc.
URL=http://10.250.50.152:8081 # personal Worldtree
HK="$RATATOSKR_HEIMDALL_KEY" # the CONSUMER key — the tripwire
```
**1 — Bind a fresh (cold) session to our provider.** `bifrost.endpoint_url`
points at the plane you want (`:8391` memory, `:8390` affect); caps are
negotiated by the handshake, not declared here (`BifrostBindingRequest` is
`{endpoint_url, scope}` only, `additionalProperties:false`). A 201 means the
handshake verified.
```bash
curl -sS -X POST "$URL/sessions" -H "Authorization: Bearer $HK" \
-H "Content-Type: application/json" \
-d '{"agent_id":"ratatoskr:smoke","end_user_id":"smoke-user",
"bifrost":{"endpoint_url":"http://10.100.10.50:8391","scope":null}}'
# -> 201 {"session_id":"...", "kind":"consumer_defined", ...}
```
**2 — Snapshot the fixture** (to detect any promotion the turn writes):
```bash
sqlite3 -readonly memory.db \
"SELECT chunk_id, json_extract(record_json,'\$.verbatim.text') FROM memory_chunks;"
```
**3 — Fire ONE turn** into the bound session, reusing ratatoskr's own SSE
client (handles composite ids + no-read-timeout). Bearer = the consumer key:
```bash
RATATOSKR_HEIMDALL_KEY="$HK" uv run python - <<'PY'
import asyncio, os, httpx
from ratatoskr.sse_client import stream_turn, Text, Done, Error, Cancelled
SESSION="<session_id from step 1>"
URL="http://10.250.50.152:8081"; KEY=os.environ["RATATOSKR_HEIMDALL_KEY"]
async def main():
async with httpx.AsyncClient(base_url=URL,
headers={"Authorization":f"Bearer {KEY}","User-Agent":"ratatoskr-selftest"},
timeout=httpx.Timeout(connect=10.0,read=None,write=10.0,pool=10.0)) as c:
async for ev in stream_turn(c, SESSION, "What kind of chocolate do I like?"):
if isinstance(ev, Done): print("ANSWER:", ev.response); return
if isinstance(ev, Error): print("ERROR:", ev.error_code, ev.message); return
asyncio.run(asyncio.wait_for(main(), 150))
PY
```
**4 — Read the inbound pair** from the provider's stdout (the observe brick).
For a background-shell provider, that is the task output file; tail it:
```
[memory-provider] memory-call search REQUEST: scope_all={...} scope_any=[...] top_k=... vec_dim=1024
[memory-provider] memory-call search RESPONSE: N hit(s) [{'chunk_id':..., 'score':..., 'scope':...}]
```
**5 — Re-snapshot the fixture** (step 2's query). A new row = the turn was
promoted (a salience-algorithm event; relevant to #296). Audit, don't blindly
delete — promoted failure-surfaces may be wanted corpus.
## Reading the result
The `search REQUEST` `scope_all`/`scope_any` vs the `search RESPONSE` hit count is
the whole diagnosis surface:
- **0 hits** → the composed v0.6 filter matched no stored chunk. `scope_all` axes
are AND-matched — an extra axis the chunks don't carry (e.g. `agent_self`) zeroes
the result even when `end_user` matches. `scope_any` is the OR/union escape hatch:
a subset-scoped chunk recalls if its scope ⊇ **any one** element. So 0 hits with a
populated store now means Worldtree sent an over-specified `scope_all` instead of a
`scope_any` union — a **scope-build** question (Worldtree-side, post-v0.6).
- **Hit present but the model says "no memory"** → we returned it; Worldtree
dropped it downstream of search → **recall-assembly / injection**
(Worldtree-side).
Either way our store + search are provable from this surface; the recall
*efficacy* must be judged at the model's answer in a **cold (history-free)**
session, never from a wire 200 (a `search` returns 200 whether or not its hits
are injected).
### Worked example (2026-06-16, #295 → #297)
Cold turn "What kind of chocolate do I like?" against the `smoke-user` fixture:
```
REQUEST: scope_filter={'end_user': 'smoke-user', 'agent_self': 'ratatoskr:smoke'} top_k=128
RESPONSE: 0 hit(s)
ANSWER: "I don't have access to your past preferences..."
```
(That capture is the **pre-v0.6 wire** — a single AND-only `scope_filter`.)
Root cause: the recall filter carried `agent_self` but the stored chunks are
`{end_user: smoke-user}` only → the `agent_self` axis excluded all of them.
Branch (a), scope asymmetry — fed to #297.
**Resolution (bifrost 0.8.0 / wire v0.6, #11):** the single `scope_filter` is split
into `scope_all` (AND) + `scope_any` (OR/union). Worldtree can now send the visible
scopes as a `scope_any` union (e.g. `[{end_user: smoke-user}, {end_user: smoke-user,
agent_self: ...}]`), so the subset-scoped chunk recalls via the matching OR member.
Our store implements this at parity with the v0.6 reference; Worldtree **adopted the
v0.6 wire and now emits `scope_any`** on the recall path (#297 client-side union recall
+ #298/#299 bifrost-v0.6 adoption, v0.35.16), so cold cross-session recall is **closed
end-to-end** — pending a live re-smoke against a personal instance running v0.35.16.
## Notes / foot-guns
- **HTTP, not HTTPS.** The spec requires `endpoint_url` be HTTPS; dev is relaxed
via the Worldtree-side allowlist. Don't "fix" our provider to HTTPS to make a
bind work — check the allowlist entry first.
- **Cold means cold.** Reuse of a session with history can satisfy a "recall"
from plain conversation history. Always bind a *fresh* session for a recall
probe.
- **Stray sessions** created by probes are harmless empty rows on the dev
instance; no cleanup required.
- This loop is the thing #17 productizes into the chat surfaces; when #17 lands,
the bind+observe steps move inside the TUI/web/CLI and this runbook becomes the
underlying contract check.
@@ -22,6 +22,16 @@ external_invariants:
- source: ~/development/bifrost/docs/implementing-a-consumer.md
invariant_id: "§5 memory plane"
revisions:
- version: "1.2"
at: 2026-06-16
summary: "Repin bifrost 0.7.0→0.8.0 (wire v0.5→v0.6, #11): search `scope_filter` split into `scope_all` (AND/intersection) + `scope_any` (OR/union over a list of conjunctive scopes). No-compat: `scope_filter` removed. Adds union-visibility recall in one call — the fix for the #295/#297 AND silent-zero foot-gun. Store at parity with the v0.6 reference `_matches_scope` / `_validate_scope`."
delta:
MODIFIED:
- "search signature: scope_filter -> scope_all + scope_any"
- "INV-005 scope isolation -> composed v0.6 (scope_all AND ∧ scope_any OR-union)"
- "PRE-003 validates axes in BOTH fields; STEP 1 = _validate_scope (shape + lattice)"
ADDED:
- "scope_any_union test (#295/#297 union capability); both-fields-empty match-all"
- version: "1.1"
at: 2026-06-15
summary: "Heid-contract-review fixup: semantic-not-byte-equal round-trip; reconcile idempotency 4-tuple; search returns top_k IN-SCOPE; define recalled_view + scope_filter + named field keys inline; clarify metadata_filter-v1 + transaction-term + delete atomicity + get_many + revision-on-replay; drop scan from INV-005."
@@ -98,9 +108,13 @@ interpreted.
- **INV-004** [hard]: **Atomic batch.** `upsert_many` applies all records + their
vec rows + the idempotency record in one transaction; on any error nothing is
persisted (no partial batch, no orphaned vec rows).
- **INV-005** [hard]: **Scope isolation.** `search` results are filtered to records
whose `record["scope"]` matches every axis in `scope_filter`; a search never
returns another scope's chunk.
- **INV-005** [hard]: **Scope isolation (wire v0.6, #11).** `search` filters by two
explicit fields: `scope_all` (AND/intersection — record ⊇ every named axis) and
`scope_any` (OR/union over a LIST of conjunctive scope dicts — record ⊇ ≥1 element,
each element AND-matched as a whole). They compose by AND; both empty → no scope
constraint. A search never returns a chunk outside the composed filter. Byte-faithful
to the reference `_matches_scope`. (`scope_any` is the union-visibility primitive that
resolves the #295/#297 silent-zero — a subset-scoped chunk now recalls via an OR member.)
- **INV-006** [hard]: **Capabilities match implementation** (advertise-⇒-implement).
`describe_store` advertises ONLY what v1 implements: `relational_edges_supported=False`,
`atomic_supersede_supported=False`, `transaction_supported=False`,
@@ -205,20 +219,23 @@ TESTS:
```
```contract
FN search(self, vector: list[float], *, top_k: int, scope_filter: dict | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped, returning the top_k IN-SCOPE chunks.
FN search(self, vector: list[float], *, top_k: int, scope_all: dict | None = None, scope_any: list | None = None, metadata_filter: dict | None = None, include: dict | None = None, fidelity_target=None) -> list[dict]
BRIEF: Vector (cosine) recall over sqlite-vec, scoped by the v0.6 scope_all/scope_any filter, returning the top_k IN-SCOPE chunks.
PRE: [PRE-001 hard] len(vector) == embedding_dim -- else InvalidArguments
PRE: [PRE-002 hard] metadata_filter is empty/None -- v1 advertises no filterable fields; a non-empty filter → InvalidArguments
POST: [POST-001 return_value] returns the top_k highest-cosine records WHOSE scope matches scope_filter — at most top_k, and never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
PRE: [PRE-003 hard] scope_all is a flat dict and scope_any a list of flat dicts (else InvalidArguments); every axis in BOTH ∈ {end_user, group, tenant, agent_self} -- else InvalidFilter (memory.invalid_filter 400); the bifrost wire-v0.6 lattice, matching the reference _validate_scope (#10 agent_self canonical, #11 scope split)
POST: [POST-001 return_value] returns the top_k highest-cosine records passing the composed v0.6 filter — `(scope_all empty OR record ⊇ scope_all) AND (scope_any empty OR record ⊇ ≥1 element)`; at most top_k, never fewer than min(top_k, in-scope count) (INV-005). Each: {chunk (verbatim), chunk_id, score, recalled_view (= chunk["distillate"] or chunk), revision} -- assert
STEPS:
1. [setup] validate scope_filter is a flat {axis: value} dict (matched against record["scope"][axis])
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only scope-matching records (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
1. [setup] scope_all ← scope_all or {}; scope_any ← scope_any or []; validate via _validate_scope (flat-dict / list-of-dicts shape + every axis ∈ the v0.6 lattice, else InvalidArguments / InvalidFilter)
2. [sequential, flexibility=indicative] rank candidates by cosine over record["embedding"]; KEEP only records passing _matches_scope(scope_all, scope_any) (INV-005); THEN take top_k — so top_k counts IN-SCOPE hits, not pre-filter hits (over-fetch from the vec index or post-filter rank as needed)
3. [cleanup] RETURN result rows (chunk verbatim + score + recalled_view + revision)
TESTS:
basic_search [happy,tracer]: upsert 3 scoped chunks, search → ranked by cosine, ≤ top_k, recalled_view present
scope_isolation [adversarial]: two scopes, search one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
empty [boundary]: search empty store → []
scope_isolation [adversarial]: two scopes, scope_all one → never returns the other's chunk, and returns top_k of the IN-SCOPE set even if out-of-scope chunks score higher (INV-005)
scope_any_union [scenario]: scope_any=[{end_user:u},{agent_self:a}] recalls BOTH a subject-scoped and a self-scoped chunk in one call (#295/#297 union capability); scope_all+scope_any compose by AND
empty [boundary]: search empty store → []; both fields empty → match all
metadata_filter_rejected [adversarial]: non-empty metadata_filter → InvalidArguments
lattice_axes [adversarial]: out-of-lattice axis in scope_all OR scope_any → InvalidFilter; non-list scope_any → InvalidArguments; agent_self admitted (wire v0.5, #10)
parity_vs_reference [scenario]: identical search envelopes vs InMemoryMemoryStore → same ranked chunk_ids/shape (#195)
```
+312
View File
@@ -0,0 +1,312 @@
---
contract_version: "2.1"
target_module: "ratatoskr.sessions + ratatoskr.provider (+ cli/tui/web trigger surfaces)"
scope: "Issue #17 v1 — make the canary chat client self-drive AND observe its own Bifrost provider. Two parts. (1) BIND: `create_session` gains an optional single-plane Bifrost binding (`BifrostBinding{endpoint_url, scope}`) authenticated with a DISTINCT consumer Heimdall key; Worldtree runs the handshake synchronously at POST /sessions, so handshake failure is a session-create failure (502), surfaced on the create path. A plane selector (`memory`→:8391 / `affect`→:8390) + the consumer key thread through CLI / TUI / web; bound-state is visible. (2) OBSERVE: a structured op-feed in the provider, instrumented at the DISPATCH/ASGI layer (where the JWT ctx / session_id lives — bifrost passes ctx to upsert_many but NOT to search/get/delete, so the existing store-method stdout shim cannot see session_id), emitting JSONL {session_id, plane, op, req_summary, resp_summary, status, ts}. OPERATOR DECISIONS LOCKED: single-plane-per-session for v1 (composite endpoint fronting both planes is PARKED — vNext); op-feed with session-level correlation for v1 (turn-correlated debug-pane UI is PARKED — needs turn_id, TBD). Provider store scope semantics MUST NOT change (AND-parity with bifrost's reference store is a hard constraint). Direct in-session TDD; live-smoke against personal Worldtree is the load-bearing acceptance gate."
depends_on:
- "httpx"
- "ratatoskr.sessions"
- "ratatoskr.provider.memory_store"
- "ratatoskr.provider.affect_store"
- "bifrost"
used_by:
- "ratatoskr.cli"
- "ratatoskr.tui"
- "ratatoskr.web.server"
language: "python"
complexity: "medium"
estimated_loc: 260
confidence: 0.78
assumptions:
- "PROVEN this session (manual end-to-end against personal Worldtree v0.35.3): `POST /sessions` with `bifrost={endpoint_url, scope:null}` runs the handshake synchronously and returns 201 when it verifies. `BifrostBindingRequest` is `{endpoint_url, scope}` ONLY (live OpenAPI, additionalProperties:false); capabilities are negotiated at the handshake, not declared in the bind request. So a session binds exactly ONE provider endpoint → ONE plane's dispatch flows."
- "PROVEN: the session-create BEARER is the identity Worldtree signs the Bifrost handshake JWT with (HS256 shared-secret). Bearer = the canary key (WORLDTREE_API_KEY) → handshake 401 `bifrost.auth_rejected` → 502 to the client. Bearer = the consumer Heimdall key (== the provider's RATATOSKR_HEIMDALL_KEY string) → handshake 200. So a BOUND session-create MUST authenticate with the consumer key, NOT the canary key. These are two distinct ratatoskr identities."
- "PROVEN: dev HTTP is accepted (spec wants HTTPS) because the provider host:port is on Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist — a Worldtree-side, infra-ops-owned config. The endpoint_url must be the WORLDTREE-VISIBLE base URL (e.g. `http://10.100.10.50:8391`), not the client's loopback. Provider routes live at `/bifrost/handshake` + `/bifrost/memory-call` (memory) and `/bifrost/affect-call` (affect) under that base."
- "PROVEN (bifrost source, memory.py:244 vs 262): `dispatch_memory_call` passes `ctx` to `upsert_many` but NOT to `search`/`get`/`delete`. So the recall verb's store method has no session_id; correlation identity must be captured at the dispatch/ASGI layer (JWT ctx), not inside the store method. turn_id (finer than session_id) availability is UNVERIFIED — a contract-stage JWT-claims/envelope dump resolves it; design the op-feed to accept a turn_id later without a schema break."
- "Provider stores MUST NOT change scope semantics. `_matches_scope` stays the v0.6 composed filter — `scope_all` (AND/intersection) ∧ `scope_any` (OR/union over conjunctive scopes) — byte-faithful to bifrost reference `reference_server/memory.py` (wire v0.6, #11); the 4-axis lattice validation (`_validate_scope`) is in place and at parity. Scope semantics are settled (the v0.6 scope split shipped, bifrost 0.8.0) and OUT OF SCOPE for #17 — observe is read-only over them."
- "The existing `create_session(client, agent_id, *, end_user_id=None)` (sessions.py:179) is extended, not replaced (pre-v1, no compat shim). The httpx client carries the canary key as its default Authorization; the bound create overrides the bearer per-request with the consumer key."
- "Tests use `respx` for the bind unit tests (mirroring tests/test_sessions.py) + the in-process op-feed; the live-smoke acceptance is manual (per the repo's load-bearing-smoke posture), captured as a documented runbook step, not a unit test. `docs/bifrost-self-test.md` is the manual procedure this feature productizes."
- "v1 ships the CLI + TUI + web trigger surfaces in lockstep (the repo's BOTH-presenters-in-lockstep rule); the op-feed is read by the debug surface as structured lines for now (pane-correlated rendering is PARKED)."
open_questions:
- "turn_id on the wire: does Worldtree put a turn_id (or traceparent) in the Bifrost request JWT claims / envelope? If yes, the op-feed echoes it and turn-correlation becomes exact; if no, correlation is session_id + time-window (weaker). RESOLVE via a JWT-claims dump during the first TDD slice. Does NOT block v1 (session_id correlation is the v1 target); it gates the PARKED pane-UI."
- "Composite endpoint (PARKED): a thin ASGI fronting both :8390/:8391 advertising both caps at handshake would let one session observe both planes. Deferred to vNext per operator. If pursued, it needs per-underlying-store parity checks + per-plane failure status (a facade routing bug is a new failure class) — NOT just `handshake lists both caps`."
- "Key identity unification (PARKED — operator's call, crosses Heimdall): v1 assumes two keys. Do not derive one from the other."
- "Auto-bind on Tier-3 agents (PARKED — operator's call): v1 is explicit opt-in only. Auto-bind hides the most important debug variable (which identity + endpoint the session bound to)."
prd:
issue: 17
issue_url: "https://gitea.phasefinal.com/vh/ratatoskr/issues/17"
body_sha256_16: "58a420956e6226fb"
lock_in_comment_id: null
lock_in_sha256_16: null
lock_in_at: null
pinned_at: "2026-06-16T07:45:00+00:00"
dependencies:
- issue: 2
path: "src/ratatoskr/sessions.py"
reason: "create_session is the bind site. Same posture: caller-owned httpx client, async-native, no Worldtree imports, frozen-dataclass parse, exception `.body` truncated to [:1024]."
- issue: 5
path: "src/ratatoskr/sessions.py"
reason: "end_user_id already threads into the POST /sessions body; the bifrost field is added alongside it with the same optional-when-None posture."
---
# Issue #17 — Bifrost-binding the chat client: self-drive + observe
## Context
Ratatoskr is two identities: the conversation-API **canary client** (TUI/web/CLI
that runs turns and watches the SSE flow) and a Bifrost **Tier-3 provider**
(durable affect store :8390 + memory store :8391, separate ASGI apps). Until now
the canary couldn't drive its OWN provider — `create_session` never sent a Bifrost
binding, so every affect/memory round-trip was driven externally. #17 closes that:
the canary BINDS a session to its own provider and OBSERVES the resulting
affect/memory dispatch, so an operator can hunt latent cross-layer bugs from one
seat. The manual procedure proven this session lives at `docs/bifrost-self-test.md`;
#17 productizes it.
**v1 scope is deliberately narrow** (operator-locked): single-plane bind (composite
endpoint PARKED), session-level op-feed (turn-correlated panes PARKED). The load-
bearing risks are bind-time auth identity + capturing a correlation key the store
method can't see — both resolved below.
## Public surface
```python
@dataclass(frozen=True)
class BifrostBinding:
"""Session-create Bifrost binding (Worldtree BifrostBindingRequest, #160).
endpoint_url is the WORLDTREE-VISIBLE base URL of one provider plane."""
endpoint_url: str
scope: str | None = None
async def create_session(
client: httpx.AsyncClient,
agent_id: str,
*,
end_user_id: str | None = None,
bifrost: BifrostBinding | None = None,
consumer_key: str | None = None,
) -> SessionInfo:
"""POST /sessions. When `bifrost` is set the request authenticates with
`consumer_key` (NOT the client's default canary bearer) and carries the
`bifrost` field; Worldtree handshakes synchronously before 201. See FN
create_session."""
def endpoint_for_plane(plane: str, base_host: str) -> str:
"""'memory'->:8391, 'affect'->:8390 → f'http://{base_host}:{port}'. The
Worldtree-visible base URL. See FN endpoint_for_plane."""
```
```python
# Provider-side observe feed (ratatoskr.provider.opfeed) — dispatch-layer.
@dataclass(frozen=True)
class OpEvent:
ts: str # ISO 8601 UTC, capture time
plane: str # "memory" | "affect"
op: str # verb: search / upsert_many / emit / get / delete / handshake
session_id: str | None # from the JWT ctx at the DISPATCH layer — present for ALL
# JWT-carrying verbs (not just upsert_many; bifrost withholds
# ctx from search/get/delete STORE methods, but dispatch sees
# the JWT); None only if the JWT genuinely omits it
status: str # "ok" | "error"
req_summary: dict # per-verb, scope-only (see "Op-feed summary shapes"); no record bodies
resp_summary: dict # per-verb counts + ids/scores; never verbatim content
turn_id: str | None = None # INV-005 reservation made LITERAL: the field exists now,
# unused in v1 (session-level correlation), populated when
# Worldtree propagates a turn id (open question)
def instrument_provider_app(app, *, plane: str, sink: OpSink):
"""Wrap the dispatch/ASGI layer so every inbound bifrost-call emits one
OpEvent to `sink`, reading session_id off the JWT ctx where bifrost exposes
it. Does NOT touch store scope semantics. See FN instrument_provider_app."""
```
## Exception classes
```python
class BifrostHandshakeFailed(Exception):
"""502 bifrost_handshake_failed on bound session-create. Carries the
spec-level `detail.bifrost_error` (e.g. 'bifrost.auth_rejected')."""
def __init__(self, *, bifrost_error: str | None, body: bytes) -> None: ...
bifrost_error: str | None
class BifrostConsumerKeyMissing(Exception):
"""A bifrost binding was requested without a consumer_key. Raised BEFORE
HTTP (the bind must never silently fall back to the canary key)."""
```
## Invariants
- **INV-001 (auth identity, never fall back).** A `bifrost` binding REQUIRES a
non-empty `consumer_key`; absence raises `BifrostConsumerKeyMissing` before any
HTTP. The bound POST /sessions authenticates with `consumer_key`; an unbound
create authenticates with the client's default canary key. The two call sites
never cross. On 401-rooted handshake failure the error names the mismatch.
- **INV-002 (bind-time, not turn-time, failure).** The handshake runs
synchronously on POST /sessions. A bad URL / down provider / wrong key / HTTPS
rejection fails SESSION CREATION (502 → `BifrostHandshakeFailed`), surfaced on
the create path BEFORE any turn / before alt-screen (TUI) — never deferred to
first-turn. Mirrors issue #6's pre-alt-screen error routing.
- **INV-003 (one plane per session).** A binding targets exactly one endpoint =
one plane. v1 documents this limit explicitly; binding both planes for one turn
is the PARKED composite-endpoint feature, not a v1 path.
- **INV-004 (no scope-semantics change).** The observe instrumentation is
READ-ONLY over the dispatch path; it MUST NOT alter `_matches_scope`, the v0.6
`scope_all`/`scope_any` semantics, or any store behavior. The op-feed reports the
effective scope used per op; it never rewrites scope client-side.
- **INV-005 (correlation key at the dispatch layer).** session_id is captured from
the JWT ctx at the dispatch/ASGI layer — present for ALL JWT-carrying verbs,
INCLUDING search/get/delete (bifrost withholds ctx from those STORE methods, but
the dispatch layer still verifies + reads the JWT). `session_id=None` ONLY if the
JWT genuinely omits it (a claims-dump open question, not the store-method gap).
`OpEvent` carries a literal `turn_id: str | None = None` field — the reservation
is a real field defaulted to None in v1, not a future schema change.
- **INV-006 (data hygiene).** Bound debug sessions write to DURABLE stores. The
smoke procedure uses an explicit test scope (`end_user:smoke-user`) and a
documented cleanup path; the contract's acceptance asserts the fixture
before/after so a debug run's promotions are visible, never silent. (Promotion
of a turn is expected behavior, not a bug — but it must be observable.)
- **INV-007 (observe captures failures + late ops).** The op-feed records non-2xx
/ error ops (status="error"), never hides or double-counts them. The `OpSink` is
a CONTINUOUS append-only feed — NOT per-session-scoped, no per-session teardown in
v1; late ops that land AFTER the SSE turn-end simply append with their timestamp,
so a consumer can apply a post-turn grace window (the PARKED pane-UI's concern; v1
just must not drop late ops). A sink write that FAILS is swallowed from the
dispatch path (instrument_provider_app POST-003) BUT logged to stderr — an observe
gap is never silent.
- **INV-008 (both presenters in lockstep).** The bind trigger + bound-state
indicator land in CLI, TUI, and web together (the repo's add-to-all-presenters
rule). Web creates the bound session SERVER-SIDE; the consumer key never reaches
the browser.
- **INV-009 (secret hygiene).** Both keys are redacted in all UI/logs. The
consumer key is PRIVILEGED (it is the handshake identity), not equivalent to
read-only conversation access; config that stores it uses no weaker file
permissions than the canary key (cf. provider.env mode 600).
## Data flow
BIND (client → Worldtree): CLI/TUI/web resolve `plane` + the consumer key →
`endpoint_for_plane(plane, worldtree_visible_host)``BifrostBinding`
`create_session(..., bifrost=binding, consumer_key=...)` → POST /sessions with
bearer = consumer key → Worldtree handshakes synchronously to the provider
endpoint → 201 (bound) or 502 (`BifrostHandshakeFailed`).
OBSERVE (Worldtree → provider → feed): a bound turn makes Worldtree dispatch
affect/memory bifrost-calls to the provider → `instrument_provider_app` wraps the
dispatch layer → one `OpEvent` per call (session_id from JWT ctx when present) →
`OpSink` (JSONL) → the debug surface reads structured lines. The store path is
untouched (INV-004); observe is read-only over dispatch.
## Function contracts
```contract
FN create_session(client: httpx.AsyncClient, agent_id: str, *, end_user_id: str | None = None, bifrost: BifrostBinding | None = None, consumer_key: str | None = None) -> SessionInfo
BRIEF: POST /sessions; when a bifrost binding is given, authenticate with the consumer key (not the canary key) and carry the binding so Worldtree handshakes synchronously to our provider.
PRE: [PRE-001 hard] bifrost is not None ⇒ consumer_key is a non-empty str -- else BifrostConsumerKeyMissing, before any HTTP (INV-001)
PRE: [PRE-002 soft] bifrost is None ⇒ request uses the client's default canary bearer -- unchanged pre-#17 path
POST: [POST-001 return_value] 201 → SessionInfo (unchanged parse) -- assert
POST: [POST-002 exception] 502 → BifrostHandshakeFailed(bifrost_error=detail.bifrost_error, body) (INV-002) -- assert
POST: [POST-003 exception] 422 ephemeral_does_not_accept_bifrost → SessionApiFailed -- assert
POST: [POST-004 exception] 404 → AgentNotFound; other non-201 → SessionApiFailed -- assert (unchanged)
STEPS:
1. body = {"agent_id": agent_id}; if end_user_id: body["end_user_id"] = end_user_id
2. if bifrost: body["bifrost"] = {"endpoint_url": bifrost.endpoint_url, "scope": bifrost.scope}; headers = {"Authorization": f"Bearer {consumer_key}"}
3. else: headers = {} (httpx client default bearer = canary key)
4. resp = await client.post("/sessions", json=body, headers=headers); route status per POST-*
```
```contract
FN endpoint_for_plane(plane: str, base_host: str) -> str
BRIEF: Map a plane name to the Worldtree-visible provider base URL (memory->:8391, affect->:8390).
PRE: [PRE-001 hard] plane in {"memory", "affect"} -- else ValueError
POST: [POST-001 return_value] returns f"http://{base_host}:{port}", port 8391 (memory) / 8390 (affect) -- assert
STEPS:
1. port = 8391 if plane == "memory" else 8390
2. return the Worldtree-VISIBLE base URL (not client loopback); HTTPS relaxation is allowlist-side, not a URL concern
```
```contract
FN instrument_provider_app(app, *, plane: str, sink: OpSink) -> ASGIApp
BRIEF: Wrap the provider's dispatch layer so each inbound bifrost-call emits one structured OpEvent (session_id from the JWT ctx) without touching store semantics.
PRE: [PRE-001 hard] app is a built provider ASGI app; sink is an OpSink -- guard
POST: [POST-001 side_effect] emits exactly one OpEvent per inbound bifrost-call, incl. handshake + error ops (INV-007) -- assert
POST: [POST-002 state_change] OpEvent.session_id = JWT ctx session_id when present, else None (INV-005) -- assert
POST: [POST-003 side_effect] a sink failure never propagates into the dispatch path — observe must not break serve -- assert
POST: [POST-004 return_value] store scope semantics untouched; read-only over dispatch (INV-004) -- assert
STEPS:
1. wrap the dispatch/ASGI layer so each inbound bifrost-call yields one OpEvent
2. read session_id off the JWT ctx if present; else None
3. summarise req (scope_all/scope_any/top_k for search; record-count+scopes for upsert) + resp (hit-count+ids/scores | upserted+replayed | error code) — NEVER verbatim content
4. emit to sink; swallow sink errors
```
## ERROR_ROUTING
| Wire | Exception | Surfaced |
|---|---|---|
| 502 `bifrost_handshake_failed` | `BifrostHandshakeFailed(bifrost_error)` | create path, names the bifrost_error; TUI pre-alt-screen |
| (pre-HTTP) bifrost w/o consumer_key | `BifrostConsumerKeyMissing` | config/CLI validation, before any request |
| 422 `ephemeral_does_not_accept_bifrost` | `SessionApiFailed` | create path |
| 401 at provider handshake (manifests as 502 to client) | `BifrostHandshakeFailed('bifrost.auth_rejected')` | error text: "bound create requires the consumer key, not WORLDTREE_API_KEY" |
## Acceptance — the ordered live-smoke gate (load-bearing)
The repo's smoke-is-load-bearing posture: this gate IS acceptance, run manually
against personal Worldtree, mirroring `docs/bifrost-self-test.md`.
```
1. providers up → memory :8391 serving + op-feed sink attached (affect :8390 is symmetric, OPTIONAL for the memory-plane gate)
2. consumer key set → RATATOSKR_BIFROST_CONSUMER_KEY present; canary key separate
3. allowlist OK → endpoint_for_plane host on Worldtree's BIFROST_CLIENT_ALLOWED_HOSTS
4. bind 201 → ratatoskr --bifrost-plane memory → bound session, handshake 200
5. negative: canary → binding with the canary key → BifrostHandshakeFailed, the auth_rejected message names the consumer-key mismatch
6. turn → one turn into the bound session
7. assert op-feed → OpEvent captured with session_id == the BOUND session's id (not merely "some session_id"); {plane, op, req/resp summary, status}
8. assert hygiene → fixture before/after asserted (per docs/bifrost-self-test.md); promotions visible not silent
```
Unit tests (respx) cover: bind body shape, consumer-key override, the
missing-key precondition, 502→BifrostHandshakeFailed mapping, 422 ephemeral, and
the op-feed emitting one OpEvent per dispatched call incl. error + late op.
## v1 clarifications (paraphrase-gate fixups)
Pinned in response to the `/heid-contract-review` panel — closing under-specs that
let an implementer comply while violating intent.
- **Op-feed summary shapes (per verb).** `req_summary` / `resp_summary` are
scope-only, never verbatim content:
- `search` → req `{scope_all, scope_any, top_k}`; resp `{hit_count, [{chunk_id, score}]}`
- `upsert_many` → req `{record_count, [scope]}`; resp `{upserted, replayed}`
- `get` / `get_many` → req `{ids}`; resp `{found_count}`
- `delete_many` → req `{ids}`; resp `{deleted}`
- `emit` (affect) → req `{actor-scope}`; resp `{status}` (affect stays conduit-opaque)
- `handshake` → req `{caps_requested}`; resp `{caps_granted, ok}`
- **`BifrostBinding.scope` is an opaque pass-through** (Worldtree spec: ≤256 chars,
copied into the JWT payload unchanged). ratatoskr does NOT interpret it; v1 sends
`null`. A non-null value is operator-supplied and meaningful only to Worldtree.
- **Web bind split (INV-008 sharpened).** The web UI selects the PLANE; the consumer
key is SERVER-HELD (env/config), never sent from the browser; the server
constructs the bound session. The browser never sees the consumer key.
- **Bound-state indicator** shows at least `plane + endpoint + bound|failed status`,
not a bare boolean (so the operator can see WHICH identity/endpoint bound).
- **401-handshake message scoping.** The "use the consumer key, not WORLDTREE_API_KEY"
text is keyed on `bifrost_error == "bifrost.auth_rejected"`; other 502 handshake
failures surface the generic `BifrostHandshakeFailed` with their own `bifrost_error`.
- **`endpoint_for_plane` is the DEV helper** (returns `http://`, allowlist-relaxed).
A production HTTPS endpoint is supplied directly via `--bifrost-url`, bypassing the
plane shortcut — HTTPS is not constructed by `endpoint_for_plane`.
- **422 `ephemeral_does_not_accept_bifrost``SessionApiFailed` is deliberate** (no
distinct exception; it is an operator config error, surfaced as a generic create
failure). Not an oversight in the ERROR_ROUTING overlap with POST-004.
## Out of scope / PARKED (anti-creep)
- Composite endpoint (both planes, one session) — vNext; needs per-store parity + per-plane failure status.
- Turn-correlated debug-pane UI — needs turn_id (open question) + grace-window buffering + client read channel.
- Key identity unification (Heimdall) — operator's call, crosses service boundary.
- Auto-bind on Tier-3 agents — operator's call; v1 is explicit opt-in only.
- Provider axis-lattice validation + the v0.6 scope split — RESOLVED (shipped: bifrost 0.8.0/wire v0.6, `_validate_scope` 4-axis + `scope_all`/`scope_any`); no longer a #17 concern.
+142 -21
View File
@@ -61,7 +61,7 @@ When you call `POST /sessions` against an agent, the authorization check that fi
### Tier 1 — foundational agents (no `:` in agent_id)
Agents bundled with Worldtree: `mimir`, `lofn`, `soong`, `forseti`, `domari`, `vili`, `actor`, `saga`, `bragi`, `leif`, `troi`, `cara`, `glados`, and any future Asgardian. The agent_id is a simple slug like `mimir` — no colon.
Agents bundled with Worldtree: `mimir`, `lofn`, `forseti`, `domari`, `vili`, `mask`, `echo`, `muninn`, and any future Asgardian. The agent_id is a simple slug like `mimir` — no colon.
> **About tiers:** Your `tier` is set on the `users` table row your API key resolves to, assigned at key-mint time (see `POST /admin/keys`). Tiers are `anonymous` (dev-mode unauthenticated), `user` (default for newly-issued keys), `free`/`pro` (subscription-shaped, not actively differentiated), and `admin`. The tier you have is visible via `GET /me`'s `tier` field. Tier-derived scopes come from `config/policies.yaml > tiers.<tier>.scopes` — there is no per-key scope override.
@@ -989,7 +989,7 @@ Create a new conversation session with an agent.
**Bifrost field validation:**
- `endpoint_url`: required, must be an HTTPS URL.
- `scope`: optional, ≤ 256 chars, opaque string passed through to the JWT payload unchanged.
- Bifrost binding is **incompatible with ephemeral (Saga) sessions** — returns 422 `ephemeral_does_not_accept_bifrost`.
- Bifrost binding is **incompatible with ephemeral (Echo) sessions** — returns 422 `ephemeral_does_not_accept_bifrost`.
- Requires the `bifrost:invoke` scope (included in the `user` tier by default).
**Response:** `201 Created`
@@ -1535,7 +1535,7 @@ for (const tc of items) {
Ephemeral templates are a second tier of agent, distinct from foundational persistent agents (Mimir, Soong, etc.). They have no persona, no memory, no tools, and no motivational context. The consumer supplies the system prompt and (optionally) the model at session-create time; that config is frozen for the session's lifetime.
**Saga** is the first ephemeral template — Norse goddess of history and chronicle, a blank-slate actor that becomes whatever the consumer's system prompt instills.
**Echo** is the first ephemeral template — a blank-slate per-session host that becomes whatever the consumer's system prompt instills.
### Discovering available templates
@@ -1547,7 +1547,7 @@ Authorization: Bearer <any valid key>
```json
{
"ephemeral_templates": {
"saga": {
"echo": {
"allowed_models": ["glm5-turbo", "glm4.7", "glm4.5-air", "granite-structured", "qwen3.6-35-a3b"],
"default_model": "glm5-turbo",
"system_prompt_max_bytes": 32768
@@ -1556,14 +1556,14 @@ Authorization: Bearer <any valid key>
}
```
`GET /capabilities` does not require `instantiate:saga` scope — any authenticated caller can read what's available before deciding to instantiate.
`GET /capabilities` does not require `instantiate:echo` scope — any authenticated caller can read what's available before deciding to instantiate.
### Creating an ephemeral session
```json
POST /sessions
{
"agent_id": "saga",
"agent_id": "echo",
"config": {
"system_prompt": "You are a careful, skeptical frame-clarifier...",
"model": "glm5-turbo"
@@ -1580,16 +1580,16 @@ POST /sessions
| `system_prompt_required` | `config.system_prompt` missing or null |
| `system_prompt_empty` | `config.system_prompt` is whitespace-only |
| `system_prompt_too_large` | `config.system_prompt` > 32768 bytes UTF-8 |
| `model_not_allowed` | `config.model` present but not in `saga_allowed_models` |
| `model_not_allowed` | `config.model` present but not in `echo_allowed_models` |
**`config.model` resolution:** When `config.model` is omitted (or `null`), the server resolves it to `saga.default_model` from `config/defaults.yaml`. The resolved value is always populated in the session snapshot; `model` is never left absent or null in the stored config.
**`config.model` resolution:** When `config.model` is omitted (or `null`), the server resolves it to `echo.default_model` from `config/defaults.yaml`. The resolved value is always populated in the session snapshot; `model` is never left absent or null in the stored config.
**Response:** Same 201 shape as foundational sessions, with two new fields:
```json
{
"session_id": "...",
"agent_id": "saga",
"agent_id": "echo",
"kind": "ephemeral",
"config": {
"system_prompt": "You are a careful, skeptical frame-clarifier...",
@@ -1601,7 +1601,7 @@ POST /sessions
}
```
**`kind` field:** `"ephemeral"` for Saga sessions, `"foundational"` for all other sessions. Present on both `GET /sessions` list items and `GET /sessions/{id}`.
**`kind` field:** `"ephemeral"` for Echo sessions, `"foundational"` for all other sessions. Present on both `GET /sessions` list items and `GET /sessions/{id}`.
### Sending messages to an ephemeral session
@@ -1617,9 +1617,9 @@ SSE, cancel, `persist_partial`, rate limits, and error shapes are bit-identical
### Scope
Creating a Saga session requires the `instantiate:saga` scope. This scope is bundled in the `user` tier. Tier `admin` inherits it via the wildcard.
Creating an Echo session requires the `instantiate:echo` scope. This scope is bundled in the `user` tier. Tier `admin` inherits it via the wildcard.
### What Saga does NOT do
### What Echo does NOT do
- No persona injection (`PersonaRegistry.inject_context` not called)
- No post-turn appraisal (`PersonaRegistry.update_after_turn` not called)
@@ -2415,7 +2415,7 @@ The `POST /sessions/{session_id}/messages` endpoint also accepts an additive `mo
- Override is per-call only. Stored `CharacterSchema.model` is NOT mutated.
- Validated against the same `available_for_characters` allowlist that gates `CharacterSchema.model` at create time (#153 INV-091).
- Override displaces the character's bound model when both are set (per-call wins).
- Override is REJECTED on ephemeral (Saga) sessions — their config is frozen at session-create per INV-161-2.
- Override is REJECTED on ephemeral (Echo) sessions — their config is frozen at session-create per INV-161-2.
**Validation:**
1. Pydantic validates `model`: optional string, non-empty after stripping whitespace.
@@ -2672,7 +2672,7 @@ The override client has a fresh 25-call reentrancy budget, independent of the se
| Condition | HTTP | `error_code` | `bifrost_error` |
|-----------|------|-------------|----------------|
| `endpoint_url` is not HTTPS | 422 | `validation_failed` | — |
| Ephemeral (Saga) session | 422 | `validation_failed` | — |
| Ephemeral (Echo) session | 422 | `validation_failed` | — |
| Missing `bifrost:invoke` scope | 403 | `auth_scope_denied` | — |
| `consumer_id` not in Heimdall or not Bifrost-registered | 502 | `bifrost_consumer_not_found` | — |
| Handshake failed (network, auth, etc.) | 502 | `bifrost_handshake_failed` | spec error code |
@@ -2753,6 +2753,61 @@ Caller must:
`agent_name` is a strict slug `[a-z][a-z0-9-]{2,63}` and immutable
after definition.
The 201 response includes an advisory `warnings` array (#219) — see
"Model-assignment warnings" under `PATCH` below.
##### Motivational layer (Phase 2.2, #187)
`motivational` is **active** as of Phase 2.2 (persona + memory activated in
Phase 2.1; only `valence` still returns `layer_deferred`). It carries the
agent's goals + fears — the same substrate Tier 1 agents author in
`agents/<name>/motivation.yaml`:
```json
"motivational": {
"goals": [
{
"id": "successful_handoff",
"type": "achievement", // maintenance | achievement | avoidance
"salience": 0.85, // [0.0, 1.0]
"description": "You succeed when the user lands with the right specialist.",
"positive_signals": ["talk to mimir"], // optional
"negative_signals": ["stay with me"] // optional
}
],
"fears": [
{
"id": "specialist_displacement",
"salience": 0.90,
"description": "You fear being mistaken for the specialist the user needs.",
"trigger_signals": ["actually mimir would"] // optional (NB: fears use trigger_signals)
}
]
}
```
Semantics:
- **Per-agent, not per-(agent, end_user).** Goals/fears are an identity trait of
the agent — identical for every end-user and session.
- **Immutable post-define.** `PATCH` with `motivational` returns 422
`field_not_mutable`. To change motivations, define a new agent.
- **Rendered into the system prompt.** The config is captured on the session's
`AgentContext` at session-create and rendered into the prompt on each turn
(only goals/fears with `salience >= 0.5` surface). Tier 3 agents bypass the
persona registry; the render reuses the Tier 1 substrate so output is
identical to an equivalent Tier 1 `motivation.yaml`.
Validation rejects malformed payloads at define-time with these 422 codes:
`motivational_id_collision` (id duplicated across goals AND fears — case-sensitive),
`motivational_goal_invalid_type`, `motivational_salience_out_of_range`,
`motivational_description_too_short` (< 20 chars after strip),
`motivational_missing_required_field` (missing id / salience / description /
goal `type`). Unknown keys at the top level or inside a goal/fear object →
`validation_failed`. v0.1 exposes only the documented fields; advanced
`GoalConfig` knobs (`priority`, `resilient`, `completion_signal`, …) are not
consumer-settable yet.
#### `DELETE /agents/<user_id>:<agent_name>` — `204 No Content`
Owner-initiated hard-delete. Bypasses the 24h grace (distinct from the
@@ -2762,11 +2817,77 @@ session bound to this agent and revokes the owner's per-resource
#### `PATCH /agents/<user_id>:<agent_name>`
Phase 2.0 minimal: only `system_prompt` and/or `model` may be patched.
Any other key (including the immutable `agent_name`, `user_id`, or
layer fields — even `null`) returns 422 `field_not_mutable` BEFORE the
DB lookup. Active sessions continue using their cached `AgentContext`;
the new values take effect at the next session-create.
**Mutable surface (Phase 2.3, #188): `system_prompt` and/or `model` only.**
PATCH re-enforces the same validation as define — the `system_prompt`
byte-cap and the `model` allowlist. Any other key returns a 422 BEFORE
the DB lookup (so an immutable-field PATCH against a missing agent still
422s, not 404s), with the error code chosen by *why* the field can't be
set:
| Field(s) | Code | Reason |
|---|---|---|
| `agent_name`, `user_id`, `agent_id` | `field_not_mutable` | Identity — fixed at creation. |
| `persona`, `motivational` | `field_not_mutable` | Shipped traits; an agent *is* its personality/goals. Change → define a new agent. |
| `memory` | `field_not_mutable` | Rejected **wholesale** — see below. |
| `valence` | `layer_deferred` | Not a shipped layer yet (matches define-time); not a frozen trait. |
Every immutable/deferred field is rejected even when its value is `null` —
supplying the key at all is the trigger.
**`memory` is wholesale-immutable.** There is no sub-field carve-out:
`stm_capacity` / `stm_token_budget` are deprecated no-ops since the STM
tier was removed (#197), `allows_world_scope` is create-time-only (memory
scope policy must be fixed before any memory is written), and
`embedder_version` is library-pinned. Note the deliberate asymmetry with
define: `POST /agents/define` accept-and-ignores deprecated `stm_*`
(201 + deprecation warning), but `PATCH {"memory": {...}}` rejects the
whole field with `field_not_mutable`. When a real long-term-memory tuning
dial ships, its PATCH semantics will be specified at that time.
**Active sessions are unaffected.** A PATCH never mutates an in-flight
session's cached `AgentContext`; new `system_prompt` / `model` values take
effect only at the next session-create.
**Audit.** A successful PATCH emits one `agents.patch` event whose
`changes` detail records before/after per mutated field: `model` as literal
`{before, after}` values, and `system_prompt` as `{before_bytes,
after_bytes}` only — the raw prompt text is never written to the audit log
(potential PII).
**Model-assignment warnings (#219).** A `model` swap is **not blocked** for
capability or context-window compatibility, but PATCH (and `define`) attach an
advisory `warnings` array to the response — see the shared subsection below.
Correctness for over-budget prompts remains the runtime `context_overflow`
guard; the warnings are an early, best-effort heads-up.
##### Model-assignment warnings (`define` + PATCH)
Both `POST /agents/define` (201) and `PATCH /agents/<id>` (200) include a
`warnings` array in the response body (always present; `[]` when none). It is
**advisory and non-blocking** — never a rejection — and appears only on these
two mutation responses, not on `GET /agents/<id>`. Each entry is
`{code, severity, message, details}`. The closed code set:
| code | severity | when |
|---|---|---|
| `model_context_window_unknown` | `info` | The assigned model has no recorded context window (`0`/absent in the registry). |
| `model_context_window_smaller` | `warning` | Both prior and new model have known windows and the new one is smaller. `details: {before, after}`. |
| `model_capability_downgrade` | `warning` | The new model **explicitly** advertises fewer capabilities than the prior — drops `tools`, `vision`, or `audio`. `details: {dropped: [...]}`. |
Semantics:
- **`define`** has no prior model, so only `model_context_window_unknown` can
fire there. **PATCH** computes warnings only when the payload changes `model`
(a `system_prompt`-only PATCH returns `warnings: []`); the comparison is
against the resulting model.
- Capability warnings are **conditional by nature**: a Tier 3 agent row does
not record whether it uses tools/vision/audio (tools arrive per-session via
Bifrost), so the message is phrased "if your sessions rely on these…". A
downgrade is reported only when both models carry explicit registry metadata.
- Messages never claim a hard failure. The stored `system_prompt` cap is a
**byte** limit (32 KiB), independent of any model's token budget — it is not
a fit guarantee. A too-large prompt for the chosen model still surfaces at
runtime as `context_overflow`.
#### `POST /sessions` — Tier 3 routing
@@ -2860,8 +2981,8 @@ endpoint isn't reachable.
| `agent_name_invalid` | 422 | `agent_name` violates `[a-z][a-z0-9-]{2,63}`. |
| `system_prompt_too_large` | 422 | `system_prompt` > 32 KiB. |
| `model_not_available` | 422 | `model` not in `providers.yaml`. |
| `layer_deferred` | 422 | One of `persona` / `motivational` / `valence` / `memory` set. |
| `field_not_mutable` | 422 | PATCH carries an immutable key (any value, even `null`). |
| `layer_deferred` | 422 | `valence` set on define OR PATCH (the only still-deferred layer; persona/motivational/memory activated in Phase 2.1/2.2). |
| `field_not_mutable` | 422 | PATCH carries an immutable key — identity (`agent_name`/`user_id`), `persona`, `motivational`, or `memory` (any value, even `null`). `valence` → `layer_deferred` instead. |
| `end_user_id_required` | 422 | Tier 3 session-create without a non-empty `end_user_id`. |
| `tier3_user_id_unsupported` | 403 | Caller's `ctx.user_id` not slug-safe. |
| `auth_scope_denied` | 403 | Missing `agents.define` or wrong owner. |
+309 -13
View File
@@ -137,6 +137,68 @@ Sessions are persistent via SQLite. On server restart, existing sessions are
loadable from the store (lazy-loaded on first access). In-memory cache is
rebuilt on demand, not at startup.
## Memory-partition scope (#245 / ADR-0011)
`end_user_id` is the per-end-user memory partition key (distinct from `user_id`,
the API-key owner). It is REQUIRED at session-create for Lofn (Tier-1) and Tier-3
agents and must survive a store reload, because "remember me next session" is by
definition a reload. Memory partition resolution flows through ONE resolver that
cannot hand an authenticated session the shared `local_dev` partition.
- **INV-245-1 (end-user-id-durable)**: `end_user_id` is persisted as a `sessions`
table column at create and rehydrated onto the `ConversationSession` on every
cache-miss load (`get_session`). A session loaded from the store carries the
same `end_user_id` it was created with. Pre-migration rows read as `None`.
- **INV-245-2 (end-user-id-threaded-all-tiers)**: the `POST /sessions` handler
forwards `body.end_user_id` to `create_session` for EVERY agent, not only
Tier-3. (The pre-fix `if tier3_agent_context is not None else None` conditional
dropped it for Lofn despite the create gate requiring it.)
- **INV-245-3 (no-authenticated-local-dev)**: the two MEMORY partition sites —
auto-recall (read) and the ContextPromotion producer (write) — resolve via
`memory_scope_for_session`. An authenticated, memory-bearing session (one not
carrying the explicit `local_dev` sentinel) NEVER resolves to `local_dev`; a
missing `end_user_id` raises `MemoryScopeError`, and because both sites are
best-effort (recall is fire-and-forget; the producer is `_run_promotion_safe`),
the caller skips memory — it never silently writes to the shared partition.
- **INV-245-5 (persona-plane-corrected-by-persistence)**: the three PERSONA-plane
sites (`inject_context`, `get_state`, `update_after_turn` — ADR-0008 mood/PAD/
valence) keep their `session.end_user_id or "local_dev"` form but are on the
main turn path where a raise would break the turn. They are corrected by
INV-245-1/2: once `end_user_id` is persisted + threaded, the fallback yields a
real partition for authenticated sessions and `local_dev` only for the explicit
terminal path. Unifying the persona plane under the resolver (with main-path
error semantics) is follow-up, tracked with the #246-adjacent hardening.
- **INV-245-4 (terminal-explicit-local-dev)**: the internal terminal transport
creates its sessions with `end_user_id="local_dev"` explicitly. `local_dev` is
reached only by this positive assertion, never by omission. (External API
callers passing `local_dev` are still rejected per #216.)
```contract
FN memory_scope_for_session(session) -> MemoryScope
BRIEF: The single authority resolving a session to its memory partition scope.
Returns a typed MemoryScope(scope_type, scope_id); scope_type ∈
{local_dev, end_user, room, tenant} (only local_dev + end_user active in
v1; room/tenant reserved for ADR-0010). Cannot yield local_dev for an
authenticated session.
PRE: [PRE-001 soft] callers have already gated ephemeral / consumer_defined
sessions out (those skip memory before resolution)
POST: [POST-001 return_value] end_user_id == "local_dev" -> MemoryScope("local_dev", "local_dev")
POST: [POST-002 return_value] end_user_id truthy and != "local_dev" -> MemoryScope("end_user", end_user_id)
POST: [POST-003 exception] end_user_id is None/empty -> raise MemoryScopeError (NEVER local_dev)
ERRORS:
MemoryScopeError -> caller skips memory (best-effort) + emits an audit/log line; turn proceeds
STEPS:
1. [setup] read euid = session.end_user_id
2. [branch] euid == "local_dev" -> RETURN MemoryScope("local_dev", "local_dev") (terminal sentinel)
3. [branch] euid truthy -> RETURN MemoryScope("end_user", euid)
4. [error_handler] else (None/empty) -> RAISE MemoryScopeError (never silently local_dev)
TESTS:
end_user_partition [happy,tracer]: session end_user_id="alice" -> MemoryScope("end_user","alice")
terminal_local_dev [boundary]: session end_user_id="local_dev" -> MemoryScope("local_dev","local_dev")
authenticated_none_raises [boundary]: foundational session end_user_id=None -> raises MemoryScopeError, NOT local_dev
isolation_roundtrip [happy]: create_session(end_user_id="alice") write + clear cache + reload + recall isolates from a "bob" session; negative-assert no local_dev write
```
```contract
FN ConversationService.startup() -> None
BRIEF: Discover agents, build per-agent contexts, initialise shared infrastructure
@@ -1658,13 +1720,13 @@ Ephemeral templates are a new agent kind that bypass persona, memory, tools, and
**Invariants added by issue #161:**
- **INV-161-1 (ephemeral-template-bypass)**: For sessions where `session.ephemeral_config is not None`, `PersonaRegistry.inject_context` is NOT called pre-turn; `PersonaRegistry.update_after_turn` is NOT called post-turn; valence side-channel is NOT called; tool list passed to provider is `[]`.
- **INV-161-2 (frozen-session-config)**: Once a session is created with an `ephemeral_config` snapshot, subsequent mutations to `agents/saga/config.yaml`, `config/providers.yaml → saga_allowed_models`, or `config/defaults.yaml → saga.default_model` do NOT affect that session's per-turn `system_prompt` or `model`.
- **INV-161-2 (frozen-session-config)**: Once a session is created with an `ephemeral_config` snapshot, subsequent mutations to `agents/echo/config.yaml`, `config/providers.yaml → echo_allowed_models`, or `config/defaults.yaml → echo.default_model` do NOT affect that session's per-turn `system_prompt` or `model`.
- **INV-161-3 (no-tools-for-ephemeral)**: Tool list passed to the provider for an ephemeral session is `[]` regardless of any `tools:` block in the template's config.yaml.
- **INV-161-4 (foundational-flow-unchanged)**: For sessions where `session.ephemeral_config is None`, the per-turn path is bit-identical to pre-#161 — same system_prompt loading, same persona injection, same tool list, same audit-log shape.
- **INV-161-5 (config-required-for-ephemeral-create)**: `POST /sessions` against an ephemeral template MUST reject the request with 422 if `config` is missing or fails any validation step.
- **INV-161-6 (model-allowlist-enforcement)**: `config.model`, when supplied, MUST be in `saga_allowed_models` at session-create time. When omitted, server resolves to `saga.default_model` (startup-validated to be in the allowlist).
- **INV-161-6 (model-allowlist-enforcement)**: `config.model`, when supplied, MUST be in `echo_allowed_models` at session-create time. When omitted, server resolves to `echo.default_model` (startup-validated to be in the allowlist).
- **INV-161-7 (full-prompt-in-audit)**: Session-create audit entries for ephemeral sessions include `tier: 2` and `ephemeral_config` (full JSON).
- **INV-161-8 (cross-user-isolation)**: A Saga session created by user A is invisible to user B — `GET /sessions/{id}` returns 404.
- **INV-161-8 (cross-user-isolation)**: An Echo session created by user A is invisible to user B — `GET /sessions/{id}` returns 404.
- **INV-161-9 (foundational-rejects-config)**: `POST /sessions { agent_id: "<foundational>", config: {...} }` returns 422 with `error_code: "foundational_does_not_accept_config"`.
- **INV-161-10 (capabilities-public-shape)**: `GET /capabilities` is callable by any authenticated key. The response has `ephemeral_templates` at top-level.
- **INV-161-11 (template-kind-immutable-at-runtime)**: The `kind` field on a loaded `AgentContext` is set once at startup and never mutated.
@@ -1673,16 +1735,16 @@ Ephemeral templates are a new agent kind that bypass persona, memory, tools, and
| code | HTTP | trigger |
|---|---|---|
| `ephemeral_requires_config` | 422 | saga session without `config:` |
| `ephemeral_requires_config` | 422 | echo session without `config:` |
| `foundational_does_not_accept_config` | 422 | foundational agent with `config:` |
| `system_prompt_required` | 422 | `config.system_prompt` missing or null |
| `system_prompt_empty` | 422 | `config.system_prompt` whitespace-only |
| `system_prompt_too_large` | 422 | > 32768 bytes UTF-8 |
| `model_not_allowed` | 422 | model not in `saga_allowed_models` |
| `model_not_allowed` | 422 | model not in `echo_allowed_models` |
**New `AgentContext` fields:** `kind: str = "foundational"`, `saga_allowed_models: list | None`, `saga_default_model: str | None` — populated for ephemeral templates, `None` for foundational agents.
**New `AgentContext` fields:** `kind: str = "foundational"`, `echo_allowed_models: list | None`, `echo_default_model: str | None` — populated for ephemeral templates, `None` for foundational agents.
**Startup failfast:** server refuses to start if `agents/saga/config.yaml` is missing/malformed OR `saga.default_model` is not in `saga_allowed_models`. Raises `ConfigurationError` before binding any port.
**Startup failfast:** server refuses to start if `agents/echo/config.yaml` is missing/malformed OR `echo.default_model` is not in `echo_allowed_models`. Raises `ConfigurationError` before binding any port.
**Function-level contracts for issue #161** are documented in `docs/contracts/issues/161.contract.md`.
@@ -1704,7 +1766,7 @@ Bifrost allows consumers to expose tools to Worldtree agents. `POST /sessions` a
- **INV-160-1 (handshake-at-create)**: When `POST /sessions` carries `bifrost: {endpoint_url, ...}`, the handshake completes BEFORE the 201 response. No "create session, handshake later" path in v0.1. Verifiable via test: handshake-failing endpoint → 502; session not in store.
- **INV-160-2 (one-connection-per-session)**: Each Bifrost-bound session owns exactly one MCP connection. Two sessions binding to the same `endpoint_url` open two independent connections. No pooling, no sharing.
- **INV-160-3 (saga-incompatible)**: A session cannot be both ephemeral (Saga, `kind: "ephemeral"`) AND Bifrost-bound. Session-create rejects with 422 `ephemeral_does_not_accept_bifrost`. Verifiable: `POST /sessions { agent_id: "saga", config: {...}, bifrost: {...} }` → 422.
- **INV-160-3 (echo-incompatible)**: A session cannot be both ephemeral (Echo, `kind: "ephemeral"`) AND Bifrost-bound. Session-create rejects with 422 `ephemeral_does_not_accept_bifrost`. Verifiable: `POST /sessions { agent_id: "echo", config: {...}, bifrost: {...} }` → 422.
- **INV-160-4 (jwt-bound-to-session-expiry)**: JWT TTL is bound to session expiry — far-future `expires_at` for sessions without a fixed TTL. Re-mint happens only when a re-handshake fires (connection-loss recovery). No standalone JWT-staleness check.
- **INV-160-5 (reentrancy-25-per-turn)**: At most 25 successful Bifrost tool invocations per agent turn. The 26th returns `bifrost.reentrancy_cap_exceeded` without contacting the consumer. Counter resets per turn via `BifrostClient.reset_turn_counter()`. Enforced inside `BifrostClient.invoke_tool`.
- **INV-160-6 (tool-list-cached-per-session)**: Bifrost tools are fetched once at handshake and cached on `ConversationSession.bifrost_tools`. Per-turn dispatch reads from the cache; never re-fetches mid-session except on connection-loss recovery.
@@ -1751,7 +1813,7 @@ class BifrostEndpointOverride(BaseModel):
1. HTTPS URL check — Pydantic field validator; 422 on miss.
2. `bifrost:invoke` scope check — same as session-bound path; 403 on miss.
3. Ephemeral session rejection — 422 `ephemeral_does_not_accept_bifrost` when session is Saga (extends INV-160-3).
3. Ephemeral session rejection — 422 `ephemeral_does_not_accept_bifrost` when session is Echo (extends INV-160-3).
4. Heimdall consumer lookup — 502 `bifrost_consumer_not_found` on miss or unregistered.
5. Instantiate a new `BifrostClient` with the override consumer's algorithm + key; set `_jwt_ttl_seconds = 60`.
6. `await override_client.connect()` — 502 `bifrost_handshake_failed` on failure.
@@ -1789,7 +1851,7 @@ In the `finally` block, `await override_client.disconnect()` is called unconditi
The conversation API grows a three-tier agent model. Tier 1 is the
foundational set (Mimir, Bragi, Leif, ...) wired at startup. Tier 2 is
the ephemeral template surface (Saga). Tier 3 is the consumer-defined
the ephemeral template surface (Echo). Tier 3 is the consumer-defined
class addressed by `<user_id>:<agent_name>` and stored in Heimdall's
SQLite `consumer_agents` table.
@@ -1813,9 +1875,13 @@ SQLite `consumer_agents` table.
- **INV-181-5 (agent-name-immutable, Phase 2.0 scope)**: PATCH rejects
any payload that includes `agent_name`, returning 422
`field_not_mutable` BEFORE the DB lookup.
- **INV-181-6 (layer-immutable-in-patch, Phase 2.0 scope)**: PATCH
rejects payloads carrying any of `persona`, `motivational`,
`valence`, `memory` even when set to `null`.
- **INV-181-6 (layer-immutable-in-patch, Phase 2.0 scope; AMENDED #188)**:
PATCH rejects payloads carrying any of `persona`, `motivational`,
`memory` even when set to `null`, returning `field_not_mutable`.
**Amended by #188 (Phase 2.3):** `valence` was moved out of this
`field_not_mutable` set — it now returns `layer_deferred` (see
INV-188-1), because valence is a not-yet-shipped layer, not a frozen
trait. `memory` is rejected wholesale (see INV-188-2).
- **INV-181-7 (owner-delete-hard, Phase 2.0 scope)**: `DELETE
/agents/<id>` is a hard-delete; bypasses the 24h grace.
- **INV-181-8 (cascade-key-scoped, Phase 2.0 scope)**: Key revocation
@@ -1881,6 +1947,79 @@ SQLite `consumer_agents` table.
through `_publish`, so SSE resume / replay handles them with no
special case.
## Amendment — Suspended-tier license-state gate (issue #174, INV-174-1..9)
Adds a `suspended` tier with empty scope set to drive license-expiry
transitions without destroying user state. Endpoint
`POST /admin/users/{user_id}/tier` mutates the tier; the
`_http_exception_handler` rewrites `AUTH_SCOPE_DENIED` →
`USER_SUSPENDED` for any 403 raised against a non-anonymous caller with
an empty scope-set (the suspended-tier defining property). Ships in
v0.29.1.
- **INV-174-1 (closed tier vocabulary)**: `POST /admin/users/{user_id}/tier`
validates `body.tier` against the hard-coded set `{anonymous, user, free,
pro, admin, suspended}`. Out-of-set values return 422 `invalid_tier`.
Vocabulary is NOT derived from `policies.yaml` at runtime — a typo in
YAML must not silently expand the accepted set.
- **INV-174-2 (admin-only mutation)**: endpoint requires
`admin.users.write.tier_change` scope. Listed explicitly in admin
tier's scope set in `policies.yaml` for grep-discoverability (admin
also carries `*` umbrella).
- **INV-174-3 (tier mutation primitive)**:
`UserStore.update_user_tier(user_id, new_tier) -> User` is the storage
primitive. Raises `LookupError` for unknown user_id (endpoint converts
to 404 `user_not_found`).
- **INV-174-4 (suspended scope-set is exactly empty)**:
`policies.yaml.tiers["suspended"].scopes == []`. The empty set is what
makes the auth-denial work for free; the
`_http_exception_handler` rewrite uses
`ctx.user_id != "anonymous" and not ctx.scopes` as the
suspended-detection heuristic since `SecurityContext` deliberately
excludes `tier` (per `core/integration/types.py:64`).
- **INV-174-5 (uniform suspended error code via exception handler)**:
The `_http_exception_handler` (registered for `StarletteHTTPException`)
intercepts every 403 with `error_code: auth_scope_denied`; if the
request's stashed `SecurityContext` has an empty scope-set (and
non-anonymous user_id), it rewrites the detail to
`{error_code: "user_suspended", message: "Account is suspended."}`.
Single seam — covers every existing and future scope-deny site
without per-endpoint refactor. The ctx is stashed by
`get_security_context` on `request.state.security_context`.
- **INV-174-6 (/me carve-out)**: `/me` does NOT call `authorize()` and
therefore never raises `AUTH_SCOPE_DENIED`. Suspended users with
empty scopes reach the /me handler normally and see
`{user_id, tier: "suspended", scopes: [], ...}`. Adding a scope check
to /me without preserving the suspended-tier visibility would be a
contract violation — the carve-out is structural, not coded.
- **INV-174-7 (audit emission)**: every tier-change attempt emits
`conversation_api:admin:user:tier_changed` via `_audit_admin_action`
with `actor_user_id`, `target_user_id`, `outcome ∈
{success, denied}`, and `extra = {from_tier, to_tier, reason}` for
successes; `extra = {reason: <reason_code>}` for denials
(`invalid_tier`, `user_not_found`).
- **INV-174-8 (reversibility via audit replay)**: the user record does
NOT carry a `previous_tier` column. Restoration of a suspended user
requires reading the audit log to find the most recent
`tier_changed` event with `to_tier="suspended"` and replaying its
`from_tier` as the new target. Operational responsibility of SEA's
billing integration; Worldtree provides only the read (audit log) and
write (endpoint) surfaces.
- **INV-174-9 (no cross-tier session invalidation)**: a tier change for
a user with active SSE turns in flight does NOT cancel those turns.
The next request after the tier change picks up the new scope-set;
in-flight streams complete under the old tier. If SEA needs
immediate-cutoff semantics, that requires `disable_user`-style
hard-revoke, not a tier change.
## Amendment — AwaitingLLMFirstToken heartbeat (issue #201, INV-201-1..7)
Adds a periodic SSE heartbeat event during the gap between
@@ -2027,3 +2166,160 @@ Lofn introduces zero net-new persistence surface. No table, no
column, no Mimir KB collection. No new audit-event types. Existing
session-create / session-revoke audit covers Lofn the same way it
covers Mimir / Forseti.
## Amendment — Tier 3 motivational layer (issue #187, Phase 2.2)
Activates the `motivational` layer field on `POST /agents/define`, narrowing the
Phase 2.0 `layer_deferred` rejection (INV-181-3) to `valence` only. Full FN-level
spec at `docs/contracts/issues/187.contract.md`.
- **INV-187-1 (motivational-activated)**: `POST /agents/define` accepts a non-null
`motivational` object `{goals, fears}`; `_tier3_validate_layer_fields` rejects
only `valence` now. (Persona + memory were activated in Phase 2.1 / #189.)
- **INV-187-2 (define-validation)**: `validate_motivational_define_payload` enforces
the documented 422 codes — `motivational_id_collision` (case-sensitive, across
goals AND fears), `motivational_goal_invalid_type`,
`motivational_salience_out_of_range`, `motivational_description_too_short`
(< 20 chars after strip), `motivational_missing_required_field`. Unknown top-level
OR nested (per goal/fear) keys → `validation_failed` (sub-models extra-forbid).
Stricter than the Tier 1 `validate_motivation` (which only warns on short text).
- **INV-187-3 (per-agent-scope)**: motivational is per-agent, NOT
per-(agent, end_user) — stored once on the row, identical across all end-users.
- **INV-187-4 (immutable-in-patch)**: `PATCH` with `motivational` → 422
`field_not_mutable` (already covered by INV-181-6's `_IMMUTABLE_FIELDS` gate).
- **INV-187-5 (tier3-render-bridge)**: Tier 3 agents are NOT registered with the
`persona_registry`; the stored config rides on the per-session `AgentContext`
(`motivational_config`) and is rendered into the prompt per-turn in `stream_turn`
via `_append_motivational_context_section`, before the memory-context section.
- **INV-187-6 (fear-signal-shape)**: fears carry `trigger_signals`; goals carry
`positive_signals` + `negative_signals` (matches the `GoalConfig`/`FearConfig`
substrate).
- **INV-187-7 (tier-uniformity)**: the render reuses `core.persona.goals.load_goals`
+ `render_motivational_context`, so a Tier 3 motivational config produces a
byte-identical block to an equivalent Tier 1 `motivation.yaml`.
- **INV-187-8 (storage)**: persisted in `consumer_agents.tier3_layers_json` under
the `"motivational"` key; round-trips via `ConsumerAgent.motivational`; null/omitted
→ `None` (no fabricated defaults; no migration).
### Audit
`agents.define` audit `extra` gains `presence_motivational: bool` alongside
`presence_persona` / `presence_memory`.
## Amendment — Tier 3 PATCH mutability policy (issue #188, Phase 2.3)
Settles which Tier 3 agent fields are editable post-define. #197 deleted the
STM tier between this issue's filing (2026-05-19) and its implementation, so the
"mutable memory dials" the original issue envisioned no longer exist; the policy
collapses to: `system_prompt` + `model` mutable, everything else fixed, with
`valence` distinguished from the immutable traits by error code. No new
endpoint, no new storage, no new invariant philosophy — a clarification +
error-code alignment + audit enrichment over the Phase 2.0 PATCH baseline.
- **INV-188-1 (valence-deferred-in-patch)**: `PATCH /agents/<id>` carrying a
`valence` key (any value, including `null`) → 422 `layer_deferred` with
`field: "valence"`, matching define-time (INV-181-3). Rationale: valence is
a layer that does not exist yet, not a real-but-frozen trait; `layer_deferred`
is the truthful reason and gives consumers ONE code for "valence unavailable"
across both define and PATCH. The check precedes the DB lookup (INV-181-5/6
ordering), so a `valence` PATCH against a missing agent still 422s, not 404s.
- **INV-188-2 (memory-wholesale-immutable-in-patch)**: `PATCH` carrying a
`memory` key → 422 `field_not_mutable` with `field: "memory"`, rejected at the
WHOLE-field level. No sub-field carve-out exists: `stm_capacity` /
`stm_token_budget` are deprecated no-ops post-#197, `allows_world_scope` is
create-time-only (toggling it after memory is written breaks scope-visibility
invariants — memory scope policy must be fixed before any memory is written),
and `embedder_version` is library-pinned. A real LTM tuning dial would warrant
a deliberate per-sub-field PATCH contract at that time; pre-splitting for dead
fields is not done. NOTE the deliberate define/PATCH asymmetry: `define`
accept-and-ignores deprecated `stm_*` (201 + DeprecationWarning per
INV-197-19), but `PATCH memory:{...}` rejects wholesale (422). Acceptable
transitional artifact; disappears when the shims are removed.
- **INV-188-3 (patch-audit-before-after)**: a successful `agents.patch` audit
event's `extra.changes` records before/after for each mutated field —
`model: {before, after}` (literal values; allowlist enum, not PII) and
`system_prompt: {before_bytes, after_bytes}` (byte-length only; raw prompt
content is excluded as potential PII, consistent with `emit_consumer_agent_event`'s
exclusion rule). `changes` contains only keys for fields actually present in
the PATCH payload. `patched_fields` (the Phase 2.0 name list) is retained.
- **INV-188-4 (mutable-surface-unchanged)**: the mutable surface stays exactly
`system_prompt` + `model` (per INV-181 Phase 2.0). PATCH re-enforces the
define-time `system_prompt` byte-cap and `model` allowlist. #188 does NOT add
model-swap capability/context-window validation — that gap (a swap to a
smaller-context or non-tool model with no re-check of the existing prompt) is
tracked as a separate follow-up (#219), not folded here.
## Amendment — model-assignment advisory warnings (issue #219)
`POST /agents/define` and `PATCH /agents/<id>` attach a best-effort, **non-
blocking** `warnings` array to their 2xx response when the assigned `model`
carries metadata risk (smaller context window, unknown window, or an explicit
capability downgrade). This is advisory-only by deliberate design: hard
rejection was rejected (Heid panel + operator, 2026-05-29) because model
metadata coverage is partial (`context_window` is 0/unknown for several
allowlisted models; `supports_tools` defaults true), the stored `system_prompt`
cap is bytes not tokens, Tier 3 agent rows store no tool/modality usage (tools
arrive per-session via Bifrost, so any capability concern is inherently
conditional), and runtime already classifies the real failure as
`CONTEXT_OVERFLOW`. The warning is a receipt-note for the owner who just made a
deliberate change, not a correctness gate.
- **INV-219-1 (advisory-not-blocking)**: neither define nor PATCH ever rejects
on context-window or capability grounds. The allowlist check
(`model_not_available`) and `system_prompt` byte-cap are the only model-
related *rejections*; everything in #219 is a warning on an otherwise-2xx
response. Correctness for over-budget prompts remains the runtime
`CONTEXT_OVERFLOW` guard.
- **INV-219-2 (bounded-warning-codes)**: the closed code set is exactly —
`model_context_window_unknown` (severity `info`): the assigned model's
registry `context_window` is `0`/absent; `model_context_window_smaller`
(severity `warning`): prior and new model both have known windows and
new < prior (`details: {before, after}`); `model_capability_downgrade`
(severity `warning`): the new model EXPLICITLY drops a capability the prior
model advertised — `supports_tools`, `vision`, or `audio` (`details:
{dropped: [...]}`). No token-aware "prompt won't fit" code — deferred until
tokenizer-aware estimation exists; messages never claim a hard fit/failure.
- **INV-219-3 (when-evaluated, resulting-pair)**: warnings are computed
whenever a model is *assigned*. At define, always (prior = None → only
`model_context_window_unknown` can apply, since the comparative codes need a
prior). At PATCH, only when the payload carries a `model` key whose value
differs from the stored model (prior = stored model); a PATCH without `model`
(e.g. `system_prompt`-only) emits no model warnings. The comparison is always
against the *resulting* model.
- **INV-219-4 (capability-downgrade)**: a `model_capability_downgrade` fires
only when BOTH prior and new models resolve to registry `ModelInfo` AND the
new model's *effective* capability flags lack one the prior advertised
(`supports_tools`, `vision`, or `audio`). The "both resolve" guard is the
false-positive defense — an unresolvable model on either side yields no
downgrade claim. Beyond that, comparison uses the registry's **effective**
flags, which is asymmetric by capability because the data model collapses
absent-to-default and does not preserve a "was this declared?" bit:
- `supports_tools` defaults **true** (`ModelInfo` / `_build_model_info`), so
a tools-drop requires the new catalog entry to set `supports_tools: false`
*explicitly* — omission never triggers it.
- `vision` / `audio` default **false** (`ModelCapabilities`), so a drop is
detected whenever the prior advertised the capability and the new model does
not carry it — whether the new entry says `false` explicitly OR omits it.
This is the deliberate conservative reading: an undeclared modality is
treated as unsupported. (A vision-capable model with sloppy metadata that
omits its `vision` flag would thus be reported as a downgrade; the remedy is
to declare the flag in the catalog, not to suppress the advisory.)
Message phrasing is conditional ("if your sessions rely on these, e.g. Bifrost
tools, they may be rejected") — the agent row does not record whether tools or
modalities are actually used, so every capability warning is advisory by
nature.
- **INV-219-5 (inline-response-shape)**: the `warnings` array is added inline to
the define (201) and PATCH (200) response bodies — the existing flat
`ConsumerAgentResponse` dict gains a `warnings` key (always present, `[]` when
none). It is NOT added to the shared `ConsumerAgentResponse` pydantic model
nor to `GET /agents/<id>` — only the two mutation handlers merge it into their
returned dict, keeping persisted fields and the read path unchanged. Each
entry is `{code, severity, message, details}`.
- **INV-219-6 (single-helper)**: a single pure helper
`compute_model_swap_warnings(*, prior_model: str | None, new_model: str,
registry)` is the only source of warning logic; both define and PATCH call
it. It tolerates unresolvable specs / `None` `ModelInfo` / `context_window`
`0` by treating them as "unknown" (emitting the unknown-window info code where
applicable, never raising). Metadata improvements over time sharpen the
warnings with no API or signature change.
+154 -15
View File
@@ -1,6 +1,6 @@
# Persistent memory — ratatoskr
_Last updated: 2026-06-15_
_Last updated: 2026-06-16_
This file captures durable intent and supporting evidence (goals, decisions,
foot-gun warnings, in-flight state) across context resets. Read it at session
@@ -41,13 +41,69 @@ model output is untrusted); upstream API key stays server-side (INV-003).
## Current state / in-flight
_As of 2026-06-15:_
_As of 2026-06-18:_
**LATEST (2026-06-18) — TIER-3 MEMORY PROVEN END-TO-END LIVE; #296 FIX SHIPPED (Stage 1 + Stage 2).**
The session's opening goal — "how far from Tier-3 agents we host with their own memories?" — is
ANSWERED: a hosted agent recalled a user fact across a COLD (history-free) session, live. Full chain:
seed → promotion → our v0.17.6 `:8391` store (upsert) → cold session → WT `scope_any` recall (1 hit @
cosine 0.6994) → model answered "Dark chocolate with sea salt." Proven with `ratatoskr:terse-probe`
(plain persona, end_user `terse-proof`) vs personal WT v0.35.19.
The whole #296 arc closed this session:
- **Stage 1** (v0.35.19, recallability admission gate + real `_EXTRACTOR_SYSTEM`) VALIDATED LIVE for
normal-volume turns — clean first-person distillate (`distiller_model_id="identity"`), recalls @0.70.
- **Bisect verdict:** residual = verbose-persona VOLUME crowd-out (symptom 1). terse-probe promotes the
fact cleanly; sindra's theatrical turn drowns it → v0.35.16 emits 2 meta-descriptions (a3466fab confirmed),
v0.35.19 gate catches + hard-rejects (emit-then-reject, fail-closed).
- **Stage 2** (v0.36.0, MERGED on demo) = user-only one-call-per-turn extraction, the STRUCTURAL fix for
the verbose drop; hard-linguistic layer at #305. Live-validated eval fixture pair handed to worldtree-dev
(terse-proof POSITIVE chunk + choc-v2/sindra NEGATIVE).
**FULL COVERAGE PROVEN (the OPEN item, now DONE):** re-smoked the VERBOSE persona — `ratatoskr:sindra-probe`
(sindra's actual theatrical prompt, end_user `verbose-v2`, FRESH agent to dodge the WT promotion-dedup) against
`:8081` **v0.36.0**: the high-volume theatrical turn promoted the USER FACT cleanly (not a meta-description) and a
COLD history-free session recalled it @ 0.694 — "Dark chocolate. With sea salt." So `:8081` is on v0.36.0 and
verbose characters remember too. All session commits + tags PUSHED to origin (v0.17.6, spec repin v0.35.16,
snapshots; tags v0.17.4/.5/.6).
**NOW (operator session):** `:8391` store WIPED clean; `ratatoskr-web` UP on `0.0.0.0:8765`
(LAN `http://10.100.10.50:8765`, consumer key — owner-matches sindra, end_user `vuong`, `ratatoskr:sindra` in the
picker). Sindra switched to the **`thoughtful-character` role → `mistral-small-4-reasoning`** (via DELETE+redefine;
she chats in full persona on it). **"No persona" 404 diagnosed = Worldtree limitation, NOT ours:** `persona_state`
hard-404s every Tier-3 (colon-id) agent by design (`api.py:1220` "Phase 2.0 has no Tier 3 persona", still on
v0.36.0) — a persona block in the define does NOT help (404 fires before the row read). Pinged worldtree-dev re:
the Tier-3 persona_state roadmap (thread `01KVCR6P…`). De-uglied our web persona pane (`index.html` `loadPersona`)
to render a clear Tier-3 message instead of a bare "HTTP 404". **CAVEAT still stands: the web client does NOT bind
to `:8391`/`:8390` (#17 unbuilt) → web chat = persona + debug only; no memory persistence, no affect telemetry.**
**NEXT (fresh context): #17 — bifrost-binding the chat client.** The single unblock for BOTH (a) memory persistence
in the web/TUI/CLI chat (Sindra remembering in a real session, not just scripts) AND (b) feeding the persona pane
affect telemetry once WT ships Tier-3 persona_state. Contract `docs/contracts/issues/17.contract.md` is WRITTEN +
Heid-reviewed + drift-clean; **TDD is the next step** — slice 1 = the `create_session` bifrost-binding primitive
(consumer-key per-request bearer + missing-key precondition + 502→`BifrostHandshakeFailed`, respx-mocked), then
`endpoint_for_plane` → dispatch-layer op-feed → CLI/TUI/web triggers → live smoke. althing monitor armed.
**(2026-06-16 PM) — BIFROST REPINNED 0.7.0→0.8.0 (wire v0.5→v0.6).** The
memory `search` scope filter was split into `scope_all` (AND/intersection) +
`scope_any` (OR/union over a LIST of conjunctive scopes) — bifrost #11, the canonical
fix for the #295/#297 silent-zero AND foot-gun. Our store + contract (v1.2) + tests
reimplemented to parity with the v0.6 reference `_matches_scope`/`_validate_scope`
(no-compat: `scope_filter` REMOVED). 433 tests green incl. the new `scope_any` union
test + the parity-vs-reference test through the real 0.8.0 `dispatch_memory_call`.
Memory provider BOUNCED onto 0.8.0 (`:8391`, fresh empty `memory.db` — the prior
5-chunk #296 corpus was WIPED, operator confirmed "nothing of value", SUPERSEDES the
"KEEP PINNED" note below). Shipped as v0.17.6 (`96d61a4`, tag v0.17.6); the #17
contract's stale `scope_filter`/`_scope_matches`-AND references were synced in the same
commit. Affect plane untouched (split is memory-only); affect provider still on its
0.7.0-loaded process (bounce optional — affect wire unchanged at 0.8.0). Cold-recall
status SUPERSEDED by the 2026-06-17 block above (now CLOSED end-to-end).
**Ratatoskr now has a SECOND identity: the v1 Bifrost Tier-3 consumer** — the
durable persistence provider Worldtree writes Tier-3 agent affect/persona +
memory into — alongside the original debug-observability TUI/web. The
Bifrost-consumer work lives in `src/ratatoskr/provider/` and depends on
`bifrost>=0.6.1` (a `provider` optional-extra from the gitea PyPI index),
`bifrost>=0.7.0` (a `provider` optional-extra from the gitea PyPI index),
SEPARATE from the Worldtree conversation-API spec pin.
**AFFECT plane: SHIPPED + LIVE-PROVEN** (v0.17.2). Running now as a dev
@@ -56,13 +112,53 @@ background shell (`ratatoskr-provider`, `0.0.0.0:8390`, env-sourced from
Worldtree **v0.35.2** (`10.250.50.152`): handshake 200 + `affect.emit` 200 →
durable row persisted, opacity held.
**MEMORY plane: contract done, TDD next.**
`docs/contracts/bifrost_memory_provider.contract.md` v1.1 (Heid-panel-reviewed,
committed `1f94e5f`). NEXT (fresh session): add `sqlite-vec` to the `provider`
extra → TDD (tracer `basic_upsert` → search/scope-isolation/optimistic-lock/
conflict → #195 parity vs `InMemoryMemoryStore`) → `/heid-code-review` → ship a
`ratatoskr-memory-provider` dev shell. Worldtree v0.35.3 already negotiates the
basic memory plane — the memory server is the only missing piece.
**MEMORY plane: PROVIDER LIVE-PROVEN + recall-miss ROOT-CAUSED (upstream).**
Store + dev shell shipped (v0.17.3, `cd12951`; running on `0.0.0.0:8391`).
`memory.db` holds 5 durable chunks (all scope `{end_user:smoke-user}`): choc-fact
`498ed752`, name `8241e569`, promoted-question `c863bb6b`, + 2 LATE async promotions
from the cold-recall probe (probe-question `acc3d49`, model NON-ANSWER `4773704`
salience promoted a "I don't have memory" refusal). All are #296 corpus, KEEP PINNED.
**The #295 cold-recall miss is now ROOT-CAUSED and it's UPSTREAM, not ours**
(2026-06-16 debug-assist with worldtree-dev): a self-driven bound cold-recall
probe captured the inbound pair via our new observe log — Worldtree's recall sends
`scope_filter={end_user:smoke-user, agent_self:ratatoskr:smoke}` (TWO axes) but our
chunks carry `{end_user}` ONLY; our AND `_scope_matches` (byte-faithful to bifrost
reference `reference_server/memory.py:398`) drops everything on the unmatched
`agent_self` axis → 0 hits → the model says "no memory". So **our store + search
are SOUND**; the fix is Worldtree-side. F2 → research issue **#296** (keyword-regex
salience suspected fundamentally flawed; corpus = `c863bb6b` + the 2 late-promotions).
F1 → research issue **#297** (Worldtree-local fix = per-visible-scope single-axis
search unioned client-side). **The agent_self lattice question is RESOLVED:**
agent_self is now canonical in bifrost 0.7.0 / wire v0.5 (our foot-gun flag drove it;
worldtree-dev shipped it both sides — Worldtree v0.35.11) → #297 union build UNBLOCKED.
Our memory provider now runs **bifrost 0.7.0 + validates the 4-axis lattice**
`{end_user,group,tenant,agent_self}` (v0.17.5; restarted on it; out-of-lattice axis →
InvalidFilter, reference-parity restored).
**OBSERVE BRICK SHIPPED** (`memory_store.py`, committed v0.17.4 `2fef6e3`):
structured `[memory-provider]` request/response logging on the memory-call path —
the first concrete brick of #17's observe half, and the lens that caught #295's
root cause. Live-verified. (NOTE: this is the debug SHIM at the STORE method; #17's
real observe feed instruments the DISPATCH layer — see the contract INV-005.)
**SELF-DRIVE PROVEN BY HAND** (2026-06-16): ratatoskr's own client drove a
Bifrost-bound cold-recall end-to-end (bind → handshake 200 → turn → captured the
recall pair). Load-bearing finding: a bound session-create must use the CONSUMER
Heimdall key (`RATATOSKR_HEIMDALL_KEY`) as the bearer, NOT the canary
`WORLDTREE_API_KEY` — Worldtree signs the Bifrost handshake JWT with the
session-create bearer (canary → 401; consumer → 200). Runbook:
`docs/bifrost-self-test.md`. This is #17's substrate, proven before the contract.
**ISSUE #17 (self-drive + observe) — CONTRACT WRITTEN + HEID-REVIEWED + FIXED,
TDD NEXT.** `docs/contracts/issues/17.contract.md` (validates OK, drift-clean).
v1 scope operator-locked: single-plane bind (composite endpoint PARKED) +
dispatch-layer op-feed with session-level correlation (turn-pane UI PARKED).
`/heid-contract-review` panel caught + fixed two real internal inconsistencies
(OpEvent `turn_id` reservation made literal; the `session_id`-for-all-verbs
correction). NEXT: **TDD slice 1 = the `create_session` bind primitive** (BifrostBinding
dataclass + consumer-key per-request bearer override + missing-key precondition +
502→BifrostHandshakeFailed; respx-mocked), then endpoint_for_plane → dispatch-layer
op-feed → CLI/TUI/web → live smoke.
**Sindra:** a REGISTERED Tier-3 agent (`ratatoskr:sindra`, was model
`artemis-31b-v1i`) — registration is REQUIRED to use a Tier-3 character (a
@@ -78,12 +174,14 @@ registration.
`~/.config/ratatoskr/provider.env` (mode 600, nh3-dev) — `consumer="ratatoskr"`,
HS256 = the API-key STRING utf-8-encoded; rotate via infra-ops.
**Unpushed:** `main` is 3 commits ahead of origin (origin at `d90a58d`/v0.17.1;
local at `1f94e5f`) — `bcdcd71` (v0.17.2), `eebab46`, `1f94e5f` + the local
`v0.17.2` tag are unpushed. Push is the operator's call.
**Committed (2026-06-16, NOT yet pushed):** observe brick (logger, v0.17.4 `2fef6e3`),
`docs/bifrost-self-test.md` + #17 contract (`ca02c70`), 4-axis-validation parity
(v0.17.5, tag `v0.17.5`), + this snapshot. Tags `v0.17.4`/`v0.17.5`. Push is the
operator's call. `graphify-out/GRAPH_REPORT.md` still runs dirty
(auto-regenerated artifact, not chased).
**Still standing from before:** Worldtree spec pin v0.29.0 (`562001a`) for the
conversation-API/TUI surface (untouched by the Bifrost work). Codex-first pilot
**Still standing from before:** Worldtree spec pin now v0.35.16 (`f1b59f8`) for the
conversation-API/TUI surface (bumped 2026-06-17 from v0.29.0/`562001a`). Codex-first pilot
still dormant (no codex session spun up — see the 2026-05-29 decision). Open
issues: #10 (subject migration, deferred), #11 (AdminEvents pane, deferred).
@@ -142,6 +240,33 @@ decision. Captures rationale that won't be obvious from code alone.
- `[2026-06-15]` **Providers run as dev-box BACKGROUND SHELLS, not infra-ops/systemd** (operator call — it's a dev box). `ratatoskr-provider` (affect) + a future `ratatoskr-memory-provider` as background processes; no productionization track.
- `[2026-06-15]` **Affect plane shipped (v0.17.2) + LIVE-PROVEN end-to-end against real Worldtree v0.35.2.** Personal handshake 200 + `affect.emit` 200 from `10.250.50.152` → durable row persisted (opacity held). HS256 key = the consumer's Heimdall API-key STRING utf-8-encoded (NOT base64/raw — the tripwire); cross-subnet route + `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist all held (infra-ops-owned). worldtree-dev confirmed ADR-0009 holding as designed.
- `[2026-06-16]` **#295 cold-recall miss root-caused — UPSTREAM, branch (a) scope-axis asymmetry.** A self-driven bound cold-recall probe (our own client, consumer-key bearer) captured the inbound pair via the new observe log: Worldtree's recall filter carries `{end_user, agent_self}`; our chunks are `{end_user}`-only; AND-matching drops everything on `agent_self` → 0 hits. Our store + search are SOUND; the fix is Worldtree-side. F2 (question-promotion) → **#296** research; F1 (recall-miss) → **#297** research (worldtree-dev's Worldtree-local per-scope-union fix, HELD pending the lattice question).
- `[2026-06-16]` **agent_self → make it CANONICAL (operator decided A).** The cross-repo "is agent_self a valid bifrost scope axis?" question: bifrost's reference lattice is `{end_user, group, tenant}` only (agent_self → `invalid_filter` 400); Worldtree emits agent_self (`bifrost_memory_store.py:479` #248 agent-self primitive). Operator chose canonical-not-re-expressed; worldtree-dev filed the lattice-addition with bifrost-dev (thread `01KV7PXF…`). **Implication: our store's permissive axis-acceptance becomes CORRECT once bifrost adds agent_self — so do NOT add axis-validation; our missing `_validate_scope_filter` is HELD, not a bug to fix.** #297 union build held until the axis lands.
- `[2026-06-16]` **Self-drive auth identity: bound session-create uses the CONSUMER Heimdall key as bearer, NOT `WORLDTREE_API_KEY`.** Worldtree signs the Bifrost handshake JWT with the session-create bearer (canary key → handshake 401; consumer key → 200). Two keys, two identities. Proven by hand; documented in `docs/bifrost-self-test.md`; load-bearing for #17's Bind half.
- `[2026-06-16]` **Issue #17 v1 scope locked (operator 1A/2A): single-plane bind + dispatch-layer op-feed.** `BifrostBindingRequest` is one `endpoint_url` (one plane per session); composite-both-planes endpoint PARKED. Observe = structured op-feed instrumented at the DISPATCH layer (bifrost passes ctx to upsert_many but NOT search/get/delete — `memory.py:244`), session-level correlation; turn-correlated pane UI PARKED (needs turn_id, TBD). Direct in-session TDD (live-smoke load-bearing). Contract `docs/contracts/issues/17.contract.md` written, `/heid`-design-consulted + `/heid-contract-review`-panel'd + fixed (validates OK). NEXT: TDD.
- `[2026-06-16]` **Provider stores confirmed byte-faithful to bifrost's AND reference** (`reference_server/memory.py:398` `_matches_scope` = `all(...)`, identical to ours). OR-union was considered + rejected (ecosystem-wide change); flagged the silent-zero foot-gun to bifrost-dev (docs-only landed, bifrost stays 0.6.4). Do NOT flip `_scope_matches` to OR.
- `[2026-06-16]` **agent_self lattice SHIPPED both sides → our axis-validation gap CLOSED (v0.17.5).** bifrost 0.7.0 / wire v0.5 adds agent_self to the scope lattice `{end_user,group,tenant,agent_self}` (#10, driven by our foot-gun flag via bifrost-dev); Worldtree pinned 0.7.0 + canonical-synced the v0.5 spec (v0.35.11, `c860fb0`). **SUPERSEDES the prior "HELD, do NOT add axis-validation" note** — we DID add `_validate_scope_filter` (4-axis) to match the reference (bifrost-dev's recommendation, purely additive; out-of-lattice → InvalidFilter). #297 union build unblocked. Memory provider restarted on 0.7.0.
- `[2026-06-16]` **#17 contract reviewed + the debug-assist arc fully closed.** `/heid` design consult + `/heid-contract-review` panel both run on `docs/contracts/issues/17.contract.md` (validates OK). The #295 debug-assist that opened the session is closed end-to-end: root cause (scope-axis asymmetry) → #296/#297 research issues + corpus → a shipped bifrost protocol change (agent_self canonical) → our store at parity. NEXT durable step: #17 TDD (tracked: Gitea #17 + the contract).
- `[2026-06-16]` **Repinned bifrost 0.7.0→0.8.0 + reimplemented memory `search` to the v0.6 scope split (operator-directed).** `scope_filter``scope_all` (AND) + `scope_any` (OR/union over a list of conjunctive scopes), bifrost #11 — the canonical resolution of the #295/#297 silent-zero. **SUPERSEDES the "Provider stores byte-faithful to AND reference / Do NOT flip `_scope_matches` to OR" entry above**: the reference itself now does OR via `scope_any` (a NEW field — `scope_all` keeps the old AND semantics; this is an additive split, not a flip of the AND predicate). Store / contract (v1.2) / tests at parity with the v0.6 reference; provider bounced onto 0.8.0 with a wiped DB (operator: "nothing of value"). Cold recall now gated only on Worldtree emitting `scope_any` (#297). **#17's contract has stale `scope_filter`/`_scope_matches`-AND references (its `assumptions`, INV-004, and the op-feed `search → req {scope_filter}` summary shape) — update those to `scope_all`/`scope_any` when #17 TDD starts; INV-004's intent (observe must not alter scope semantics) still holds.**
- `[2026-06-17]` **Worldtree spec pin bumped v0.29.0→v0.35.16 (`562001a`→`f1b59f8`); cold recall CLOSED end-to-end.** Worldtree shipped #297 (client-side per-scope-value union recall) + #298/#299 (adopt the bifrost v0.6 `scope_any`/`scope_all` wire) — it now emits `scope_any` on recall, the upstream half that pairs with our v0.17.6 provider. Re-vendored `conversation-api-spec.md` + `conversation_api.contract.md`; diff-reviewed the 285-commit catch-up — no client-breaking changes (#211 agent-slug rename saga→echo/actor→mask is slugs-only; #245 end_user_id+memory-scope; #187/#188/#219 Tier-3 define/PATCH additive; error codes stable). Shipped as a `pin:` commit, NO package bump (docs/pin-only, no ratatoskr code; per SemVer SKIP for docs-only). **Remaining proof: a live cold-recall re-smoke against a personal instance on v0.35.16** — the smoke target ran v0.35.2/.3, needs infra-ops to update it.
- `[2026-06-17]` **End-to-end cold-recall proof RAN — our stack proven, #296 isolated.** Against personal WT v0.35.16 with restored `ratatoskr:sindra`: #297/#298 union recall (WT emits `scope_any`), write path (promote→upsert→persist), and cold read (scope_any returns hits) ALL proven. Lone gap = upstream #296 extraction quality. Refines the spec-repin entry's "cold recall closes end-to-end": the WIRE closed; actual fact-recall is #296-blocked. (triage `01KVBBH0…`)
- `[2026-06-17]` **DELETE+redefine `ratatoskr:sindra` (operator-authorized; pre-v1 debug surface).** She SURVIVED the rebuild but was STALE (dead model `artemis-31b-v1i` + no memory block from Phase-2.0 tooling); memory is immutable post-define, so DELETE+redefine was the only path. v0.35.16 define takes **`role`** (capability), NOT `model`: `role:"character"` → first-healthy bind `mistral-small-4`; `memory:{}` trips the promotion gate (GET does NOT echo `memory_config`). Reasoning bind needs a `character`-role bind reorder (worldtree-dev flagged to operator). Our `tier3.py` define is Phase-2.0-stale (no role/memory/persona layers) — untracked modernization follow-up.
- `[2026-06-17]` **Promotion = 4-trigger hybrid (worldtree-dev, code-grounded):** salience (regex, 90s rate-limit) / `turn_count≥6` / context_pressure / **idle `≥10min` (unconditional on quality)**; per-turn `plan_promotion_run` for consumer_defined. **DELETE does NOT drain/promote** (delete-is-delete, #276) — idle `≥10min` is the deterministic flush. The seed promoted at ~4min (per-turn salience/turn-scan, before the idle floor).
- `[2026-06-17]` **#296 triage sent to worldtree-dev** (`01KVBBH0QRGT34GDJQEX5ACTT1`): extraction SUBJECT-INVERSION (promotes assistant prose, drops the user's fact) + META-DESCRIPTION-not-content; verbose-persona aggravator. The WAD-vs-bug cut resolved to BUG (extraction quality), not idle-gating.
- `[2026-06-18]` **Tier-3 memory PROVEN end-to-end live**`ratatoskr:terse-probe` recalled a seeded user fact in a COLD history-free session (scope_any → 1 hit @ cosine 0.6994 → "Dark chocolate with sea salt"). The session's headline; closes the opening "how far from Tier-3 memory" question for normal agents. (proof in this session's transcript)
- `[2026-06-18]` **#296 Stage 1 (v0.35.19) validated live + bisect localized the residual to VERBOSE-PERSONA VOLUME.** Stage 1's recallability admission gate works for normal turns (clean distillate, `distiller_model_id="identity"`); the terse-probe-promotes / sindra-drowns bisect pinned the remaining failure to symptom 1 (assistant-prose volume crowding out the user fact at extraction).
- `[2026-06-18]` **#296 Stage 2 (v0.36.0) MERGED at worldtree-codex** — user-only one-call-per-turn extraction, the STRUCTURAL fix for the subject-inversion drop sindra exposed. Staged per operator: core fix now (validated by our terse-probe PASS + WT test suite); hard-linguistic layer (larger-model escalation, marker-less-fact regex-demotion) tracked at Worldtree #305. We handed worldtree-dev a live-validated eval fixture PAIR for #305 (terse-proof POSITIVE chunk + choc-v2/sindra NEGATIVE-drop).
- `[2026-06-18]` **Full root-cause chain traced + confirmed** (a3466fab ledger pull via infra-ops): v0.35.16 verbose-sindra EMITS 2 meta-descriptions → v0.35.19 gate hard-rejects them (emit-then-reject) → v0.36.0 user-only pass emits the fact. The 2 v0.35.16 chunks ("Assistant's detailed flirtatious response…" / "…head tilt, eye locking, freckles…") are textbook subject-inversion; banked as #305 negative-fixture examples.
- `[2026-06-18]` **FULL-COVERAGE proof — verbose-persona memory works under Stage 2 (v0.36.0).** Re-smoked `ratatoskr:sindra-probe` (sindra's theatrical prompt; FRESH agent + end_user `verbose-v2` to dodge the WT promotion-dedup): the high-volume turn promoted the user fact CLEANLY (not a meta-description) and cold-recalled @ 0.694. Confirms `:8081` is on v0.36.0 and closes the verbose-persona caveat end-to-end. Then started `ratatoskr-web` (`:8765`, consumer key, `ratatoskr:sindra` in the picker) for the operator's Sindra session — **persona + debug only; the web client does NOT bind to `:8391` (#17 unbuilt), so no memory persistence in the web chat** (memory-enabled chat is the #17 build, or a manual bound session).
- `[2026-06-18]` **Tier-3 persona_state is a Worldtree limitation (not ours); sindra → thoughtful-character.** The web Persona-pane 404 ("persona not available") = `persona_state` hard-404s ALL Tier-3 (colon-id) agents by design (`api.py:1220`); a persona block in the define doesn't help (endpoint short-circuits before the row). Pinged worldtree-dev on the Tier-3 persona_state roadmap (thread `01KVCR6P…`) + de-uglied our web pane (`index.html loadPersona`) to render a clear message (v0.17.7). Separately switched `ratatoskr:sindra` to the `thoughtful-character` role (→ `mistral-small-4-reasoning`) via DELETE+redefine (kept her prompt; added an inert persona block — inert because the pane's 404 is endpoint-side + we don't bind affect). Tier-3 persona/affect observability is gated on BOTH WT shipping Tier-3 persona_state AND our #17 affect-binding.
_For per-issue TDD implementation notes, Volva findings, and contract amendments, see the git log — every per-issue commit carries a structured message capturing the trail._
## Tried and abandoned
@@ -170,3 +295,17 @@ defense against re-attempting the same cul-de-sac.
- `[2026-06-15]` **"Sindra hasn't been registered" was an under-verified inference — WRONG.** Concluded it from grepping ratatoskr's CODE (`sindra` absent from `src/`), but Tier-3 registration is SERVER-SIDE (`POST /agents/define` on the Worldtree instance) — a code grep structurally can't see it. Registration IS required to use a Tier-3 character (a session against an unregistered `agent_id` 404s), so since Sindra has been used, she WAS registered (`ratatoskr:sindra`). **Rule: to check whether a Tier-3 agent exists, query the Worldtree instance's `GET /agents`, never the consumer repo's code.** (Residual: the v0.35.2 personal rebuild may have wiped her — re-verify.)
- `[2026-06-14]` **Artifact-only contract review can't validate against a dependency's ACTUAL behavior.** `/heid-contract-review` sees only the contract, never the external library (bifrost) — so "the consumer under-built against bifrost's real semantics" is invisible to it by construction (the affect idempotency model shipped wrong because of this). Real-lib TDD against the shipped library + the executable reference store + the #195 parity test are the gate for any consumer plane with non-trivial state semantics. Don't treat a clean contract review as evidence the code matches the dependency.
- `[2026-06-15]` **"byte-equal" round-trip slip propagated affect→memory via copy-paste.** The affect contract's byte-identical→semantic fix reappeared in the memory contract's INV-001 (sibling copy). Only an INDEPENDENT `/heid-contract-review` of the memory contract re-caught it. **Paraphrase every sibling contract fresh — don't amortize one review across a family; copies carry the parent's slips.** (also a feedback auto-memory)
- `[2026-06-15]` **Canonical sync retired the issue-scoped parser staleness** (predicted by the 2026-05-21 entry's "until canonical bumps"). `contract_parser.py` synced to v2.1 (`f1fdfdb6→e10a4460`, commit `d85ab43`): now validates issue-scoped frontmatter (`target_module`/`scope`/`prd`) + four v2.1 test categories (scenario/trace/adversarial/property). Issues #3/#4 went FAIL→WARN (0 errors). The old "treat parser ERROR-on-issue-scoped as expected" note no longer applies.
- `[2026-06-15]` **Refreshed #3/#4 presenter contracts to the shipped TUI model** (commit `335c835`). Both still described the abandoned single-`RichLog` double-display model; rewrote to the 4-pane live-Markdown reality (v0.5.0v0.14.0 + Worldtree #201/#204) across INV-005, the `[performance]` constraint, the COMPOSE sketch, the `CLASS TuiPresenterState` block, both `render`/`_stream_turn_worker` blocks, and the `_cancel_via_sse` call site — plus the STEPS the v2.1 parser flagged missing. Code unchanged; contract-truth catching up to shipped code. Scope ballooned one-block→contract-wide mid-task; surfaced to operator before rewriting the INV-005 trade-off invariant.
- `[2026-06-15]` **Memory plane TDD'd + shipped** (commit `cd12951`, v0.17.3). Impl decisions worth keeping: vec0 `distance_metric=cosine` set at table creation (`score = 1 distance`); `search` over-fetches ALL candidates by cosine then scope-filters in Python so `top_k` counts IN-SCOPE hits (INV-005, contract STEP 2 `indicative`); idempotency_id = reference 4-tuple `("default",verb,_ctx_actor(ctx),key)` pipe-joined as the SQLite PK, digest = sha256 canonical-JSON; `_ctx_actor` = `job_id|jwt_sub|session_id` (memory reference's 3-level, vs affect's 2-level). **heid-code-review panel returned zero true drift**; adopted 5 cheap contract-anchored fixups (scope_filter dict guard, `top_k≤0→[]`, stronger scope-isolation / delete-hit-search / handshake-POST tests), accepted 6 with reasoning. **Partial-map optimistic-lock semantics pinned to the reference via an `expected_revisions` parity test** — resolved a Hulda finding deterministically (the affect-plane lesson: TDD against the shipped lib is the gate, not judgment).
- `[2026-06-15]` **Memory provider LIVE-PROVEN against personal v0.35.3 (persist + dispatch + search-correctness); recall-injection is upstream.** worldtree-dev's Tier-3 promotion recipe (via infra-ops): memory-call fires from Tier-3 PROMOTION, gated at `service.py:2623` on `ctx.kind=="consumer_defined"` AND `ctx.memory_config is not None` (the agent must be DEFINED WITH a `memory` block — `ValidatedMemoryConfig {tier3_dreaming:false}`, dim 1024) AND handshake-granted memory caps AND `embedding_dim==1024`. `memory.agent_self_enabled` is NOT the gate (only the #248 self-candidate branch). Binding = `POST /sessions BifrostBindingRequest{endpoint_url}`, handshake `caps=["affect","memory"]`, **`binding.scope` null** (per-op scopes auto-minted: upsert_many→`memory:write`, search→`memory:read`). A `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist gates the endpoint (Worldtree-side config — infra-ops added `:8391`). HTTP + HS256 both work in dev. (smoke wiring thread `01KV7D82MJYB…`)
- `[2026-06-15]` **Diagnostic: our recall-search is SOUND — the cross-session recall gap is UPSTREAM, not the store — and it caught an upstream bug.** Embedded the recall query via gateway `qwen3-embedding` + searched our live store directly → the dark-chocolate fact recalls at cosine 0.60, correctly ranked above the unrelated name fact (0.16). So the cold-session recall failure is Worldtree's recall-assembly/injection (hits not reaching the prompt), NOT our search. ALSO found a latent UPSTREAM bug: a recall QUESTION got promoted as a durable chunk and ranks **#1 (0.70 > the fact's 0.60)**, polluting recall. Relayed to worldtree-dev (thread `01KV7JH8…`). **This is exactly #17's thesis — ratatoskr-as-provider caught an upstream bug invisible from the chat side.**
- `[2026-06-15]` **"Wire 200 ≠ recall works" — prove recall efficacy at the model's answer in a COLD (history-free) session, not on the wire.** A `search`/memory-call returns 200 whether or not its results are injected into the prompt, and same-session "recall" can be plain session history. infra-ops' cold cross-session probe caught my premature "all-green" (search dispatched 200, model had no memory). Don't call cross-session recall proven from a clean wire.
- `[2026-06-15]` **Issue #17 filed — Bifrost-binding for the chat client (self-drive + correlated-log affect/memory ops).** REVERSES design-brief §6's "no Bifrost-binding consumer support" — that negative clause predates ratatoskr's provider identity (2026-06-14), so the canary now owns both ends but its client can't drive its own provider (`create_session` sends only `{agent_id, end_user_id}`; no Bifrost `endpoint_url`). Today's smoke proved the substrate (bind→dispatch→persist); only the observe/log channel design (open question #5) remains. The recall-injection caveat is upstream and doesn't block #17. NEXT on #17: `/heid` consult on the now-grounded framing → contract → TDD. (tracked: Gitea #17, labels enhancement/observability/tui)
- `[2026-06-16]` **`scripts/contract_drift_check.py` defaults `GITEA_REPO` to "Worldtree"** (line 74), so a bare run in ratatoskr false-positives DRIFT by hashing Worldtree's same-numbered issue. Always `export GITEA_REPO=ratatoskr GITEA_OWNER=vh` (env.sh leaves the GITEA vars commented out) before running the drift-checker here.
- `[2026-06-16]` **My #295 coupling hypothesis (the promoted question crowds out the fact at small top_k) was REFUTED** — worldtree-dev's recall over-fetches `top_k=128` (`injector.py:203`/`_store_helpers.py:101`), so the question can't crowd the fact out at search level. Reasonable cross-frontier hypothesis, correctly framed as a hypothesis not a conclusion; the real cause was the scope-axis asymmetry. Lesson: offer provider-side hypotheses, let the upstream owner check them against their code.
- `[2026-06-16]` **Contract drifted from its own design in two spots, caught only by `/heid-contract-review` (not same-author paraphrase):** the `OpEvent` dataclass omitted the `turn_id` that INV-005 promised; the `session_id` comment said "None for search/get/delete" contradicting the dispatch-layer design (the JWT carries session_id for all verbs at dispatch). Cross-model paraphrase is load-bearing for catching an author's own contract-vs-intent drift.
- `[2026-06-16]` **"No promotion" was checked TOO EARLY — Tier-3 promotion is ASYNC (lands AFTER the SSE turn-end).** The cold-recall probe's immediate post-turn fixture check showed 3 chunks (no promotion); a later check (during the v0.17.5 provider restart) found 5 — the probe HAD promoted 2 chunks (its question `acc3d49` + the model's non-answer `4773704`), just late. Don't trust an immediate post-turn fixture snapshot to judge promotion; it lands after the turn completes. (Same family as the "wire-200 ≠ recall, prove it in a cold session" lesson, extended to promotion timing — and the reason #17's contract pins a post-turn grace window + fixture before/after assertion.)
- `[2026-06-17]` **"sindra is GONE" (infra-ops, from `GET /agents` + admin token) was a FALSE NEGATIVE.** Consumer-defined Tier-3 agents are OWNER-SCOPED (separate `consumer_agents` table) — invisible to the foundational `GET /agents` roster even with an admin token. To check whether a Tier-3 agent exists, `GET /agents/<owner>:<name>` with the OWNER key. (Extends the 2026-06-15 "Sindra registration under-verified" lesson — even the roster query can't see consumer agents; only the owner-scoped point-GET does.)
- `[2026-06-17]` **"Promotion didn't fire → #296" was PREMATURE — twice over.** (1) Polled the op-feed only ~2min, but the upsert landed at ~4min — promotion is async + multi-trigger; watch a longer window. (2) It DID fire; the real bug is extraction QUALITY (assistant-prose promoted, user-fact dropped), not non-firing. "No upsert while a session is live and `<10min` idle" is WAD, not a miss. (Extends the 2026-06-16 "checked too early" entry with the multi-trigger timing + the fired-but-bad-content distinction.)
- `[2026-06-18]` **Wiping our `:8391` store does NOT reset Worldtree's promotion-side dedup** — the v0.35.19 sindra re-smoke (same agent + fresh end_user `choc-v2`) returned `reason_code=noop_duplicate` / `candidate_count=0`: the extractor NEVER RE-RAN, dedup short-circuited against the earlier `sindra-proof` promotion. So a "0 chunks / empty store" result can be a DEDUP artifact, not Stage 1 fail-closing — I mis-read it as fail-closed, worldtree-dev caught it from the ledger. **For a clean promotion smoke, use a BRAND-NEW agent + end_user (never-used names)** — exactly why terse-probe was valid. (Also: `llm_calls_used=0` is NOT the "did the extractor run" tell — `noop_duplicate` is; a3466fab promoted 2 candidates with `llm=0`.)
+3 -3
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "ratatoskr"
version = "0.17.3"
version = "0.17.7"
description = "Worldtree Conversation API debug TUI — multi-pane observability dashboard"
readme = "README.md"
requires-python = ">=3.12"
@@ -30,7 +30,7 @@ web = [
# from the debug TUI. Recipe: bifrost/docs/implementing-a-consumer.md.
provider = [
"ratatoskr[web]", # reuse the starlette + uvicorn ASGI stack
"bifrost>=0.6.1", # consumer engines + library (0.6.0 yanked: circular import)
"bifrost>=0.8.0", # consumer engines + library (0.8.0/wire-v0.6: scope_filter split into scope_all (AND) + scope_any (OR/union, #11); 0.7.0/v0.5 added agent_self)
"jsonschema>=4", # bifrost runtime dep — envelope validation
"sqlite-vec>=0.1.6", # vector index for the memory plane (vec0 virtual table)
]
@@ -59,7 +59,7 @@ Repository = "https://gitea.phasefinal.com/vh/ratatoskr"
# Ratatoskr is built against Worldtree at this commit; the vendored
# spec snapshot in docs/ reflects that SHA.
[tool.ratatoskr.spec-pin]
worldtree-spec-rev = "562001af28d752c3a60d449c7ddd09f44fa9dc9a"
worldtree-spec-rev = "f1b59f8cd6fe41e497d0be9dad9d3110451f0d9a"
worldtree-version = "v0.29.0"
pinned-on = "2026-05-26"
+84 -11
View File
@@ -14,7 +14,9 @@ from __future__ import annotations
import hashlib
import json
import logging
import sqlite3
import sys
import time
from typing import Any
@@ -23,6 +25,7 @@ from bifrost.consumer import ConsumerRegistration, build_memory_app
from bifrost.memory import (
IdempotencyConflict,
InvalidArguments,
InvalidFilter,
RevisionMismatch,
StoreCapabilities,
)
@@ -31,6 +34,22 @@ from bifrost.reference_server import JwtVerifier
_SHORT_RETRY_TTL_SECONDS = 300
_DURABLE_JOB_TTL_SECONDS = 24 * 60 * 60
# bifrost wire v0.6 scope lattice: three subject axes + agent_self (the #248
# agent-identity axis, canonical since #10/v0.5). An axis outside it is InvalidFilter
# (-> memory.invalid_filter 400), matching bifrost's reference _validate_scope.
_SCOPE_LATTICE = {"end_user", "group", "tenant", "agent_self"}
# Inbound memory-call observability (#17 observe brick). A self-contained
# stdout handler so the lines reliably reach the provider's stdout regardless
# of uvicorn's logging config. INFO-level, no propagation to root.
_log = logging.getLogger("ratatoskr.provider.memory")
if not _log.handlers:
_h = logging.StreamHandler(sys.stdout)
_h.setFormatter(logging.Formatter("%(asctime)s [memory-provider] %(message)s"))
_log.addHandler(_h)
_log.setLevel(logging.INFO)
_log.propagate = False
def _ctx_actor(ctx: Any) -> str:
"""Reference `_ctx_actor`: actor = job_id | jwt_sub | session_id (never the record)."""
@@ -68,11 +87,37 @@ def _record_vector(record: dict) -> list[float]:
return [float(v) for v in value] if isinstance(value, list) else []
def _scope_matches(record_scope: Any, scope_filter: dict) -> bool:
"""INV-005: record is in-scope iff every scope_filter axis matches record["scope"]."""
if not isinstance(record_scope, dict):
def _scope_subset(record_scope: dict, filter_dict: dict) -> bool:
"""True iff record_scope has EVERY axis of filter_dict (AND). Empty filter = match."""
return all(record_scope.get(axis) == value for axis, value in filter_dict.items())
def _matches_scope(record_scope: Any, scope_all: dict, scope_any: list) -> bool:
"""INV-005 (wire v0.6, #11): a record passes iff
`(scope_all empty OR record scope_all) AND (scope_any empty OR it matches 1 element)`.
scope_any is OR/union over a LIST of conjunctive scope dicts (each AND-matched as a whole),
never single axes flattened together (the Worldtree #297 over-broadening foot-gun).
Byte-faithful to bifrost reference `reference_server/memory.py:_matches_scope`.
"""
scope = record_scope if isinstance(record_scope, dict) else {}
if not _scope_subset(scope, scope_all):
return False
return all(record_scope.get(axis) == value for axis, value in scope_filter.items())
if scope_any and not any(_scope_subset(scope, element) for element in scope_any):
return False
return True
def _validate_scope(scope_all: dict, scope_any: list) -> None:
"""STEP 1: scope_all is a flat dict, scope_any a list of flat dicts; every axis in BOTH
fields the v0.6 lattice {end_user, group, tenant, agent_self} (else InvalidFilter),
matching the reference `_validate_scope`."""
if not isinstance(scope_all, dict):
raise InvalidArguments("scope_all must be a flat {axis: value} dict")
if not isinstance(scope_any, list) or any(not isinstance(e, dict) for e in scope_any):
raise InvalidArguments("scope_any must be a list of {axis: value} dicts")
for scope in (scope_all, *scope_any):
if any(axis not in _SCOPE_LATTICE for axis in scope):
raise InvalidFilter("scope_filter contains unsupported axis")
def _validate_injection(record: dict) -> None:
@@ -113,6 +158,11 @@ class RatatoskrMemoryStore:
) -> dict:
if not (isinstance(idempotency_key, str) and idempotency_key): # PRE-001
raise InvalidArguments("idempotency_key required")
_log.info(
"memory-call upsert_many REQUEST: %d record(s) idempotency_key=%s actor=%s scopes=%s",
len(records), idempotency_key, _ctx_actor(ctx),
[r.get("scope") for r in records],
)
# INV-002: idempotency_id = ("default", verb, actor-from-ctx, key); digest over payload.
digest = _payload_digest({"records": records, "expected_revisions": expected_revisions})
idempotency_id = "|".join(("default", "upsert_many", _ctx_actor(ctx), idempotency_key))
@@ -176,7 +226,8 @@ class RatatoskrMemoryStore:
vector: list[float],
*,
top_k: int,
scope_filter: dict | None = None,
scope_all: dict | None = None,
scope_any: list | None = None,
metadata_filter: dict | None = None,
include: dict | None = None,
fidelity_target: Any = None,
@@ -185,13 +236,34 @@ class RatatoskrMemoryStore:
raise InvalidArguments(f"vector length {len(vector)} != embedding_dim {self._dim}")
if metadata_filter: # PRE-002: v1 advertises no filterable metadata fields
raise InvalidArguments("metadata_filter is unsupported in v1")
if scope_filter is not None and not isinstance(scope_filter, dict): # STEP 1
raise InvalidArguments("scope_filter must be a flat {axis: value} dict")
scope_all = scope_all or {}
scope_any = scope_any or []
_validate_scope(scope_all, scope_any) # STEP 1 (raises InvalidArguments / InvalidFilter)
_log.info(
"memory-call search REQUEST: scope_all=%r scope_any=%r top_k=%s vec_dim=%d",
scope_all, scope_any, top_k, len(vector),
)
def _emit(rs: list[dict]) -> list[dict]:
_log.info(
"memory-call search RESPONSE: %d hit(s) %s",
len(rs),
[
{
"chunk_id": r["chunk_id"],
"score": round(r["score"], 4),
"scope": r["chunk"].get("scope"),
}
for r in rs
],
)
return rs
if top_k <= 0: # POST-001: at most top_k
return []
return _emit([])
total = self._conn.execute("SELECT COUNT(*) FROM memory_vec").fetchone()[0]
if total == 0:
return []
return _emit([])
# Over-fetch every candidate ranked by cosine distance, then scope-filter and
# take top_k — so top_k counts IN-SCOPE hits (INV-005), not pre-filter hits.
rows = self._conn.execute(
@@ -203,7 +275,7 @@ class RatatoskrMemoryStore:
results: list[dict] = []
for chunk_id, distance, record_json, revision in rows:
record = json.loads(record_json)
if scope_filter and not _scope_matches(record.get("scope"), scope_filter):
if not _matches_scope(record.get("scope"), scope_all, scope_any):
continue
results.append(
{
@@ -216,7 +288,7 @@ class RatatoskrMemoryStore:
)
if len(results) >= top_k:
break
return results
return _emit(results)
async def get(self, chunk_id: str) -> dict | None:
# INV-001: verbatim round-trip + an attached revision key, or None.
@@ -241,6 +313,7 @@ class RatatoskrMemoryStore:
async def delete_many(self, ids: list[str]) -> dict:
# One transaction: chunk row + its vec row leave together (no orphan vec rows).
_log.info("memory-call delete_many REQUEST: ids=%s", ids)
deleted = 0
with self._conn:
for chunk_id in ids:
+14 -1
View File
@@ -704,7 +704,20 @@ async function loadPersona(agentId) {
renderPersonaPane(snap);
setPersonaStrip(snap);
} else {
$("pane-persona").innerHTML = `<div class="empty">persona not available (HTTP ${esc(r.status)})</div>`;
let code = "";
try { code = (await r.json()).error_code || ""; } catch (_) {}
let msg;
if (r.status === 404 && code === "persona_not_configured" && agentId.includes(":")) {
msg = "persona telemetry isn't exposed for Tier-3 (consumer-defined) agents on this Worldtree yet — " +
"the agent still responds in character; only this affect / OCEAN readout is gated.";
} else if (r.status === 404 && code === "persona_not_configured") {
msg = "this agent has no persona configured.";
} else if (r.status === 403) {
msg = "persona telemetry requires the persona.read scope.";
} else {
msg = `persona unavailable (HTTP ${esc(r.status)}${code ? " · " + esc(code) : ""}).`;
}
$("pane-persona").innerHTML = `<div class="empty">${msg}</div>`;
}
} catch (e) {
$("pane-persona").innerHTML = `<div class="empty">persona fetch failed</div>`;
+109 -11
View File
@@ -10,7 +10,12 @@ from __future__ import annotations
import types
import pytest
from bifrost.memory import IdempotencyConflict, InvalidArguments, RevisionMismatch
from bifrost.memory import (
IdempotencyConflict,
InvalidArguments,
InvalidFilter,
RevisionMismatch,
)
from ratatoskr.provider.memory_store import (
build_memory_provider_app,
@@ -171,7 +176,7 @@ async def test_basic_search_ranks_by_cosine_with_recalled_view():
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_filter=scope)
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_all=scope)
assert [r["chunk_id"] for r in results] == ["c1", "c3"] # nearest to [1,0] by cosine
top = results[0]
assert top["chunk"] == c1 # verbatim chunk, no revision attached
@@ -191,7 +196,7 @@ async def test_scope_isolation_excludes_other_scope_even_if_closer():
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_filter={"end_user": "u1"})
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_all={"end_user": "u1"})
assert [r["chunk_id"] for r in results] == ["u1-far"] # u2-near excluded despite ranking first
@@ -214,18 +219,109 @@ async def test_search_wrong_vector_dim_rejected():
await store.search([1.0, 0.0], top_k=5)
async def test_search_non_dict_scope_filter_rejected():
# search STEP 1: scope_filter must be a flat {axis: value} dict
async def test_search_non_dict_scope_all_rejected():
# search STEP 1: scope_all must be a flat {axis: value} dict
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
with pytest.raises(InvalidArguments):
await store.search(_vec(1.0), top_k=5, scope_filter="u1")
await store.search(_vec(1.0), top_k=5, scope_all="u1")
async def test_search_non_list_scope_any_rejected():
# search STEP 1: scope_any must be a LIST of {axis: value} dicts (#11)
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
with pytest.raises(InvalidArguments):
await store.search(_vec(1.0), top_k=5, scope_any={"end_user": "u1"})
async def test_search_out_of_lattice_scope_axis_rejected():
# v0.6 scope lattice = {end_user, group, tenant, agent_self}; an axis outside
# it is InvalidFilter (-> memory.invalid_filter 400) in EITHER field, matching the reference.
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
with pytest.raises(InvalidFilter):
await store.search(_vec(1.0), top_k=5, scope_all={"bogus_axis": "x"})
with pytest.raises(InvalidFilter):
await store.search(_vec(1.0), top_k=5, scope_any=[{"bogus_axis": "x"}])
async def test_search_agent_self_axis_accepted():
# agent_self became canonical at wire v0.5 (#10) — admitted, not rejected.
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
await store.upsert_many(
[_chunk("a1", scope={"agent_self": "ratatoskr:smoke"})],
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(
_vec(1.0), top_k=5, scope_all={"agent_self": "ratatoskr:smoke"}
)
assert [r["chunk_id"] for r in results] == ["a1"]
async def test_search_top_k_zero_returns_empty():
# POST-001: at most top_k — zero means zero
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
await store.upsert_many([_chunk("c1")], idempotency_key="k1", ctx=_ctx())
assert await store.search(_vec(1.0), top_k=0, scope_filter={"end_user": "u1"}) == []
assert await store.search(_vec(1.0), top_k=0, scope_all={"end_user": "u1"}) == []
async def test_search_no_scope_matches_all():
# v0.6: both fields empty -> no scope constraint (match all, within top_k).
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
await store.upsert_many(
[
_chunk("u1", scope={"end_user": "u1"}),
_chunk("u2", scope={"end_user": "u2"}),
],
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(_vec(1.0), top_k=10)
assert {r["chunk_id"] for r in results} == {"u1", "u2"}
async def test_search_scope_any_unions_across_scopes():
# v0.6 (#11): scope_any is OR/union over a LIST of conjunctive scopes. A {end_user:u1}
# chunk AND an {agent_self:a} chunk are BOTH recalled in ONE call — the capability
# that resolves the #295/#297 silent-zero AND foot-gun (subset-scoped chunks now recall).
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
await store.upsert_many(
[
_chunk("subj", embedding=_vec(1.0, 0.0), scope={"end_user": "u1"}),
_chunk("self", embedding=_vec(0.9, 0.1), scope={"agent_self": "ratatoskr:sindra"}),
_chunk("other", embedding=_vec(0.8, 0.2), scope={"end_user": "u9"}),
],
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(
_vec(1.0, 0.0),
top_k=10,
scope_any=[{"end_user": "u1"}, {"agent_self": "ratatoskr:sindra"}],
)
assert {r["chunk_id"] for r in results} == {"subj", "self"} # union; u9 excluded
async def test_search_scope_all_and_scope_any_compose_by_and():
# v0.6: a record passes iff (record ⊇ scope_all) AND (matches ≥1 scope_any element).
store = open_memory_store(":memory:", embedding_dim=EMBEDDING_DIM)
await store.upsert_many(
[
# tenant t1 AND (end_user u1 OR u2) — only these pass
_chunk("t1u1", embedding=_vec(1.0, 0.0), scope={"tenant": "t1", "end_user": "u1"}),
_chunk("t1u2", embedding=_vec(0.9, 0.1), scope={"tenant": "t1", "end_user": "u2"}),
_chunk("t1u9", embedding=_vec(0.8, 0.2), scope={"tenant": "t1", "end_user": "u9"}),
_chunk("t2u1", embedding=_vec(0.7, 0.3), scope={"tenant": "t2", "end_user": "u1"}),
],
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(
_vec(1.0, 0.0),
top_k=10,
scope_all={"tenant": "t1"},
scope_any=[{"end_user": "u1"}, {"end_user": "u2"}],
)
assert {r["chunk_id"] for r in results} == {"t1u1", "t1u2"} # t1u9 fails any; t2u1 fails all
async def test_scope_isolation_fills_top_k_from_in_scope_past_higher_out_of_scope():
@@ -242,7 +338,7 @@ async def test_scope_isolation_fills_top_k_from_in_scope_past_higher_out_of_scop
idempotency_key="k1",
ctx=_ctx(),
)
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_filter={"end_user": "u1"})
results = await store.search(_vec(1.0, 0.0), top_k=2, scope_all={"end_user": "u1"})
# exactly top_k in-scope (the 2 nearest u1 chunks); the higher-ranked u2 chunk is excluded
assert [r["chunk_id"] for r in results] == ["u1-near", "u1-mid"]
@@ -271,7 +367,7 @@ async def test_delete_hit_removes_chunk_and_vec_row():
assert _row_count(store, "memory_chunks") == 1
assert _row_count(store, "memory_vec") == 1 # c1's vec row gone too (no orphan)
# delete_hit: search no longer surfaces it (vec/chunk coupling held)
hits = await store.search(_vec(1.0), top_k=5, scope_filter={"end_user": "u1"})
hits = await store.search(_vec(1.0), top_k=5, scope_all={"end_user": "u1"})
assert all(r["chunk_id"] != "c1" for r in hits)
@@ -361,7 +457,7 @@ async def test_parity_search_ranked_ids_vs_reference_through_dispatch():
await dispatch_memory_call(up, wctx, mine)
search_env = {
"operation": "search",
"args": {"vector": [1.0, 0.0], "top_k": 2, "scope_filter": {"end_user": "u1"}},
"args": {"vector": [1.0, 0.0], "top_k": 2, "scope_all": {"end_user": "u1"}},
}
rstatus, rbody = await dispatch_memory_call(search_env, rctx, ref)
mstatus, mbody = await dispatch_memory_call(search_env, rctx, mine)
@@ -388,7 +484,9 @@ async def test_parity_expected_revisions_vs_reference_through_dispatch():
"args": {"records": [_ref_record("a", [1.0, 0.0]), _ref_record("b", [0.0, 1.0])]},
"idempotency_key": "seed",
}
assert await dispatch_memory_call(seed, wctx, ref) == await dispatch_memory_call(seed, wctx, mine)
ref_seed = await dispatch_memory_call(seed, wctx, ref)
mine_seed = await dispatch_memory_call(seed, wctx, mine)
assert ref_seed == mine_seed
# partial map: only "a" is locked (revision 1); "b" is omitted from expected_revisions
partial = {
Generated
+5 -5
View File
@@ -190,14 +190,14 @@ wheels = [
[[package]]
name = "bifrost"
version = "0.6.1"
version = "0.8.0"
source = { registry = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" }
dependencies = [
{ name = "jsonschema" },
]
sdist = { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.6.1/bifrost-0.6.1.tar.gz", hash = "sha256:2eaf93c6da91faa6faa80a4c9a8d0c66161f4a7cc31ff041b0ae64daf3c161ea" }
sdist = { url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.8.0/bifrost-0.8.0.tar.gz", hash = "sha256:28194877c81a056a0803b052e86902c092e965d4ce63a5623d7a31240cedb645" }
wheels = [
{ url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.6.1/bifrost-0.6.1-py3-none-any.whl", hash = "sha256:ed505d2c08cf4cdd0a84c68ec42f8732b4c1baf7d72befe5eacf75d88381d5ce" },
{ url = "https://gitea.phasefinal.com/api/packages/vh/pypi/files/bifrost/0.8.0/bifrost-0.8.0-py3-none-any.whl", hash = "sha256:2aac5e4a7828d718389748a78dae6baeb5e9ee4a801a10c427c06c5cc7ed6597" },
]
[[package]]
@@ -1052,7 +1052,7 @@ wheels = [
[[package]]
name = "ratatoskr"
version = "0.17.3"
version = "0.17.7"
source = { editable = "." }
dependencies = [
{ name = "httpx" },
@@ -1086,7 +1086,7 @@ web = [
[package.metadata]
requires-dist = [
{ name = "bifrost", marker = "extra == 'provider'", specifier = ">=0.6.1", index = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" },
{ name = "bifrost", marker = "extra == 'provider'", specifier = ">=0.8.0", index = "https://gitea.phasefinal.com/api/packages/vh/pypi/simple/" },
{ name = "httpx", specifier = ">=0.27" },
{ name = "httpx-sse", specifier = ">=0.4" },
{ name = "jsonschema", marker = "extra == 'provider'", specifier = ">=4" },