docs(#17): self-drive+observe contract, bifrost self-test runbook + snapshot
- docs/contracts/issues/17.contract.md — issue-scoped v2.1 contract for #17 (Bifrost-binding the chat client). v1 scope = single-plane bind + dispatch-layer op-feed (composite endpoint + turn-pane UI parked). Design consulted via /heid, paraphrase-gated via /heid-contract-review panel; two internal inconsistencies fixed (OpEvent turn_id reservation made literal; session_id-for-all-verbs correction). Validates OK, prd drift-clean. - docs/bifrost-self-test.md — reusable runbook for driving + observing the full Bifrost round-trip against our own provider (the manual form of #17; pins the consumer-key-as-bearer tripwire). - persistent-memory.md — snapshot: observe brick shipped, self-drive proven, #295 root-caused (upstream, scope-axis asymmetry) -> #296/#297, agent_self -> canonical decided.
This commit is contained in:
@@ -0,0 +1,163 @@
|
||||
# Bifrost round-trip self-test
|
||||
|
||||
How to drive **and** observe a full Tier-3 Bifrost round-trip against
|
||||
ratatoskr's *own* provider — bind a Worldtree session to our affect/memory
|
||||
store, fire a turn, and read exactly what Worldtree dispatched to us,
|
||||
correlated with the turn that triggered it.
|
||||
|
||||
This is the **manual form of issue #17** (self-drive + correlated-log
|
||||
affect/memory ops). Until #17 ships that capability inside the TUI/web/CLI,
|
||||
this runbook is the reproducible loop — and it is the substrate worldtree-dev's
|
||||
#296 (salience-algorithm research) and #297 (recall scope/assembly research)
|
||||
diagnose against.
|
||||
|
||||
> First proven by hand 2026-06-16 while isolating #295's recall leg.
|
||||
|
||||
## The load-bearing tripwire: which key signs the bind
|
||||
|
||||
A bound session-create authenticates as the **Bifrost consumer**, not the
|
||||
canary client. Worldtree signs the Bifrost handshake JWT with the
|
||||
**session-create bearer token** (HS256 shared-secret model); our provider
|
||||
verifies it against `RATATOSKR_HEIMDALL_KEY`.
|
||||
|
||||
So the bearer on `POST /sessions` **must be the consumer Heimdall key**
|
||||
(`RATATOSKR_HEIMDALL_KEY`, in `~/.config/ratatoskr/provider.env`) — **not**
|
||||
`WORLDTREE_API_KEY` (the mimir/foundational TUI key in `env.sh`). They are two
|
||||
different keys for two identities of ratatoskr.
|
||||
|
||||
| Bearer used on `POST /sessions` | Handshake at our provider |
|
||||
|---|---|
|
||||
| `WORLDTREE_API_KEY` (mimir/TUI key) | **401** `bifrost.auth_rejected` → 502 to client |
|
||||
| `RATATOSKR_HEIMDALL_KEY` (consumer key) | **200 OK** → session bound |
|
||||
|
||||
ratatoskr is two identities: the conversation-API **canary client**
|
||||
(`WORLDTREE_API_KEY`) and the Bifrost **consumer/provider**
|
||||
(`RATATOSKR_HEIMDALL_KEY`). Self-driving a bound session crosses into the
|
||||
consumer identity, so it uses the consumer key. #17's Bind half has to carry
|
||||
this distinction.
|
||||
|
||||
## Prereqs
|
||||
|
||||
- Provider(s) running on this box (nh3-dev, `10.100.10.50`):
|
||||
- memory → `ratatoskr-memory-provider` on `:8391`
|
||||
- affect → `ratatoskr-provider` on `:8390`
|
||||
- Each is a dev background shell, env-sourced from `provider.env`. The memory
|
||||
provider's stdout carries the inbound observe log (`[memory-provider]` lines
|
||||
added in `memory_store.py`).
|
||||
- The provider endpoint is reachable + allowlisted from Worldtree
|
||||
(`10.250.50.152`): `http://10.100.10.50:8391`. The allowlist
|
||||
(`BIFROST_CLIENT_ALLOWED_HOSTS`) is **Worldtree-side, infra-ops-owned** — if a
|
||||
bind 502s with a route/allowlist error, that entry is the thing to check.
|
||||
- A memory-enabled Tier-3 agent defined on the instance. `ratatoskr:smoke`
|
||||
(scope `end_user:smoke-user`) is the standing fixture; it is hidden from
|
||||
`GET /agents` (Tier-3 agents are not in the public roster) but resolves on
|
||||
session-create.
|
||||
|
||||
## Steps
|
||||
|
||||
```bash
|
||||
cd ~/development/ratatoskr
|
||||
set -a && . ~/.config/ratatoskr/provider.env && set +a # RATATOSKR_HEIMDALL_KEY etc.
|
||||
URL=http://10.250.50.152:8081 # personal Worldtree
|
||||
HK="$RATATOSKR_HEIMDALL_KEY" # the CONSUMER key — the tripwire
|
||||
```
|
||||
|
||||
**1 — Bind a fresh (cold) session to our provider.** `bifrost.endpoint_url`
|
||||
points at the plane you want (`:8391` memory, `:8390` affect); caps are
|
||||
negotiated by the handshake, not declared here (`BifrostBindingRequest` is
|
||||
`{endpoint_url, scope}` only, `additionalProperties:false`). A 201 means the
|
||||
handshake verified.
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "$URL/sessions" -H "Authorization: Bearer $HK" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"agent_id":"ratatoskr:smoke","end_user_id":"smoke-user",
|
||||
"bifrost":{"endpoint_url":"http://10.100.10.50:8391","scope":null}}'
|
||||
# -> 201 {"session_id":"...", "kind":"consumer_defined", ...}
|
||||
```
|
||||
|
||||
**2 — Snapshot the fixture** (to detect any promotion the turn writes):
|
||||
|
||||
```bash
|
||||
sqlite3 -readonly memory.db \
|
||||
"SELECT chunk_id, json_extract(record_json,'\$.verbatim.text') FROM memory_chunks;"
|
||||
```
|
||||
|
||||
**3 — Fire ONE turn** into the bound session, reusing ratatoskr's own SSE
|
||||
client (handles composite ids + no-read-timeout). Bearer = the consumer key:
|
||||
|
||||
```bash
|
||||
RATATOSKR_HEIMDALL_KEY="$HK" uv run python - <<'PY'
|
||||
import asyncio, os, httpx
|
||||
from ratatoskr.sse_client import stream_turn, Text, Done, Error, Cancelled
|
||||
SESSION="<session_id from step 1>"
|
||||
URL="http://10.250.50.152:8081"; KEY=os.environ["RATATOSKR_HEIMDALL_KEY"]
|
||||
async def main():
|
||||
async with httpx.AsyncClient(base_url=URL,
|
||||
headers={"Authorization":f"Bearer {KEY}","User-Agent":"ratatoskr-selftest"},
|
||||
timeout=httpx.Timeout(connect=10.0,read=None,write=10.0,pool=10.0)) as c:
|
||||
async for ev in stream_turn(c, SESSION, "What kind of chocolate do I like?"):
|
||||
if isinstance(ev, Done): print("ANSWER:", ev.response); return
|
||||
if isinstance(ev, Error): print("ERROR:", ev.error_code, ev.message); return
|
||||
asyncio.run(asyncio.wait_for(main(), 150))
|
||||
PY
|
||||
```
|
||||
|
||||
**4 — Read the inbound pair** from the provider's stdout (the observe brick).
|
||||
For a background-shell provider, that is the task output file; tail it:
|
||||
|
||||
```
|
||||
[memory-provider] memory-call search REQUEST: scope_filter={...} top_k=... vec_dim=1024
|
||||
[memory-provider] memory-call search RESPONSE: N hit(s) [{'chunk_id':..., 'score':..., 'scope':...}]
|
||||
```
|
||||
|
||||
**5 — Re-snapshot the fixture** (step 2's query). A new row = the turn was
|
||||
promoted (a salience-algorithm event; relevant to #296). Audit, don't blindly
|
||||
delete — promoted failure-surfaces may be wanted corpus.
|
||||
|
||||
## Reading the result
|
||||
|
||||
The `search REQUEST` `scope_filter` vs the `search RESPONSE` hit count is the
|
||||
whole diagnosis surface:
|
||||
|
||||
- **0 hits** → the filter didn't match any stored chunk. Compare the filter's
|
||||
axes against the stored `scope`. INV-005 requires **every** filter axis to
|
||||
match, so an extra axis on the filter that the chunks don't carry (e.g.
|
||||
`agent_self`) zeroes the result even when `end_user` matches. That's a
|
||||
**scope-build / persist-recall-symmetry** question (Worldtree-side).
|
||||
- **Hit present but the model says "no memory"** → we returned it; Worldtree
|
||||
dropped it downstream of search → **recall-assembly / injection**
|
||||
(Worldtree-side).
|
||||
|
||||
Either way our store + search are provable from this surface; the recall
|
||||
*efficacy* must be judged at the model's answer in a **cold (history-free)**
|
||||
session, never from a wire 200 (a `search` returns 200 whether or not its hits
|
||||
are injected).
|
||||
|
||||
### Worked example (2026-06-16, #295 → #297)
|
||||
|
||||
Cold turn "What kind of chocolate do I like?" against the `smoke-user` fixture:
|
||||
|
||||
```
|
||||
REQUEST: scope_filter={'end_user': 'smoke-user', 'agent_self': 'ratatoskr:smoke'} top_k=128
|
||||
RESPONSE: 0 hit(s)
|
||||
ANSWER: "I don't have access to your past preferences..."
|
||||
```
|
||||
|
||||
Root cause: the recall filter carried `agent_self` but the stored chunks are
|
||||
`{end_user: smoke-user}` only → the `agent_self` axis excluded all of them.
|
||||
Branch (a), scope asymmetry — fed to #297.
|
||||
|
||||
## Notes / foot-guns
|
||||
|
||||
- **HTTP, not HTTPS.** The spec requires `endpoint_url` be HTTPS; dev is relaxed
|
||||
via the Worldtree-side allowlist. Don't "fix" our provider to HTTPS to make a
|
||||
bind work — check the allowlist entry first.
|
||||
- **Cold means cold.** Reuse of a session with history can satisfy a "recall"
|
||||
from plain conversation history. Always bind a *fresh* session for a recall
|
||||
probe.
|
||||
- **Stray sessions** created by probes are harmless empty rows on the dev
|
||||
instance; no cleanup required.
|
||||
- This loop is the thing #17 productizes into the chat surfaces; when #17 lands,
|
||||
the bind+observe steps move inside the TUI/web/CLI and this runbook becomes the
|
||||
underlying contract check.
|
||||
Reference in New Issue
Block a user