docs(diagnostics): fold mimir tool-query + wing-scope into #393 fixture

Two folds from worldtree-dev's cross-check on #393:
1. raw-ranking now reports mimir's ACTUAL search_library query (tool_start q=), the
   reformulation seam it previously discarded — separates reformulation-at-the-agent
   from ranking.
2. Name the two regimes by WING SCOPE (they were silently conflated): raw-ranking is
   ALL-WING (mimir, ~9800 rows across kb+main+fiction); consumer is FICTION (donut is
   fiction-scoped, ~1578). The consumer regime now reports where the ENTITY and DECOY
   rank in Donut's fiction results, so reformulation-induced absence (ent@None) is
   distinguishable from true subject-selection (ent present + mis-bind).

Corrects the prior "fiction-scope subject-selection with entity present" read: the fold
shows Donut DISTILLS "the guy with the roid rage" to bare "roid rage", so the entity
drops even at fiction scope and she binds a present decoy (Jack). Both agents lose the
entity but by different seams — mimir preserves the phrase (cross-wing dilution), Donut
distills it (fiction-scope absence). Unifying lever: disambiguating-vocabulary expansion.
This commit is contained in:
2026-08-06 22:45:35 -07:00
parent 6c83a3be85
commit 4f4b5ad947
+41 -17
View File
@@ -9,16 +9,26 @@ is resolved to an entity, which is the open problem. Downstream, the consumer so
to the wrong co-retrieved subject and cross-contaminates details (a confident, fluent mis-bind
assembled from real-but-mismatched rows, not a hallucination).
Two regimes, one variable set:
* RAW RANKING (persona-independent): drive the `mimir` librarian with the descriptive query
and its variants; show the top-k rows flagged for the intended ENTITY vs the DECOY. The
load-bearing fact is "intended entity absent from top-k."
* CONSUMER: drive `ratatoskr:donut` N times on the descriptive question; classify each answer
BINDS-ENTITY vs MIS-BINDS-DECOY vs OTHER and report the rate.
Two regimes at DIFFERENT WING SCOPES (the distinction is load-bearing — Worldtree #393):
* RAW RANKING — ALL-WING (mimir searches kb+main+fiction, ~9800 rows). Drive `mimir` with the
descriptive query + variants; report the query mimir ACTUALLY passed (tool_start q=, since it
reformulates) and where the intended ENTITY lands. Cross-wing dilution can push a one-arm
vector bridge out of the bge rescue window entirely -> entity ABSENT from top-k.
* CONSUMER — FICTION (ratatoskr:donut is fiction-scoped, ~1578 rows). Drive Donut N times;
classify BINDS-ENTITY vs MIS-BINDS-DECOY vs OTHER, and report where the ENTITY and DECOY rank
in her fiction-scoped results. The ranks tell WHICH failure fired per run: entity PRESENT +
mis-bind = subject-selection; entity ABSENT + a present decoy = reformulation-induced absence
(Donut distilled the descriptive phrase to bare tokens that don't carry the vocabulary bridge).
Empirically Donut mostly does the latter — she distills "the guy with the roid rage" to bare
"roid rage", so Juicer drops out even at fiction scope and she binds a present decoy (Jack).
Two-regime finding (2026-08-07, personal instance): roid-rage mis-binding SURVIVES the #389 arc
(b172 -> v1.0.0b181); "dangerous crown" is mostly RESOLVED by the bge rerank + source annotations
(it resolves to essentially one entity, "the Enchanted Crown of the Sepsis Whore"). The gap is
Two-seam finding (2026-08-07, v1.0.0b181) — both lose the entity, by DIFFERENT reformulation seams:
(1) mimir PRESERVES the phrase -> cross-wing dilution (kb+main+fiction) drops the entity from its
all-wing top-k; (2) Donut DISTILLS the phrase to bare tokens -> the entity drops even at fiction
scope (ent@None) and she binds a present decoy. The fold's ENT@/DEC@ ranks separate reformulation-
absence from true subject-selection per run. Unifying lever: disambiguating-vocabulary expansion
(the full phrase, or +attribute like "steroid") surfaces the entity at fiction scope — Donut's
expand-runs bind correctly. "dangerous crown" is mostly RESOLVED (one entity). Root gap:
attribute->entity resolution, upstream of names_subject by construction.
Self-contained: the only third-party dependency is httpx (`uv run --with httpx`). Config from env:
@@ -105,17 +115,27 @@ def _flag(excerpt: str, entity: str, decoy: str | None) -> str:
return " "
def _rank_in(res: dict, pattern: str | None) -> int | None:
"""Rank of the first row whose excerpt matches `pattern` (None if absent / no pattern)."""
if not pattern:
return None
rows = res.get("hits", res.get("results", [])) if isinstance(res, dict) else []
return next((i for i, r in enumerate(rows) if isinstance(r, dict)
and re.search(pattern, r.get("excerpt", ""), re.I)), None)
def raw_ranking(base, headers, end_user, case) -> None:
print("\n [raw ranking] intended entity absent from top-k?")
print("\n [raw ranking — ALL-WING/mimir] actual tool query (reformulation seam) + entity rank")
for query in case["variants"]:
sid = _session(base, headers, "mimir", end_user) # fresh session per query
_, res, _ = _drive(base, headers, sid, f"Use search_library to find: {query}")
mimir_q, res, _ = _drive(base, headers, sid, f"Use search_library to find: {query}")
rows = res.get("results", []) if isinstance(res, dict) else []
entity_ranks = [i for i, r in enumerate(rows) if isinstance(r, dict)
and re.search(case["entity"], r.get("excerpt", ""), re.I)]
top = entity_ranks[0] if entity_ranks else None
print(f" query={query!r:36} entity first appears at rank "
f"{top if top is not None else 'ABSENT (not in top-k)'}")
rank = entity_ranks[0] if entity_ranks else "ABSENT (not in top-k)"
# mimir_q is load-bearing for #393: separates reformulation-at-the-agent-seam
# (mimir distilled/expanded the phrase) from ranking (the tool ranked it low).
print(f" instructed={query!r:28} mimir_q={mimir_q!r:38} entity_rank={rank}")
for i, r in enumerate(rows[:6]):
if isinstance(r, dict):
ex = (r.get("excerpt") or "").replace("\n", " ")
@@ -125,15 +145,19 @@ def raw_ranking(base, headers, end_user, case) -> None:
def consumer(base, headers, end_user, case, runs) -> Counter:
verdicts: Counter = Counter()
print(f"\n [consumer] Donut x{runs} on {case['question']!r}")
print(f"\n [consumer — FICTION/donut] x{runs} on {case['question']!r}"
f" (ent@/dec@ = rank in Donut's fiction-scoped results)")
for run in range(1, runs + 1):
sid = _session(base, headers, "ratatoskr:donut", end_user) # fresh session per run
q, _, ans = _drive(base, headers, sid, case["question"])
q, res, ans = _drive(base, headers, sid, case["question"])
binds = bool(re.search(case["entity"], ans, re.I))
mis = bool(case["decoy"]) and bool(re.search(case["decoy"], ans)) and not binds
v = "BINDS-ENTITY" if binds else ("MIS-BINDS-DECOY" if mis else "OTHER")
verdicts[v] += 1
print(f" run{run}: {v:16} q={q!r:34} :: {ans.strip()[:80]}")
er, dr = _rank_in(res, case["entity"]), _rank_in(res, case["decoy"])
print(f" run{run}: {v:16} ent@{er} dec@{dr} q={q!r:30} :: {ans.strip()[:56]}")
# a present entity (ent@ not None) co-occurring with a mis-bind is subject-selection,
# NOT ranking-absence — the fiction-scope half of the #393 two-mechanism split.
print(f" >>> {case['label']}: {dict(verdicts)}")
return verdicts