Companion to lexical_recall_gate.py for the #397 order_by="chapter" flag (deployed
personal b184). Drives narrative/temporal queries and measures three axes end-to-end:
- ADOPTION: does the agent invoke order_by="chapter" for a temporal query? (schema
teaches it; usage varies — the #397 analog of query-formulation variance)
- MECHANISM (flag applied): are served hits' provenance.chapter monotonically
non-decreasing (earliest first)?
- VALUE (flag not applied): the relevance baseline is NOT chapter-sorted — the
applied-vs-not monotonicity gap is the flag's payoff.
Built against the real live shapes (order_by enum ["chapter"], result carries
ordered_by, provenance.chapter), not guessed. Baseline @ b184 (--runs=3, 15 trials):
adoption 47%, flag-applied->monotone 100%, not-applied->monotone 0%. So the mechanism
is a clean discriminator; the residual is adoption (same class as the crown's
query-formulation variance — the irreducible prompt-side gap).
Diagnostics fixture, no production runtime — no version bump. persistent-memory
snapshot alongside (commit-along).
Generalizes the crown repro (Worldtree #400 / thread 01KZETD98T) beyond its anchor
into a before/after regression instrument for the class property: when the corpus
holds a chunk whose text literally carries a queried surface form, a natural query
should serve >=1 such chunk at a usable rank.
- Anchors tagged stress (common word + competing dense cluster, e.g. crown) vs
control (distinctive name — should sit ~0% miss).
- Binary per trial: does a natural query serve >=1 term-containing chunk within
top-10 (USABLE_K)? Ranks >=8 flagged KNIFE-EDGE (the RRF fused-rank 9-11 window
residual worldtree-dev's decomposition measured).
- Real-world end-to-end: drives the agent (it composes its own reference_knowledge
query, as in production); --runs samples query-formulation variance to estimate a
true miss-rate.
- Extensible anchor list; --anchor filters.
This is the deciding instrument for the rerank_hybrid_floor lever: its stress-class
miss-rate (alongside brokkr's fleet demotion rate) rules the floor in or out after
the BM25 stemming fold deploys. Pre-fold baseline captured today (the "before"):
control 0% miss / stress[crown] 100% miss / 0% knife-edge, 11 trials.
Diagnostics fixture, no production runtime — no version bump. persistent-memory
snapshot committed alongside (commit-along).
Two folds from worldtree-dev's cross-check on #393:
1. raw-ranking now reports mimir's ACTUAL search_library query (tool_start q=), the
reformulation seam it previously discarded — separates reformulation-at-the-agent
from ranking.
2. Name the two regimes by WING SCOPE (they were silently conflated): raw-ranking is
ALL-WING (mimir, ~9800 rows across kb+main+fiction); consumer is FICTION (donut is
fiction-scoped, ~1578). The consumer regime now reports where the ENTITY and DECOY
rank in Donut's fiction results, so reformulation-induced absence (ent@None) is
distinguishable from true subject-selection (ent present + mis-bind).
Corrects the prior "fiction-scope subject-selection with entity present" read: the fold
shows Donut DISTILLS "the guy with the roid rage" to bare "roid rage", so the entity
drops even at fiction scope and she binds a present decoy (Jack). Both agents lose the
entity but by different seams — mimir preserves the phrase (cross-wing dilution), Donut
distills it (fiction-scope absence). Unifying lever: disambiguating-vocabulary expansion.
Self-contained re-runnable probe for the attribute->entity resolution gap: a descriptive
query ("the guy with the roid rage") matches multiple canon subjects on a shared token,
so the intended entity can be absent from top-k while topically-adjacent decoys rank
above it, and the consumer sometimes binds to the wrong co-retrieved subject. Two
regimes: raw ranking (entity-absent-from-top-k, persona-independent) and consumer
classification (binds-entity vs mis-binds-decoy over N runs). Two-regime finding
(b172 -> v1.0.0b181): roid-rage mis-binding survives the #389 arc; dangerous-crown
mostly resolved by the bge rerank. Filed upstream as Worldtree #393; this is its
canonical fixture. Only dep is httpx (uv run --with httpx); config from env.
Two bugs R42 (brokkr-smithy-dev) surfaced on first live-index contact:
1. Session-reuse degradation. run_yardstick/run_term reused one mimir
session across terms; mimir returns EMPTY search_library results after
a session's first query (Worldtree #391), silently scoring every later
term a false-MISS. Fixed by making search_library and reference_knowledge
self-session (fresh session per call) so no caller can re-hoist it. Live
yardstick now reproduces all four anchors HIT top-10. Fresh-session-per-
query is the pinned arm-2 protocol; folded into the conventions docstring.
2. Curly-vs-ASCII apostrophe. _on_target substring-matched raw ASCII while
the b170 extraction stores U+2019, so possessive-named subjects
false-MISSed. _on_target now NFKC-normalizes + quote-folds both sides
(NFKC alone does not fold U+2019, so the explicit fold is load-bearing).
Adds tests/test_fiction_wing_probe.py covering the apostrophe fold both
directions with a negative control.
Self-contained, re-runnable probe requested by brokkr-smithy-dev for R42 (fiction-wing
retrieval characterization) and the standing #389 ranking acceptance gate. Two paths kept
separate by noise property: search_library (mimir, fixed-string, deterministic — ranking
arm) and reference_knowledge (donut, captures her reformulated tool_query — the query-
formulation/arm-4 surface). Scoring conventions baked in: high/medium/low RRF buckets
(0.030/0.016), on-target = a row whose excerpt names the subject, bucket-distribution over
N runs. Carries the frozen artifact yardstick (4 source-verified items + epithet-dropped
variants). Config from env (no secrets). Smoke-verified live: reproduces the Crown-HIT /
other-three-MISS baseline and the near-floor bucket flips.
vh