Commit Graph
1309 Commits
Author SHA1 Message Date
vh a5ddfed81e memory: snapshot — McCarthy and Krakauer D1 built, Krakauer's quotation scope is an open operator call 2026-09-17 08:26:52 -07:00
vh 4be063071a lv-krakauer D1: 126 units, 422,880 words — and an unmeasured fraction is not his prose
The first non-fiction corpus in this line. Builds clean and should not be trained on until
an operator scope call is made; the reason is in the module docstring and the manifest.

  into-the-wild               25u   67,606w  caps-title    [smallcaps 21][back -1,015][epi -52]
  missoula                    32u  115,841w  chapter-word  [smallcaps  8][front -858][back -2,874]
  under-the-banner-of-heaven  33u  118,171w  caps-title
  where-men-win-glory         36u  121,262w  chapter-word  [smallcaps  3][front -1,548][back -6,093]

⚠⚠ THE UNRESOLVED PROBLEM IS QUOTATION, AND IT IS NOT MEASURED BECAUSE IT CANNOT BE.
Krakauer quotes constantly and at length -- McCandless's journals and letters, Tillman's
diaries, court transcripts, depositions, Mormon historical documents, and whole paragraphs
of Jack London and Wallace Stegner at the chapter heads. In print those are indented or
italic; the extraction lost both, so inside the master they are ordinary paragraphs and no
signal this builder can read separates them from his own sentences.

Only 52 words were removable -- chapter-head epigraphs whose all-caps attribution line
survived. That is 0.01% and it is NOT the answer: the method would report 0.0% for a book
made entirely of undated block quotes. The stated floor rather than the number is what a
reader needs. This is the same error as excluding The Torrents of Spring from Hemingway --
another author's style under the target's name -- distributed rather than concentrated, and
the fraction is unknown. Scope is the operator's call, exactly as fiction-only was.

THREE DEFECTS THE NAME GUARD CAUGHT, none of which the build would have reported otherwise:

  1. Back matter searched only the LAST unit. Where Men Win Glory's ACKNOWLEDGMENTS sits at
     94.8% and the splitter made 41 units, so the apparatus landed in unit 37 with NOTES and
     BIBLIOGRAPHY after it -- all past a strip that only looked at unit 41. Into the Wild
     kept its acknowledgments AND a full-page advertisement for another of his books. Now
     windowed to the last 25% and cut before the split.
  2. Relying on the splitter to drop front matter did not work. Units begin at the first
     heading mark, and in two works the ebook's table of contents sits above the author's
     note -- giving the splitter a `Chapter Thirty-Two` to start on, so unit 1 swallowed the
     apparatus and its signed `Jon Krakauer , February 2015`. Now cut at that signature,
     windowed to the first 10%.
  3. Zero was the wrong bar. 21 survivors became 2, and both were read: `Lewis Krakauer
     loved his five children deeply` is Krakauer writing about his own father in the two
     autobiographical chapters of Into the Wild, and the other is a reader's letter he
     quotes calling him a kook. Hemingway's own name in his corpus was always publisher
     apparatus, so 0 was right there; this author writes about himself. The allowance is
     pinned at 2 and every survivor is printed with context, so a master change or a strip
     that stops working fails loudly instead of widening in silence.

Both strips are windowed in OPPOSITE directions from McCarthy's, which is the point worth
carrying: McCarthy's apparatus is at the end and the earliest marker wins; Krakauer's is at
both ends and the same marker words appear in his front matter at 0.0-0.6% of the file.
2026-09-17 08:24:59 -07:00
vh f3bf3ca89c lv-mccarthy D1: 167 units, 584,756 words, and a style that looks exactly like damage
Six complete novels from the licensed Kvasir masters. Same record schema as the Brontë,
Yarros and Hemingway builders, so entities.py, rename.py, leak_gate.py and the trainers run
unchanged. Splits via the new shared split_units module.

  all-the-pretty-horses  33u   99,242w  paragraph-blocks  [back -1,768w] [drop cap restored]
  blood-meridian         23u  116,651w  roman-numeral     [back   -354w]
  cities-of-the-plain    30u   90,166w  paragraph-blocks
  no-country-for-old-men 13u   69,841w  roman-numeral     [back   -463w]
  the-crossing           49u  149,985w  paragraph-blocks  [back    -30w]
  the-road               19u   58,871w  paragraph-blocks

THE THING THIS BUILDER PROTECTS IS A VOICE THAT READS AS A DEFECT. McCarthy uses no
quotation marks and drops the apostrophe from most contractions -- dont, aint, wont, didnt.
Measured over the built corpus: 0.0 quote marks per 10k words against Hemingway's 838, and
123 apostrophes against his 241. repair_typography.py normalises "toward what the text
does" and would put the quotes back, deleting the single most identifiable thing about the
author before training starts. This builder runs NO typography normalisation and then
ASSERTS the quote density, so a future well-meaning change fails the build instead of
quietly undoing it.

⚠ That same property will make the voice gate easy to pass for the wrong reason.
voice_distance.py is Burrows's Delta over character bigrams; an adapter that learns only
"emit no quotation marks" moves delta_cb a long way without having learned a sentence. A
punctuation-normalised secondary read needs pre-registering before this one is gated.

Exclusions, measured rather than assumed:
  - two truncated catalogue rows dropped for their complete mobi siblings (Blood Meridian
    epub 1,167w, The Crossing epub 222w -- both real prose, both `accepted`)
  - nothing else. All 15 cross-work 8-gram containment pairs measured on the Hemingway
    precedent; worst is 0.10%. Six independent works, no subsumption.

Back matter rides inside the last unit in four of six works and the marker differs every
time -- THE END, a dumped Table of Contents, a Reader's Guide, an About-the-Author, press
blurbs, a CIP page. It carried the author's own name 26 times across the raw masters. Both
guards report and gate: name 26 -> 0, quotes 0.0/10k.

⚠⚠ The back-matter strip runs BEFORE the split here, inverting the Hemingway order. Blood
Meridian and The Crossing end with a dumped table of contents made of bare roman numerals on
their own lines -- the exact shape of a chapter marker. Splitting first feeds the TOC to the
splitter as two dozen extra chapters; only the 150-word floor accidentally saves it today.

One lost drop cap is patched by name, not by heuristic: the All the Pretty Horses epub opens
`HE CANDLEFLAME` because the decorative T was an image the extractor dropped. A general
restore-the-missing-initial rule would have to guess the letter, so this is asserted against
the known string and fails loudly if the master ever changes.

The alphabet is re-derived, not inherited: 1,411 non-ASCII letters across 14 forms
(á é í ñ ó ú ü). The Border Trilogy is half set in Mexico, so the Yarros ASCII-only
conclusion does not transfer -- same finding as Hemingway, same reason.
2026-09-17 08:17:57 -07:00
vh 705fa3a65b split_units: choose a unit mode by SIZE, not by count, and fall back to paragraph blocks
McCarthy and Krakauer both need this before a corpus can be built, so it is a shared module
rather than a third copy of the Hemingway splitter.

THE INHERITED RULE IS "MOST UNITS ABOVE A FLOOR" AND IT BREAKS ON PART MARKERS. Measured:

  Cities of the Plain   4 roman marks -> 4 units, median 22,312w   <- the book's PARTS
  The Crossing          4 roman marks -> 4 units, median 37,310w   <- same

"Most units" scores 4 over the 1 that finding-nothing gives, so it wins, and the existing
guard only fires at exactly one unit. A 37,000-word "chapter" sails through and every
downstream tool accepts it. Size is now the eligibility test: a mode qualifies only if its
median unit is inside [600, 12000] AND no single unit holds half the work.

TWO THINGS A CONTROL RUN CAUGHT, BOTH NOW FIXED IN THE RULE. The first version scored
eligible modes by "median closest to target". Run over Hemingway, whose markers are known
good, it chose caps-title over the book's own chapters on True at First Light:

  bare-numeral  20 units  median 5,337w  max  11,155   <- the real chapters
  caps-title     6 units  median   777w  max 113,886   <- median looked BETTER

caps-title matched five stray all-caps lines, so five tiny units sat beside one holding 97%
of the book. A median cannot see that distribution; a max bound can. And caps-title is the
weakest of the four signals, which is why the tiebreak among eligible modes is now PRIORITY
(contents > chapter-word > roman > bare-numeral > caps-title), not size.

CONTROLS, both green after the fix:
  positive  Hemingway's ten works, markers known good -> 8/10 reproduce the shipped mode and
            unit count exactly. The two differences are explained, neither is a mode error:
            short-stories used `contents`, which the harness does not supply, and The Old Man
            and the Sea was deliberately kept whole as CONTINUOUS.
  negative  40,000 words with no blank lines -> 1 unit. It refuses to fabricate divisions
            out of unstructured text rather than returning a plausible section count.

Result on the two new authors: McCarthy 167 units / 587,233 words, Krakauer 135 / 431,938,
both median ~3,200-3,500w against Hemingway's 3,128.

⚠ CORRECTION TO AN EARLIER SURVEY. I reported that all four Krakauer works carry zero
chapter markers. That was wrong and it was my regex, not the books: the survey pattern
required "Chapter" followed by a numeral, and Krakauer writes "CHAPTER ONE". Missoula and
Where Men Win Glory split on chapter-word (33 and 41 units); Into the Wild and Under the
Banner of Heaven on caps-title (28 and 33). Only McCarthy's All the Pretty Horses, Cities of
the Plain, The Crossing and The Road actually need the fallback.

The Hemingway builder is deliberately NOT repointed at this module. Its corpus is shipped and
its provenance sha is pinned by a live adapter; the one behavioural difference (The Old Man
and the Sea would section into 9 rather than stay whole) is an improvement nobody asked for
on a corpus nobody should churn.
2026-09-17 08:12:09 -07:00
vh 9f35c8d659 booth: four arms, one beat, one author-neutral prompt
Six beats through voices-base, lv-bronte, lv-yarros and lv-hemingway, all served from the
same process on fv-ml1 :8027 so only the adapter varies. Operator-requested side-by-side.
http://10.100.10.50:8090/b/lv-voices-four-arms/ (24h TTL; also on the link board)

THE PROMPT NAMES NO AUTHOR, deliberately. Each adapter trained under a prompt naming its
own, so driving all four with any one of those hands that arm a hint the others do not get
and the page would be measuring the prompt rather than the voice. The shared task skeleton
is kept and the author clause removed. One asymmetry is disclosed on the page: Brontë and
Hemingway trained on "a SHORT PASSAGE ... may run to several paragraphs" while Yarros
trained on "ONE paragraph", so the neutral prompt sits slightly off-distribution for all
three rather than for one.

THE CONTROL GETS A 4x LARGER TOKEN BUDGET, and publishing it any other way would have been
dishonest. Measured at the gate's 320-token budget:

  voices-base   median 26 prose words, 181-257 words of <think> planning first,
                and 5 of 12 cells never reach the prose at all
  the adapters  0 of 12 failures each, empty think block in 12 of 12, median 97-105 words

The adapters learned to skip the reasoning phase; the carrier has not. Showing the starved
control would conflate voice with budget discipline, so the control runs at 1200 tokens and
finishes every time, median 121 words. Both numbers are on the page.

Two seeds per cell behind a toggle, because one sample of a sampled process is an anecdote,
and a blind-mode toggle that hides which column is which. Sampler matches the gate harness
(temperature 0.9, top_p 0.95, "BEAT: " prefix).

Checked before publishing rather than after: all 36 adapter generations scored for verbatim
8-gram reuse, each arm against ITS OWN corpus. Brontë 0, Yarros 0, Hemingway 2 of 12 with a
longest run of 8 words, that run being "i don t know i don t know".

Layout verified by rendering it, not by reading the CSS: four equal 374px columns at 1600px
wide, no horizontal overflow, 24 cards, 48 panes. ⚠ nh3-dev's shared /opt/ms-playwright
tops out at chromium-1234, so playwright must be pinned to 1.61.0; a bare `npm i playwright`
pulls 1.63 and asks for a browser build that is not there.
2026-09-17 07:53:08 -07:00
vh 300ecc1276 voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.

Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.

  A. VOICE   PASS 6.4x   +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
                         all-arms floor (0.113) too, so this verdict does not lean on the
                         rule change. Closes 73.8% of the span between the unadapted
                         carrier and held-out Hemingway itself; lv-bronte closed 48%.
  B. NOT COPIED  see below
  C. NO DAMAGE   PASS    ran-on +0.08, on-beat -0.14, both inside a 0.217 floor

AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.

  HELD-OUT HEMINGWAY (never trained)   370 chunks   0.01 hit-rate   mean-longest 0.1   max 10
  base-unadapted                       240 gens     0.00                        0.0        0
  ckpt850 (shipped)                    240 gens     0.07                        0.6        9
  positive control (train vs train)                                             160

The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.

The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.

SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.

servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
2026-09-17 03:36:29 -07:00
vh 5e6611466c audit_pairs_sourcenames: --filter-out, so the detector is also the fix
An already-built pair set cannot be repaired by build_sft_pairs.py --source-entities;
that flag only works at generation time. Hemingway's and Yarros's sets both predate it.

The contamination is in the BEAT, so dropping the row removes it outright. Measured on
the Hemingway train pairs: 7,094 -> 7,024, 70 dropped, 0.99% of the training data. That
is cheaper and cleaner than regenerating 70 beats against a second generator session,
which would leave the set mixed-provenance for the sake of 1% more data.

Verified by read-back rather than by the write succeeding: re-auditing the filtered file
reports 0 of 7,024 on both columns, controls green, GATE PASS.

Two refusals rather than a best-effort write:
  - a contaminated RESPONSE column aborts. That is a different fault -- pairs built
    against an unrenamed corpus -- and dropping rows would hide it instead of fixing it.
  - more than one --pairs input aborts, because the output is a single file and would
    silently merge train and val into one.

Also cross-validated the detector against the lv-bronte pair sets on real data, where the
answer is already on the record:
  pairs-full + pairs-val (post-fix)   0 of 3,858   matches the recorded "0 leaks across
                                                   3,858 pairs" exactly
  pairs-full.CONTAMINATED             15 of 792 = 1.89%, Rochester x6, Jane, Brocklehurst
                                      x2, Beck, Fairfax, Burns, Helen, Eyre -- against a
                                      record of "13 of the first 714 beats (1.8%)" with
                                      the same names
An independently written instrument reproducing a documented finding at the right
magnitude, on the right names, is the control that says its zeroes mean absent and not
blind.
2026-09-17 02:37:35 -07:00
vh 2e9b118e70 lv-bronte: the voice axis passes under the corrected floor rule — amended, not rewritten
lv-bronte shipped 2026-09-17 with a FAILED voice axis written into its compose comment,
its NFS README and its gate record. That verdict no longer stands, and this records the
correction in all three places without deleting what they said.

The floor rule is now pairwise (commit 0bb4938, pre-registered for lv-hemingway before
any Hemingway number existed). Re-scoring the SAME 360 generations, no re-run, no changed
delta_cb:

  ckpt475 (shipped)      +0.193  vs pairwise floor 0.091  -> PASS, 2.1x
  ckpt925 (not shipped)  +0.210  vs its own spread 0.251  -> still fails

As run, the floor was 0.251 for every candidate, contributed entirely by ckpt925's single
outlier seed — a candidate nobody was shipping failed the one that was.

Why this is not a threshold chosen to produce a verdict: the previous session found the
defect, wrote it into this very file, and deliberately declined to act on it. The rule was
changed prospectively on an argument independent of the answer — the sampling variability
of a difference A-B depends on A and B, not on a third arm C. voice_distance.py now prints
both floors and flags disagreement so neither can be quoted without the other.

What changes for a reader: the sensitivity floor is 0.091 rather than 0.251, and "do not
cite lv-bronte as evidence pair-SFT works for this author" is withdrawn. What does not
change: NOT-COPIED and NO-DAMAGE as recorded, ckpt475 over ckpt925 for the same reasons,
and the two-epoch recipe still not transferring to Brontë.

Amendments are append-only in all three artifacts. The on-host compose is unchanged so far
— this edit is comment-only and will ride with the next real deploy rather than triggering
a model reload for a comment.
2026-09-17 02:27:52 -07:00
vh dcc1abc7ea orientation: override the gitea NAME on the host, not each repo's remote
nh3-dev was reaching gitea over the public route from every repo on the box.
brokkr-smithy-dev flagged it while pushing a new repo: brokkr-smithy, sleipnir,
Galdrabok and kvasir all carried git@gitea.phasefinal.com remotes, and
brokkr-smithy is pushed several times a week, so the fail2ban trigger this doc
already warned about was live and recurring rather than dormant.

Measured before changing anything, because the plausible explanation was a
split-horizon rewrite making the public name internally correct:

  getent hosts gitea.phasefinal.com  -> 38.120.12.44 (public, ana-srv1)
  grep -i gitea ~/.ssh/config        -> nothing
  ssh -G git@gitea.phasefinal.com    -> hostname gitea.phasefinal.com, port 22

No rewrite, no alias, no per-repo exception. A `Host gitea.phasefinal.com` block
pointing at 10.250.50.70:222 now covers every repo on the box at once, which beats
rewriting N remotes: it also catches repos nobody audited and fresh clones that
copy the public URL out of a README, and nothing has to be remembered next time.

Verified as a route change and not just a config edit: both paths already
authenticated as `vh` with the same key, and `git ls-remote origin HEAD` succeeds
over the alias in brokkr-smithy and in this repo. Backup at
~/.ssh/config.bak-20260917-020929. The alias is per-host; the doc now says to
check `ssh -G` rather than assume another host inherits it.
2026-09-17 02:10:20 -07:00
vh 051b99e063 audit_entity_map: the rename can damage the prose and no gate will ever say so
audit_stoplist.py finds surfaces wrongly held OUT of the entity map -- a stoplisted
character is an undetectable leak. This is the mirror: surfaces wrongly held IN it.
leak_gate.py only ever asks whether the author's names are GONE, never whether
non-names were spared, so renaming `the Chinese` into an invented surname passes it
perfectly.

Found sideways on Hemingway. The pairs audit reported beats naming African, Chinese,
X-ray, Republican and Cezanne as leaks -- correctly, those surfaces really were removed
from the corpus. Reading why turned up the larger defect: they should never have been
renameable in the first place.

Measured on the Hemingway map, both controls green:
  positive  `other` 764/1356 article-preceded = 0.56
  negative  100 honorific-confirmed people, highest Inglés at 0.26, bulk 0.00-0.06
  FLAGGED   130 of 946 surfaces, 1,616 instances = 0.162% of corpus words

The signal is an article in front of the surface: you write `the Frenchman` and `a
Martini`, never `the Rinaldi`. It is a heuristic and every hit is reported FOR READING,
never auto-removed -- `the Widow` and `the Informer` are genuine Hemingway epithet-names
that SHOULD be renamed, and the band's own top entry makes the point, since Inglés at
0.26 is an in-world nickname deliberately kept renameable and sits just under the bar.

Initials are excluded from the negative-control band rather than admitted to it. `Mr. P.`
is an initial, not a person, so letting it in lets a map defect poison the control that
validates the detector -- on Hemingway `P` (0.32, every occurrence `the P. O. U. M.`) was
the one surface failing a band whose next highest was 0.26. Initials take no article and
are invisible to the scan anyway, so every surface of two characters or fewer is now
listed unconditionally. Sixteen of them are in this map, C at 274 occurrences; the same
class as the `G` that was caught by hand about to be renamed to a surname 248 times.

The unresolved count that drives the exit code is computed over every flagged surface,
not the --show slice. Tying a gate's verdict to a display flag is the same defect as a
log filter that turns a real event into a clean zero.

Also corrects a wrong claim in audit_pairs_sourcenames.py's docstring: the Hemingway
rename did not HOLD 591 surfaces. Paris, Madrid and Spain survive because the stoplist
keeps them out of the entity map before it is built, so the map is exactly the removed
set -- 941 surfaces, 941 removed, 0 kept. Measured per run rather than assumed, because
a pipeline that carried kept surfaces into the map would report every `Paris` as a leak.
2026-09-17 02:01:01 -07:00
vh 0bb4938518 lv-hemingway: pre-register the v2 gate, and fix the floor rule that decided lv-bronte
The gate design is written before any generation exists, because lv-bronte's
verdict turned on a choice that was only visible after the numbers printed.

THE FLOOR RULE IS NOW PAIRWISE. lv-bronte computed the noise floor as the largest
within-arm seed spread across ALL arms present. Its ckpt475 shipped at +0.193
against a 0.251 floor set entirely by ckpt925 -- a third arm nobody was shipping,
on one outlier seed. Scored against the arm it was actually compared to, the floor
is 0.092 and the same gap clears at 2.1x. A candidate's verdict must not depend on
which other arms happened to be generated. voice_distance.py now prints both floors
and flags any disagreement, so the lv-bronte record stays comparable.

audit_pairs_sourcenames.py closes the blind spot leak_gate.py has by construction:
it reads the corpus and the renamed copies, never the generated beats, so it cannot
see a beat-writing model restoring the author's real character names. Run over the
Hemingway pairs, which predate build_sft_pairs.py --source-entities:

  val    0 of 200     -- the eval fixture is clean, the gate is unconfounded
  train  70 of 7,094 (0.96%)  -- Santiago x16, Catherine x7, Rinaldi x3, Brett,
                                 Harry, Jake, Pablo, Nick, Maria ...
  responses 0 of 7,294        -- the lv-bronte beat-only signature exactly

A matched surface is only counted when the rename actually removed it, verified
against the renamed copies, so a beat naming a held real-world place is not a leak.
Controls run every time: 941/941 surfaces found in the unrenamed source, nonce
absent from both trees, and 6 planted canonical names detected 6/6.

voice_distance.py --author is now REQUIRED. It was hardcoded "Yarros" and printed
"reference: held-out Yarros" over Brontë's numbers into a committed artifact. A
default would have moved the silent-wrong-label failure rather than removed it. The
stale "one seed-pair per arm / corroborates Base < Instruct" footer is replaced with
what the run actually carries.

Gate design: three arms (base-unadapted, ckpt1750, ckpt850), 60 beats, 4 seeds.
ckpt850 is present because the loss curve cannot separate it from ckpt1750 -- +0.0040
against a 0.0044 median neighbour jitter, with three checkpoints inside one jitter of
the minimum. adapter/ is excluded: +0.0762 is 17.4x the jitter and is resolved without
a gate.
2026-09-17 01:45:02 -07:00
vh c445ce9e93 memory: snapshot — lv-bronte shipped with a failed voice axis, next goal is landing lv-hemingway
In-flight rewritten for the next goal. lv-hemingway is TRAINED and nothing else
has been done to it: ship candidate is checkpoint-1750 (ep 1.97, eval 2.2783),
the end-of-run adapter is 0.0763 worse, and the v2 gate has not been run. Every
instrument it needs was parameterised during the lv-bronte run tonight and the
in-flight section names all four with their traps.

New detail files:
  2026-09-17-lv-bronte-gate.md            shipped, voice axis failed, why anyway
  2026-09-17-beat-contamination-leak.md   the leak the corpus gate cannot see
  2026-09-17-esh-fiber-outages.md         two Cityside failures, rotation fragility

Also commits the memorization_check.py parameterisation, which was left
uncommitted: its hardcoded Yarros defaults would have compared a Hemingway arm
against the Yarros corpus and reported a meaningless clean zero.

Auto-archival: index was 415 lines pre-run, over the 300 cap. Only five entries
cleared the 14-day age guard, and three of those carry open deferred pointers
(fused MoE park 47, nconnect=8, AI-tab belayed) and are referenced by in-flight.
A fourth — every CI job on pfi-fleet runs as root on ana-docker — is a live
security property rather than settled history, so it is held back deliberately.
One entry archived. The file stays over cap, which is the guard working: an
over-cap file that keeps live decisions beats a scannable one that lost them.
2026-09-17 01:29:09 -07:00
vh 61840f3131 voices-seat: ship lv-bronte (ckpt475) with its failed voice axis on the record
lv-bronte is live on vllm-voices (fv-ml1 GPU0 :8027) alongside voices-base and
lv-yarros. The seat lists all three; container healthy; GPU0 96092 -> 96090 MiB,
so the adapter cost nothing measurable.

IT DID NOT PASS ITS VOICE GATE, and the artifact says so in three places — this
commit, a comment in the compose file, and a README beside the adapter on NFS —
because an adapter found without its provenance will otherwise be read as a pass.

  VOICE       FAIL  +0.193 delta_cb vs base, against a 0.251 measured noise floor
  NOT COPIED  PASS  8-gram hit-rate 0.00, longest 0 - identical to the control
  NO DAMAGE   PASS  ran-on +0.15 against a 0.400 floor

Shipped on three grounds, none of them that the number was nearly good enough:
it is additive (a named LoRA nobody reaches without asking for it), reversible
(one compose line; hot-unload measures 0.003 s), and clean on the axis that
carries actual risk - verbatim regurgitation of the source, on a public-domain
corpus, measured against a positive control that saturates at 160.

The voice result is UNDERPOWERED rather than absent: it closed 48% of the span
from base to the same-author target and beat the control on every individual
seed. The cause is structural - 81 val pairs against Hemingway's 200, from a
678k-word corpus against 994k - and neither more beats nor more seeds fixes it,
because the floor is a range statistic and ranges widen with n.

ckpt475 over ckpt925: indistinguishable on voice (0.017 apart), but ckpt925 has a
verbatim 8-gram hit where this has none, and is 2.7x less stable seed-to-seed
(0.251 vs 0.092) with a degeneracy probe showing no collapse to explain it.
2026-09-17 01:22:03 -07:00
vh 9b360e477d memory: lv-bronte gated — voice axis fails, ship decision open
Records the full v2 gate result and three findings that outlive the ship call:

1. The effect is UNDERPOWERED, not absent. Both candidates closed 48-52% of the
   achievable span to held-out Bronte and beat base on every individual seed, but
   the gaps sit under the measured floor. Sensitivity floor stated so the negative
   is falsifiable: cannot resolve better than ~0.251 delta_cb at 30 beats x 4 seeds.
   Cause is structural — 81 val pairs against Hemingway's 200, from a 678k-word
   corpus against 994k — and neither more beats nor more seeds fixes it.

2. A DEFECT IN THE v2 RULE. The floor is the largest within-arm spread across ALL
   arms, so adding a third noisier arm raised the bar that failed the clean one.
   Run as a two-arm gate the floor would have been 0.092 and the candidate would
   have cleared at 2.1x. Deliberately NOT exploited — choosing the floor that
   passes your preferred answer is the failure pre-registration exists to prevent —
   but the rule should state whether the floor spans the compared pair or every arm
   present. As written, a verdict depends on which other arms you happened to run.

3. The two-epochs-on-a-three-epoch-schedule recipe did NOT transfer. Bronte's two
   minima are 0.0022 apart against a 0.0046 jitter; epoch 2 buys nothing over epoch
   1. The epoch-3 collapse (+0.075, ~16x jitter) is the only robust part.

The outlier seed was diagnosed rather than waved away: a repeat-5gram degeneracy
probe is uniform at 0.0078-0.0102 across every seed and both arms, so it is genuine
delta_cb variance and the floor stands.
2026-09-17 01:15:29 -07:00
vh 8bb7686a16 audit_stoplist: a stoplist entry is an assertion the leak gate cannot check
Stoplisting a surface removes it from the entity map, so rename never touches it
and the gate never scans for it. That is exactly what a stoplist is FOR when the
surface is a real-world referent — and exactly how a wrongly stoplisted CHARACTER
becomes an undetectable leak. The gate reports 0 of N surviving and is telling the
truth about the set it was given.

Found by luck on lv-bronte: a generated beat said "Mrs. Leaven", and Leaven had
been filed under scripture as the bread noun. Reading it back: "Robert Leaven,
the coachman" — Bessie's married surname in Jane Eyre.

Running the audit instead of trusting that luck caught two more:
  Pierrot  "Madame Pierrot: she comes from Lisle, in France" — a teacher in
           The Professor, filed as the commedia dell'arte figure
  Samuel   "Mr. Samuel Wynne" — filed as scripture
and correctly CLEARED two:
  Wellington  "that Baal of a Lord Wellington" — the real Duke
  Moses       "the Rev. Moses Barraclough" — the documented dual-use

Signal is an honorific in front of the surface: real-world referents are not
addressed as Mr/Mrs/Miss/Madame/Lord. It is a heuristic and not a proof, which is
why every hit is REPORTED FOR READING and never auto-removed — Wellington and
Moses both trip it and both are correct. Exit 1 on anything not on --allow, so it
can gate a pipeline.

Blast radius of the three errors was 16 of 3781 train pairs and 3 of 80 val —
small, but they are the author's characters in training data, which is the one
thing this pipeline exists to prevent. Corpus rebuilt rather than dropping the
affected pairs: a corpus on disk that disagrees with its committed config is how
superseded claims get made. Gate re-passes at 0 of 368 (three more surfaces than
before, exactly the restored characters), both controls green.
2026-09-16 21:40:34 -07:00
vh e9e8c40b83 eval harness: sample the beat fixture from held-out val, and bind the eval prompt to the trained one
Two harness defects that would each make a voice number uninterpretable.

build_beat_fixture.py — the fixture is now SAMPLED from the val split rather than
hand-written. The original BabyYarros fixture was five hand-written beats about a
stray dog and a kitten: wrong genre, so 'He licked her clean' came back as
explicit sex from a romantasy adapter, and n=5 had a noise floor of 0.800 that
manufactured a +0.45 result which collapsed to +0.08 at n=120. Sampling from val
makes it in-genre and held out by construction, spread across works so a naive
head(30) is not one novel. Refuses outright if the pairs carry any split but val,
because a fixture drawn from training data makes every downstream number a
memorisation measurement wearing a voice label.

gen_beats_chat_yarros.py --system-from — the SYS constant in this harness is
Yarros's. Driving a Bronte or Hemingway adapter with it measures the arm under a
system prompt it was never trained on and confounds the carrier change with a
prompt change. Rather than duplicate the register table and rely on whoever runs
it to pick the matching one, read the prompt out of the pair build's own
provenance, which is the artefact that records what the adapter actually saw.
2026-09-16 21:18:44 -07:00
vh 7964d077de bronte-corpus: runbook — the five deviations and what the controls caught
Records the reproducible chain and, more usefully, why it diverges from the
Yarros/Hemingway pipeline in five places, each forced by a measurement rather
than a preference.

Includes the control post-mortem, which is worth keeping because in three of
four cases the CONTROL was wrong and the detector was right — the opposite of
the reflex. Adele vs Adele-with-a-grave, Hollow at a 0.235 lowercase ratio, and
Grace at 0.224 were all correct refusals. Blanche, at 0.0526 against a 0.05 bar,
was the one real detector miss.
2026-09-16 21:03:55 -07:00
vh 533cc0ce81 build_sft_pairs: reject beats that name characters the rename removed
A leak the corpus gate structurally cannot see, found on lv-bronte.

The rename strips the author's names from the prose and leak_gate.py proves
they are gone — 0 of 365 surviving on Brontë, both controls green. But the beat
is written by an LLM that READ THE PASSAGE, and if it recognises the book it
supplies the canonical names out of its own training. The beat is the
INSTRUCTION half of the pair, so training on it re-teaches exactly the
inventions the rename pipeline exists to remove, and the gate never looks at it:
the gate reads the corpus and the renamed copies, never the generated beats.

MEASURED on the first 714 Brontë pairs, before the filter existed:
  13 beats (1.8%) named source characters — Rochester x6, Jane x3,
  Brocklehurst x2, Beck, Fairfax, Helen, Burns, Eyre, Reed, Rivers
  0 of 714 RESPONSES did. The rename was perfect; the instruction side was not.
One beat read "Saoirse confirms Rochester's flaws, then agrees in English to
marry him" — a renamed name and a canonical one in the same sentence, which is
the mechanism in miniature.

Exposure scales with how well the generator knows the book, so it is WORST for
public-domain classics and mildest for recent work. That is exactly why the
Yarros and Hemingway runs came up clean and Brontë did not — their clean runs
are NOT evidence this cannot happen to them, and both should be rebuilt with
--source-entities if they are ever regenerated.

Adds a `sourcename` reject to vet() plus --source-entities, which takes the
UNRENAMED entity map and refuses any beat naming a surface from it. Firing at
roughly 3% of attempts on Brontë.

Also adds a `bronte` register. Brontë is the far end of the same axis from
Hemingway and the register has to say so, or the beat-writer produces modern
summary prose the passages never match.
2026-09-16 21:02:49 -07:00
vh fc834a8a23 bronte-corpus: gate lv-bronte for real — 0 of 365 with both controls green
The Brontë corpus's "0 of 203" was a HAND COUNT made before leak_gate.py
existed. On Yarros the automated instrument read 212 surviving where a hand
count said 86, so the hand count was never evidence. This runs the real gate,
and getting it to pass required fixing four defects the hand count could not
have seen.

CORPUS DEFECTS (repair_corpus_bronte.py, both measured):
  - 1,922 words of publisher back matter inside Shirley's last unit — a
    T. Nelson & Sons catalogue advertising Ainsworth, Marryat, Verne, Kingsley
    and Dickens, plus a Gutenberg transcriber's punctuation list. Not Brontë,
    and the source of the entity CHARLES. Same structural cause as the
    Hemingway run: a splitter cuts on headings, nothing follows the final one.
  - 1,368 Gutenberg italic spans. Two harms: they teach the adapter to emit
    underscores, and the underscore is a word character, so the gate's
    word-boundary scan cannot match inside an italicised name. An entity in
    italics is invisible to the gate — the same never-renamed-AND-never-
    reported shape as Yarros's possessive-only Afendra.

DETECTOR GAPS (phrase_map_bronte.json):
  - Blanche is 19 capitalised against ONE lowercase — ratio 0.0526, over the
    0.05 bar by a single token, so a named character with 19 mentions is
    dropped by a hair.
  - Grace (0.224) and Hollow (0.235) are refused correctly — both are common
    nouns — but Grace Poole and Hollow's Mill are Brontë's. Sampling all 21
    bare capitalised Grace found 20 are the character in direct address and
    exactly one is the theological noun.
  - Five compounds whose every component is non-renameable survive verbatim:
    Moor House, Marsh End, Vale Hall, Bigben Close, Royd Lane. The other 77
    audited phrases do not, because each has a renameable component.

GENDER (pin_known_gender.py): the inherited resolver put Jane MALE across 336
occurrences. Hemingway's base-rate resolver is strictly better here (1 wrong vs
4) but still fails on Jane, and the failure is structural, not tuning — Brontë's
three narrators are first-person, so their names appear almost only in dialogue
surrounded by other characters' pronouns. Ground truth is pinned separately from
the resolver's evaluation so the two are never conflated.

Also: min-cap lowered 8 to 3, which pulled Bertha, Ferndean, Rochesters and
Creemsvort in from below the old floor; corpus-scope rename so a name below
threshold in one novel is not printed verbatim there while renamed in another.

Gate: 0 of 365 surviving, positive control 365/365, negative control clean,
phrase audit 0 of 82. Floor stated: 3 capitals per work, 5 recurrences.
2026-09-16 20:51:11 -07:00
vh a5745dcf72 playbooks: generic per-service stack image update (pull + recreate + verify)
Adds playbooks/update-stack-image.yaml — pull the newest image for one
compose stack service and recreate it, with a verify phase that asserts
the container's image id equals what the tag now resolves to rather than
trusting a 'Up' line from docker ps.

Scoped to a single service on purpose: the recreate is 'up -d <service>',
never a bare 'up -d', which would recreate every service in the project.

Go template format strings are written bare; elway's {{ identifier }}
substitution leaves them alone, but {{end}} / {{else}} would match and
die as undefined variables, so the health read uses {{json .State.Health}}
instead of an if/else.

First use: drawio on esh-docker-vm, 28.1.2 -> 31.4.6.
2026-09-16 16:34:16 -07:00
Vuong Hoang e8086941e2 memory: snapshot — lv-yarros shipped, voices-seat live, lv-hemingway training, Grok broker shelved 2026-09-16 16:19:58 -07:00
Vuong Hoang cf9d167453 grok-token-broker: shelved — operator kept the jail, so the renewal feature has no consumer 2026-09-16 16:04:29 -07:00
Vuong Hoang b907a0e46a grok-token-broker: the probe's blast radius is BOTH Groa transports, not one 2026-09-16 15:19:46 -07:00
Vuong Hoang ebc4dac6d8 grok-token-broker: hold a refreshable session credential behind a rotation-safety gate 2026-09-16 15:12:38 -07:00
Vuong Hoang d17bd3df86 voices-seat: one carrier, lv-<author> LoRA adapters, with the measured cost and placement limits 2026-09-16 13:51:51 -07:00
Vuong Hoang 6282833669 pairs: fix the three construction defects and the abbreviation-truncation bug 2026-09-16 08:12:01 -07:00
Vuong Hoang 03b4a3f62c BabyHemingway D2+D3: entities, base-rate gender resolver, rename preset, leak gate passes 2026-09-16 07:55:34 -07:00
Vuong Hoang 9598d0b4a7 BabyHemingway D1: fiction-only corpus builder with measured exclusions 2026-09-16 07:40:44 -07:00
Vuong Hoang 7cc847f33c memory: client abandon sometimes cancels and sometimes does not; the KV trajectory is the tell 2026-09-15 16:49:08 -07:00
Vuong Hoang 75051243ae BabyYarros: amend the decision rule to gate on voice and non-memorization (operator-authorised) 2026-09-15 14:04:42 -07:00
Vuong Hoang efb734586b BabyYarros: raw-surface scoring and a memorization check with both controls 2026-09-15 13:23:27 -07:00
Vuong Hoang 713e83dd5e BabyYarros: pre-register the pair-vs-rawtext decision rule before the arms are read 2026-09-15 11:19:11 -07:00
Vuong Hoang 90ed506db3 fix(pairs): apply_chat_template returns a BatchEncoding, and assert the mask boundary 2026-09-15 10:51:49 -07:00
Vuong Hoang 9b3d3c80cb BabyYarros Option C: instruction-pair builder and the assistant-masked pair trainer 2026-09-15 10:45:51 -07:00
Vuong Hoang d0108466e9 memory: the owed handoff datapoint is a tripwire, not a validation 2026-09-15 10:37:47 -07:00
Vuong Hoang 36f9240477 memory: the handoff leak has two surfaces, and the dangerous one fires on ordinary input 2026-09-15 10:29:21 -07:00
Vuong Hoang 0b6ea6f910 memory: the /snapshot handoff modality defect is fixed, and my two real runs corroborate it 2026-09-15 10:27:14 -07:00
vh 1ffb6d7af8 memory: report the /snapshot handoff defect to galdrabok, with the mechanism
Sent with both specimens. Adds the root cause, which is in SYSTEM_PROMPT rather
than the model.

Next steps is the only one of the three generated sections with no empty case.
Watch out for is told to omit itself when there are no gotchas and Resume here is
told what to say when nothing is in flight, but Next steps is told only that it is
a numbered, ordered, concrete list. With nothing in flight the sole action-shaped
nouns in the input are the deferred items, and the nothing-in-flight rule points
the model straight at them by asking it to name the most recent open pointer.

Nothing in the prompt protects modality. Invent nothing and trace every claim to
the input are both satisfied - the items really are in the input - while their
deferred-ness is exactly what gets dropped. The verbatim-identifier rule already
establishes that some attributes of the input must survive restructuring untouched;
modality is one of them and only identifiers are guarded.

Proposed two prompt changes to galdrabok: an empty-case escape for Next steps, and
a rule making deferred, parked, belayed and deliberately-not-done items constraints
belonging in Watch out for rather than steps. Offered as a caller's diagnosis since
the skill is theirs. Noted that galdrabok-dev is pull mode, so there is no herald
poke and they will see it on their next check.
2026-09-15 10:17:28 -07:00
vh 9d36c74572 memory: snapshot refresh — breeze settled, util does not predict residency, handoff defect
Incremental over 653f7fb, covering the three turns since.

breeze-tts stays on irv-ml1 and the TTS-stack move is parked at id 75. Adds the
full fv-ml1 per-seat residency table and the shuffle arithmetic behind the
recommendation: every arrangement that makes real room on GPU 1 spends the GPU 3
reserve anyway, so if breeze ever moves it should go straight to GPU 3 rather than
evacuating three working seats for the same cost plus a migration.

The finding worth keeping is that --gpu-memory-utilization does not predict resident
VRAM, and is wrong in both directions: cyberprev at util 0.40 holds 8 GB more than
its fraction, gen-small at 0.48 holds 10 GB less. An hour earlier in the session I
reasoned about placement from those fractions and would have been 8 GB out. Read
nvidia-smi --query-compute-apps instead.

Also records a characterized defect in this skill's own handoff generator. Across
two runs in one session it converted operator-deferred items into an imperative
next-steps list and twice invited the next session to commit files marked as
predating it. It fails in the blind spot of the documented fail-loud posture: the
output is structurally valid and exits 0 while inverting the intent. That matters
more than an ordinary bad summary because the handoff is the one artifact a fresh
context reads as instruction. Both runs were corrected in-session; not yet reported
to the skill's owner.
2026-09-15 10:08:57 -07:00
vh 7a33bd9f09 memory: breeze stays put; TTS-stack move to fv-ml1 parked at id 75
Operator ruling: leave breeze-tts on irv-ml1 and park moving it, bragi and
tts-gateway to fv-ml1 until the embedder, reranker and reward seats are evacuated.

Parked as move-the-tts-stack-breeze-tts-bragi-tts-gateway (id 75) with the trigger,
the footprints and the migration gotchas, so it resurfaces with everything needed
rather than as a bare line.

Two things worth having recorded against the trigger. All three services move as a
set because only breeze is GPU-resident at ~10.3 GiB and growing, while bragi and
tts-gateway are CPU-only proxies - co-location with the gateway is the entire reason
not to move breeze alone, since that is what puts a cross-site hop on every TTS call.

And the trigger as stated names gpu0, but vllm-embed, vllm-rerank-a3 and vllm-reward
are all pinned to GPU 1. GPU 1 is the constrained card at 0.975 committed with
4,336 MiB free, while GPU 0 has 11,982 MiB free and carries the live chat path, so
evacuating those three relieves GPU 1 rather than GPU 0. Recorded as a
confirm-before-executing rather than silently corrected, since it changes where the
TTS stack would land.

Also notes that bragi and tts-gateway reach each other by name only through
extra_hosts pins, because containers on irv-ml1 cannot resolve nh3.internal - those
pins travel with them and need re-pointing at the new host.
2026-09-15 09:31:12 -07:00
vh b0e7b408d9 memory: breeze-tts sizing and the fv-ml1 GPU 0 placement recommendation
The operator asked this mid-sweep and the answer never reached durable memory -
caught only because he asked again after the snapshot. Recommendation is not to
move it.

Re-measured rather than reciting the earlier figure, which was right when taken and
is now wrong: breeze holds 10,316 MiB after 53 minutes of uptime against 9,218 MiB
shortly after warm-up. The footprint grows with use, consistent with PyTorch's
caching allocator not returning memory - probably caching rather than a leak, but
resident either way and counting against any neighbour. Two points is a trend, not
a curve; whether it plateaus is unmeasured and stated as such.

That changes the placement answer. fv-ml1 GPU 0 has 11,982 MiB free, so the margin
is 1.7 GB and shrinking rather than the 2.8 GB the earlier number implied, on the
card carrying the live chat serving path.

The stronger objection is topology rather than VRAM: tts-gateway runs on irv-ml1
and reaches breeze on the same box, so moving breeze alone puts a cross-site hop on
every TTS call against a 478 ms to-first-sample budget. Moving it properly means
moving the gateway too. It is also not constrained where it sits - the 3090 still
has 10 GB free.

Also records the trap that nearly produced a wrong number: breeze reports nothing
at idle when queried on the wrong GPU, because BREEZE_GPU_DEVICES=0 is the 3090
rather than the A6000. An idle query of the A6000 shows it absent entirely.
2026-09-15 09:27:59 -07:00
vh 653f7fb939 memory: snapshot — Parakeet STT, svos_miranda live, talk v10, address sweep, secrets fix
Closes both of the previous session's named jobs and six unplanned pieces of work.
Nothing is in flight and nothing is blocked.

Parakeet STT live on fv-ml1 GPU 0 behind LiteLLM ext-stt and whisper-1; GPU 3 is now
a documented reserve after the operator caught an 800 MiB seat parked on the one
pristine 96 GB card. svos_miranda enabled and Miranda serving, with
agent.disabled_toolsets deleted and staying out by operator ruling. talk v10
deployed as the STT seat's first consumer. The irv-ml1 dead-address sweep is
complete at 0 of 112 Homepage cards, having turned up four live breakages on other
hosts. The secrets-broker concurrency bug is fixed, and ~/.local/bin/secret is a
symlink rather than a stale copy.

Auto-archival fired at 433 lines but moved only one entry: three of the four
candidates old enough to qualify carry open deferred-work pointers - a park id, an
althing thread, and an explicit 'untracked by operator choice' - and the guard held
them. The index stays over cap at 389 lines, which is the correct trade: nearly
every entry is genuinely under fourteen days old.

The generated handoff needed correcting in-session before it shipped. The model had
turned three operator-deferred items into a to-do list and invited the next session
to commit files that predate this one. Both would have read as instructions to a
fresh context, which is the durable-false-warning failure this session spent the day
documenting.
2026-09-15 09:26:26 -07:00
vh f4320ff57e docs(memory): the talk-deploy permission problem never existed
Vuong asked me to find and fix the harness issue blocking tts-dev from deploying
talk. There was no harness issue, and no issue of any kind.

/opt/docker/compose on nh3-dev is root:docker 2775, agent sessions run as lkraven,
and lkraven is in the docker group. A mkdir settles it in one second and nobody ran
one for nine days. There is also no tts-dev OS account, so the group request had no
referent.

It held together because a stale persistent-memory row supplied a plausible
mechanism and the operator's routing instruction - 'give it to infra' - was read as
corroboration of a capability limit. Those are different claims and only one was
ever stated: a routing preference explains where work went, never whether it could
have gone elsewhere. A contradicting ls -la was on screen in the same session and
was dropped. I then repeated the claim to the operator as fact in a deploy report,
which put a second name behind it.

Then I did the same thing one layer up. Finding no OS problem and no deny rule, I
inferred an auto-mode classifier refusal because the shape fit, and committed a
settings.json into tts-dev's repo on that inference. Their mkdir showed the path
writes with no refusal at all, so the hypothesis was wrong and the commit is
reverted. I had spent the night writing up this failure class and still built a fix
for a layer nobody had shown me failing.

That commit also claimed a doc correction it did not contain: the edit and the
commit were chained in one invocation, the edit's anchor assertion failed because
the target text had already been fixed, and the commit ran regardless. Amended
before reverting. Never chain an edit and its commit in one invocation.

The rule worth keeping is that 'I can't do X' from any source is a hypothesis until
someone runs the command and pastes the error, and that 'there is no error text,
because there was no error' is a possible answer.
2026-09-15 09:23:57 -07:00
vh af8d6df387 docs(memory): name the fleet's characteristic failure mode
svos-dev observed that three instances of the same shape turned up between two
agents in one night and that it is starting to look like a characteristic failure
rather than a coincidence. Collecting all nine from today, because the class is more
useful than any instance.

The shape is a check that reads the input to a transformation and gets reported as
if it read the output - or more generally, the instrument answering instead of the
system, in a form shaped exactly like a real answer. What makes it expensive is not
that things break but that the broken state is indistinguishable from a legitimate
one, so it passes review and is found later by accident. Every one of the nine
passed a check.

The tell is stated so it can be recognised prospectively: whenever 'broken' and
'legitimately empty, absent or off' produce the same output, the cheap check cannot
tell them apart by construction.

Remedies that actually worked today: measure the output rather than the input;
positive controls, since a method that has only ever passed cannot tell you it is
not blind; true-negative controls, because two apparent failures in the
secrets-broker test were names I had invented and would have been read as a partial
fix; refuse to emit the ambiguous value, which was the real fix rather than the
lock; and do not declare victory on a plausible fix, which is the only reason the
session-establishment root cause was found at all.
2026-09-15 08:53:06 -07:00
vh 0193b31aad fix(secrets-broker): bw is not concurrency-safe — serialise, and never return an empty secret with exit 0
Reported by svos-dev after parallelising four vault reads in SVOS's systemd
wrapper. Reproduced here and it is worse than reported: four concurrent secret get
calls for distinct items returned empty strings with exit code 0, zero of four
succeeding against their one of four. No error, no timeout, no diagnostic.

The shape is the problem, not the race. A caller treating an empty optional secret
as 'not configured' degrades silently and never learns otherwise - it cost SVOS the
ability to page the operator while the process logged a clean startup line.

Root cause is session establishment, not item reads. Every invocation runs bw
unlock, and concurrent unlocks against the shared appdata dir invalidate each
other. The damage then surfaces downstream as an empty listing or an empty item
body, which is why a per-call lock is useless: by the time the read runs the
session it holds is already dead. So the lock wraps the whole command instead.

Three changes. The command-level lock makes concurrent callers queue. cmd_get now
refuses an empty value rather than printing it, since a stored secret is never
legitimately zero-length. And find() no longer coerces empty stdout to '[]' - that
turned a broken read into a confident 'no such secret', the same silent-wrong-answer
shape one layer up.

Verified: four parallel reads of four real items now return all four correctly,
serialised at the honest ~17s each. A name that genuinely does not exist still
fails loudly, so the guard did not simply mute the negative case.

Also replaces the copy at ~/.local/bin/secret with a symlink to this file. It was a
plain copy in sync by luck, and every edit here silently left the live tool behind.
2026-09-15 08:49:11 -07:00
vh 66c860d6c1 fix(sweep): retire the dead 10.100.79.3 address across the fleet
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is
on no interface anywhere, so anything pointing at it gets no route at all. Homepage
went from 9 dead cards to 0 of 112.

The load-bearing part is that there is no single right target: it depends on who
resolves it. The operator's browser and the Homepage and open-webui containers on
esh-docker-vm all resolve nh3.internal, so those get the name and survive the next
renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those
get the IP.

litellm on ana-docker looked like a counterexample and is not: it resolves the name
only through its own extra_hosts entry, while asset-engine on the same host fails on
it. Test from the container you are about to change, never from a neighbour. Before
committing to the name I confirmed the Homepage container actually fetches ytvc's
healthz through it in production rather than assuming resolution implies reach.

On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history.
Seven running containers recreated so the labels took. Seven dormant ones carried
stale labels because editing a compose file does not touch an existing container
object - fixed with compose create --force-recreate, which rebuilds the container
without starting it, the right tool for a deliberately dormant stack.

The sweep's real find was off irv-ml1 entirely: four live values on two other hosts,
silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS,
asset-engine's inference host, and two skaldsong TTS URLs. Both running services were
recreated and verified reaching their targets afterwards rather than merely carrying
the new string.

One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic
label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I
caught it only because I had taken a baseline before touching it. A label-only edit
still costs a full model reload on a GPU container.
2026-09-15 08:41:44 -07:00
vh 8bc46e5132 docs(memory): record the gap tts-dev found in my served-page gate
They adopted the gate as tts-stack tools/gate_served_page.py and extended it in a
place that matters: my version would have passed a broken page.

A worklet lives inside a template literal, so a syntax error in it is invisible to
a parse of the enclosing script - it is just a string until addModule compiles it
at runtime, where it fails as a rejected promise and the page quietly falls back to
buffered playback or records nothing. Silent degradation, which is harder to notice
than a dead page rather than easier. They parse the worklet separately, and they
positive-controlled the whole thing against two deliberately broken pages rather
than assuming a gate that has only ever passed is not blind. The second control -
valid enclosing script, broken worklet - is the one my version fails.

The lesson on my own work is the useful part: I built a gate for the failure I had
just been shown and stopped at its boundary. The class is 'code that is a string at
parse time and code at run time'; an inline script is one instance and a
template-literal worklet is another. I checked the instance, not the class.

Also promotes the underlying rule to the index, since it was named twice tonight
from two unrelated directions: a check that reads an artifact as stored cannot see
a transformation that happens between storage and execution.
2026-09-15 08:29:25 -07:00
vh e113660b08 docs(memory): talk v10 deployed with a served-artifact gate
Operator-instructed via tts-dev. First consumer of the ext-stt seat stood up the
same night - talk can now listen as well as speak. Relayed authorization was fine
to act on because the work is reversible: one-line tag rollback, v1..v9 retained,
compose and .env backed up. Checked the escape hatch existed rather than believing
the message that described it.

Gated properly: build, throwaway on a non-live port, four acceptance checks, tear
down, then cut over in a separate invocation. Re-ran all four against production
afterwards, because a gate that only ever ran against the throwaway proves the
image rather than the deployment, and confirmed the two new env vars inside the
running container rather than in the file.

Added a fifth gate worth keeping. tts-dev's worst bug this cycle was a JS escape
inside a Python string arriving transformed, closing the string and killing the
entire inline script while the page still rendered and both import and node --check
passed - because the file still held the backslash. So: fetch the page over HTTP,
extract the inline script from the response body, and node --check that. Same
instrument pointed at the other side of the transformation, and over the wire it
also catches anything that mangles the body after TLS and ASGI.

That is the second instance tonight of one rule: a check that reads the artifact as
stored cannot see a transformation between storage and execution. provider=cuda in
a log is the same error - an echo of configured intent read as a measurement of
running reality.

Also notes an open question for the operator: talk deploys route through infra-ops
only because tts-dev's identity is not in nh3-dev's docker group. The durable fix is
a group membership, not a standing relay.
2026-09-15 08:27:00 -07:00
vh 01f0014489 docs(memory): SVOS/Miranda fully live; bank two restart patterns from svos-dev
svos-dev restarted :8770 at 02:17 and both roster lines printed clean. Confirmed
from this side rather than taken on their word: :8770 answers 200 on the new pid,
an unauthenticated Bifrost dispatch gets 401, and Hermes reports 29 toolsets with
svos_miranda the sole enabled=True row.

Two patterns from their restart that generalise past this service.

A dry-run boot against the still-held port: start the new process while the old
one still owns the socket, and it proves every check above the bind before dying
on EADDRINUSE. Zero downtime, no commitment, and it turns a one-way restart into a
rehearsed one. Worth doing for any service whose startup validates before binding.

And a trap: SIGTERM released the port but left the process alive for 35 seconds,
needing SIGKILL. The port was free that entire time, so a script waiting on port
availability would have started the replacement alongside a still-running old
process. Kill by PID and wait on the PID, never on the port - a freed port is not
evidence of a dead process, the same way an unreachable post office is an outage
rather than an empty inbox.
2026-09-15 02:20:10 -07:00
vh 406769e64b docs(memory): bank the Parakeet bench result and tts-dev's storage-vs-execution lesson
The IRV seat was retired on tts-dev's numbers: it lost to the FV seat at both
clip lengths and to whisper-large-v3 at 6.24s. Their length sweep fits ~58ms
fixed + 56ms per audio-second with an asymptote of ~17.8x realtime, which
independently reproduces our 17x on a different clip and harness, and the gateway
hop measured below their harness resolution so ext-stt is the right consumer path.

Two caveats recorded against our own numbers: their between-run variance is 20%
because GPU 0 carries the live chat path, and our 0.50s median came off an idle
GPU 3 - a best case, not a comparable.

Their RTFx retraction is the durable part: published RTFx is batched throughput on
datacenter hardware rather than single-stream latency, and the two differ by ~200x.

Also banks the shape their acceptance gate caught, because it generalises past
their repo. A JS escape inside a Python string arrives transformed, closing the
string and killing the whole inline script, while the page still renders and both
import and node --check pass - the file still holds the backslash. That is the
same failure as reading provider=cuda out of a log: a check that reads the
artifact as stored cannot see a transformation that happens between storage and
execution. Both check the input to a transformation and get reported as if they
checked its output.
2026-09-15 02:17:57 -07:00