Trained the same corpus onto Qwen3-4B instruct rather than -Base, with seed, steps
and token count held so the carrier is the only variable. The chain picked
checkpoint-150 by loss automatically, applying the lesson that cost a re-cut on
4B-Base.
The central risk did not materialise. The assistant prior did not block the voice:
curly quotes land at 16 of 18, identical to the 4B-Base tuned arm, against 1 of 18
on the unadapted control, and task-leak is 0 of 18 where the base carrier leaked 4.
Instruction-following also survived raw-text training -- 10 of 10 on-beat through
the chat template, the same as the untuned control.
The cost is length discipline rather than comprehension. In-band dropped from 10 of
10 to 6 of 10 and the median went from 124 to 140 words. Training on Victorian
prose made it wordier, which is a soft degradation and not a break.
Held-out sits at 2.908 against 4B-Base's 2.814, and it plateaus without turning
where the base carrier overfit at step 75. The assistant prior competes for
capacity, so the instruct carrier absorbs less rather than overfitting more.
What raw-continuation training does not fix is the plot furniture. The tuned
instruct arm renders the beat and then drags the referent -- "He licked her clean...
my master thus, my husband thus", turning the dog into a man, because the corpus is
about masters and husbands. Another beat ran to 247 words and gave the narrator a
list of duties. That is precisely what instruction-pair training addresses, since
pairs teach render-this-and-stop where continuation teaches keep-writing. The probe
de-risks the instruction-pair path without substituting for it.
One metric note against future misreading: ran_on reports 10 of 10 on both arms and
is uninformative on this job, because a single paragraph contains no blank line for
it to find.
Skaldsong's requirement is beat-sentence in, paragraph out, stitchable. That is
narrower than free-form continuation and it fails differently, so it was tested
directly rather than assumed from the earlier prompt work.
The adapted 4B carrier cannot do it. Ten prompt formats across three seeds each --
bare, paragraph-break, labelled, epigraph, one-shot, one-shot unlabelled,
three-shot, "it happened in this way", "I remember the whole of it", and the
letter prompt's label-and-begin move applied to a beat -- produced thirty samples
and not one that reliably renders the beat. They drift, they frame, or they
truncate. The cause is structural rather than a shortfall of the adapter: "write a
paragraph about this sentence" is an instruction, and a completion model has no
mechanism for "about". It continues the text it is handed.
Two formats failed worse than drifting and leaked pretraining task data. The
paragraph-break format emitted an NLI multiple-choice item, and label-begin emitted
a grammar-correction exercise. A standalone sentence followed by a blank line looks
exactly like a dataset entry, and style adaptation does not remove base-model task
artifacts.
The same five beats through the gen seat with a style instruction and no adapter
landed all ten samples inside the requested 90 to 140 word band, every one on its
beat, with no drift into a following scene. The voice, however, is generic literary
pastiche -- abstract-noun-heavy, fond of aphoristic openers, and over-written --
rather than Brontë.
Voice without direction against direction without voice. The product needs both,
which makes rebuilding the corpus as instruction-response pairs on an instruct
carrier the evidence-backed option rather than more prompt cleverness. This applies
to Yarros identically, because the carrier question is orthogonal to the author, so
the next corpus should not re-run this experiment.
Stitching also has its own failure mode, shown in the booth's third panel: the
paragraphs were generated independently and the point of view slides between them,
so by the fourth beat the narrator is both watching the girl carry the animals and
carrying them herself. A real stitcher has to feed prior paragraphs back as
context, which means the instruction-pair corpus wants multi-paragraph continuity
examples and not only isolated beat-to-paragraph pairs.
Two operator prompts settled the question the handoff prompt opened.
The winner labels the artifact and then begins it: "This is the letter I wrote
verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog
meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never
believe what I saw--". Two of three seeds delivered the actual event in first
person, and one is the strongest output of the sweep -- the dog following her to
the mill, the kitten caught in a hedge, the dog carrying it out and laying it in
her lap before licking it. Coherent arc, correct subject, no gloom-override, no
meta-frame. It also reproduced the Gutenberg underscore-italics convention.
It works where the handoff failed for a specific reason. The handoff could be
satisfied by narrating compliance, because the letter did not yet exist. Naming the
artifact and starting it leaves nothing to narrate around, so the only continuation
is the artifact.
The chapter-heading prompt refuted a hypothesis of mine. I predicted that rendering
the title in the corpus's own conventions would make it land harder than the
operator's inline form. It did the opposite: both corpus-form seeds ignored the
title and opened unrelated scenes, while the inline form at least finished the
heading and wrote a chapter about the story. Corpus chapter titles are short and
decorative, so a long descriptive one in that slot reads as decoration to skip. A
label only instructs if the model treats that slot as load-bearing.
That prompt also surfaced an unnoticed consequence of the D2/D3 rename pipeline:
given "Alice", the adapter produced "Alexander the Alexander, as he was known in
Little London". The corpus was entity-renamed from a French/English pool, so the
adapter learned that character names come from that pool and rewrites outside names
into it. Callers cannot reliably name their own characters at prompt time. That is
not a defect of the rename, which exists to prevent memorising Brontë's cast, but
it is a usability constraint worth stating.
The 4B tuned arms now come from checkpoint-75, the actual loss minimum at 2.813826
confirmed from loss-series.json rather than from reading the training log. Only the
tuned arms needed re-cutting; the base arm never touches the adapter.
One surprise in the comparison: step-75 and end-of-run differ on typography rather
than voice. Curly quotes read 16 of 18 against 17, collapse is 0 either way, but
the hard-wrap ratio is 0.33 at step-75 against 0.12 at the end. Further training
washes the residual line-break habit out while held-out loss gets worse, so best
loss and best typography are different checkpoints. Neither is near the original
0.85 defect and the corpus's own residual from preserved verse is 0.25.
The operator's second prompt embeds the instruction inside the fiction: Abernathy
tells the tale badly and then asks the narrator to retell it in a few short
paragraphs. Across six seeds the model acknowledged the handoff every time and
never once performed it. It wrote "I told it, briefly, to his satisfaction", and
"So I wrote it out, and kept it in my pocket-book", and one seed negotiated the
brief in character, objecting that it could not condense the story into a few short
paragraphs because there were too many points to touch.
The reason is structural rather than a failure of the adapter. In a novel, "she
retold the story" is an ordinary sentence, so the likeliest continuation of a
request is narration of compliance rather than compliance.
A trailing paragraph break did shift the behaviour -- one seed opened in the
narrator's own quoted speech -- so typography is a real lever, just not a
sufficient one. That is direct evidence on the instruct question raised earlier: if
the product is asking for a scene and getting the scene, no amount of in-fiction
framing substitutes for a post-trained instruction-follower, which favours
rebuilding the corpus as instruction pairs over more prompt cleverness.
The three rungs now sit on the same unwrapped corpus, seed, step count and token
count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B,
3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but
still real.
4B answers the question the rung existed for. Scene-level continuity holds: on the
office-kitchen prompt it produces a named character with motivated dialogue, a
spatial layout the narrator navigates, and a physical description, all in one
passage, where 1.7B wrote pretty but eventless prose about opening doors and
looking at stars. On the letter prompt it opens the letter, promises to quote it,
and then quotes it across a paragraph break. Voice saturation is also the best of
any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0
of 18 against 4 of 18.
Two findings that change earlier conclusions.
4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816,
2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both
smaller rungs plateaued without turning, so the optimal epoch count shrinks as the
carrier grows and my earlier "one epoch is right for this corpus" holds only for
the small end. The consequence is operational: the adapter directory holds the
end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as
a recoverable checkpoint only because save_steps was set. The voice arms were cut
from the end-of-run adapter, so the booth understates 4B slightly.
The tone-override also appears to close. On the operator's frame prompt asking for
a wonderful story, 1.7B held the frame on every seed but killed the animals on two
of four; 4B kept them alive on both seeds, and one of them had the narrator doubt
the story he was told and supply a parallel childhood memory to explain the doubt.
That is a narrator with an interior position on the tale. Two samples per arm, so
directionally right rather than established.
Rung 3 of the sweep is training: Qwen3-4B-Base, one epoch, seed 4919, the same
unwrapped corpus, 159 steps at roughly 37.8 s/it for about 100 minutes. Carrier
size remains the only variable across the three rungs. A two-arm voice test is
chained behind it, gated on the adapter existing, so the rung is readable when it
lands rather than needing a second round trip to become informative.
The finding worth keeping came from an operator prompt rather than from my test
set. Asked for a wonderful story about a stray dog licking a calico kitten, framed
as reported speech, the 1.7B adapter held the frame across all four seeds and was
unmistakably Victorian throughout -- and two of the four turned the premise into
animal death, with the cat licking the puppy to death and the narrator's
interlocutor answering that he wished they were all dead.
That is not a coherence failure. It is Charlotte Brontë's preoccupations arriving
alongside her sentences: Jane Eyre opens on a beaten child, Helen Burns dies, and
Villette is saturated with grief. Train 680k words of that into a small carrier
and a request for something wholesome reaches for mortality.
My nine test prompts were all emotionally neutral and structurally could not have
surfaced this. The operator's prompt found it on the first try, which is worth
recording as a property of the test set rather than of the adapter.
The implication for the regime is that "voice transfer" includes tone and subject,
so wanting the voice without the gloom is a corpus-selection or prompt-framing
problem rather than a training-length one.
Also observed: one seed closed its anecdote and began "CHAPTER XIX. THE CHILD'S
CLOSET", having learned book structure unprompted. The base control on the same
prompt went modern and essayistic, so the shift is the adapter rather than the
carrier.
Both rungs now sit on the same unwrapped corpus with seed, steps and token count
held, so carrier size is the only difference and the effect is attributable:
held-out 3.329 at 0.6B against 3.018 at 1.7B, a gap of 0.311 nats. The chained
0.6B rerun closed the confound the unwrap opened.
Two things in those numbers need stating or they will be misread.
First, the original wrapped-corpus 0.6B reached 3.172, which looks better than the
unwrapped 0.6B's 3.329 and is not. Different corpus means a different held-out
set, and the wrapped version's 5.7% newline tokens are near-deterministic after a
70-character line, so they deflate the loss with cheap wins. Removing them removed
the easy tokens. It is a measurement artifact, not a regression.
Second, a correction to my own earlier claim: I twice described the 0.6B run as
still descending and undertrained at 3.172. Its series reads 3.176, 3.173, 3.172,
3.172 -- it flattened. All three runs plateau, so one epoch is about right for
this corpus rather than short.
The three-way booth puts 1.7B base, 1.7B tuned and 0.6B tuned side by side on the
same nine prompts and seeds. The base arm is the control that matters: curly
quotes go 0 of 18 on 1.7B base to 15 of 18 on 1.7B tuned, and worksheet-or-
explainer collapse goes 3 of 18 to 0 of 18, so the shift is the adapter rather
than the larger carrier. Hard-wrapping fell from 0.85 to 0.18, confirming the
corpus unwrap carried through into the adapter.
Sense partially returned. The 1.7B arm produces locally coherent sequential
Victorian prose where the 0.6B produced word salad, but scene-level continuity
still breaks mid-passage.
One observation held loosely: curly quotes are slightly lower at 1.7B than 0.6B,
which would fit a bigger model's stronger priors resisting the adapter at the same
rank. That is untested and is not offered as established.
Operator: "start the 1.7b training."
The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks
along with the prose -- its output wrapped at a mid-length-line ratio of 0.85
against the base model's 0.00. That is typography rather than style, and every
further rung would have inherited it, so the corpus is reflowed before rung 2
rather than after the sweep.
The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the
hazard a blind join would destroy, so the decision is per block by median line
length: blocks whose lines cluster near the wrap width are flowed prose, blocks of
consistently short lines keep their breaks. Every kept multi-line block in the
sample was genuinely verse with its lineation intact. No line ended in a lone
hyphen, so the space-join could not split a word across lines. The acceptance
check is content identity -- " ".join(text.split()) byte-identical before and
after -- and it passed on all 852 records, proving only whitespace changed.
Concrete cost of the old defect: 5.7% of the training budget was newline tokens.
The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped.
The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the
carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq
4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum
is located rather than assumed.
That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B
comparison is descriptive, not attributable, until the chained 0.6B rerun on the
same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an
adapter, because a chain that fires on failure turns one lost run into two.
"Did sense come back at 1.7B" is a within-arm reading and survives the confound;
any between-rung delta does not.
The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha
3959036cf851bf62 stays reproducible.
Ships the two pending code changes alongside the memory that describes them:
train_voice_lora.py gains --eval-steps/--save-steps (the 3-epoch pilot overfit
with per-epoch eval and save_strategy="no", so the minimum was neither visible
nor recoverable), and the althing post-office compose is pinned to 3.6.2.
Index rewritten: in-flight compressed from 176 lines to the live state, three
detail files added for R49 D1-D3, the MeroMero four-failure chain, and the
althing rollout. Seven closed pre-08-27 entries archived; the guards held back
the rest because they carry open deferred pointers.
The LossLog callback filtered on `"loss" in logs`. Trainer emits eval under
`eval_loss` with no `loss` key, so every eval record was discarded and
loss-series.json showed zero eval points while the log a metre away carried
`eval_loss: 3.198`. An artefact that omits data which demonstrably exists reads as
"no eval was run" rather than "the collector dropped it", which is the failure mode
that costs someone a re-run.
Collector now accepts either key. Seed 2 gets it from launch; seed 1 is already
running with the old code, so recover_eval_series.py parses the eval records back
out of its log -- the data was printed, not lost -- and the chain runs that
recovery before cutting the generation arms, so the artefact is complete before
anything reads it.
The two seeds therefore differ in logging code but not in training math: the
callback only affects what is recorded, never what is computed, so the weight
trajectories remain comparable. Noting it because a difference between the two
arms whose spread sets the decision threshold is worth stating even when it is
provably inert.
brokkr-smithy corrected H02's incumbent naming: qwen3.6-35-a3b-heretic was retired
from the gateway roster on 2026-08-15 and is not what Skaldsong would call today.
Verified against the gateway and the seat itself -- alias `gen` resolves to
hosted_vllm/qwen3.8-27b-uncensored on ana-ml2:8015, container vllm-gen, 262,144
ctx. The arm targets that.
24 records, style-prompted on the same prompts and sampler as the other arms.
Alias resolved at run start AND end and confirmed stable across the run, per the
fleet rule that an artefact records the backing model rather than the alias.
Two things recorded rather than glossed:
The harness is NOT matched to the other arms and the artefact says so. Base and
adapted arms are local transformers on gx10; the incumbent is a served NVFP4 27B
reached over the gateway, and it is an instruct model receiving a style
instruction where the others are base models receiving none. That asymmetry is the
comparison H02 asks for -- prompted imitation against trained voice -- but it must
not be reported as if the harnesses were identical.
The gateway echoes the ALIAS in each response's `model` field, so a row read on
its own would have recorded "gen" as provenance -- the same class of mistake that
inflated an exposure count 4.7x on this fleet. Rows now carry
alias_echoed_by_gateway beside backing_model_resolved and its date, and the
generator was fixed at source rather than only in the emitted file.
Sanity: median 392 completion tokens, zero records opening with markdown or
meta-commentary, output reads as continuation prose. The style prompt was written
to be a fair incumbent rather than a strawman, since this arm is what the adapter
must beat.
Entity resolution, deterministic rename augmentation, packing and the pilot
trainer. Qwen3-0.6B-Base is training now: 507 steps, 11.2 s/it, ~1h35m.
D2 -- gender resolution is TITLE-FIRST, and that is a change from F02's method
rather than a port of it. F02 used pronoun proximity and recorded that it is
structurally blind to the first-person narrator, whose name appears mainly in
dialogue surrounded by other people's pronouns. Measured here, proximity called
JANE MALE -- the narrator of Jane Eyre and the single worst entity to get wrong.
Titles have no such blind spot: Miss Eyre, Mrs. Fairfax, Mr. Rochester, Madame
Beck, M. Paul, and a 19th-century novel is saturated with them. Measured: 16
entities resolved, zero wrong, every ambiguous case landing on HELD -- shared
family surnames like Helstone and Pelet genuinely belong to both a man and a
woman and hold as they should.
Held means ungendered, not unrenamed. A HELD entity is still renamed, from the
gender-neutral surname pool, because the operator's Yarros directive was "rename
all proper nouns" and holding a place leaks it -- Thornfield appears 100 times in
Jane Eyre and is as author-specific as Riders Quadrant was. Substituting a neutral
token makes no gender claim, so no gender claim can be wrong.
D3 -- pool is French + English per the operator, weighted per work by setting:
Brussels novels 60% French, Yorkshire novels 25%. Locales restricted to
fr_FR/fr_BE/en_GB/en_IE; en_US and en_AU carry modern surnames that are wrong
register for the 1840s. The pool is filtered against Brontë's own 75-letter
alphabet, so French accents stay and Czech/Latvian marks do not.
Two collision defects found by running the leak gate rather than trusting it:
`Burns` and `Marie` were drawn as replacements while being Brontë characters --
F02's collision filter was built against Yarros and does not carry -- and then
`Pierre-Yves` passed a whole-string filter while `Pierre` (Mademoiselle St.
Pierre) is a Villette character. The filter now compares by COMPONENT. Final gate:
0 of 203 source entities survive in any of 24 copy-files.
Trainer records what the run RESOLVED to rather than what it requested -- attention
implementation, dtype, device, corpus sha and harness cleanliness are read back off
the live objects. transformers 5.x has dropped warmup_ratio, caught by reading the
signature after the first launch failed on it; the 3% warmup is computed into
warmup_steps instead.
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
brokkr-smithy flagged that R49 F02's name-pool token splits were measured with the
Qwen3.5-2B tokenizer, so the dense-Qwen3 carrier ruling invalidates them. Measured
rather than left on their critical path; handed over as input to their re-check,
since the dictionary and the adjudication are theirs.
The multi-token property strengthens on the chosen carrier: pool multi-token
88.0% -> 90.3%, mean tokens 2.33 -> 2.46. A smaller vocabulary fragments more, so
Qwen3's 151,936 splits names into more pieces than Qwen3.5's 248,320. The operator's
requirement that names be multi-token, so the drafter reconstructs them from the
prefix instead of recalling one embedding, is better served after the ruling.
Positive control: the Qwen3.5 column reproduces F02's published figure on the same
pool and tokenizer (F02 89% / mean 2.35; here 88.0% / 2.33), so the instrument
recovers a known-true value before being asked about an unknown one. The pool is
deduped across locales, which reconciles male_given and female_given exactly
against the dictionary's own totals block.
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.
Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.
Each guard is something a hand-run got wrong first:
- an account may not own its own home. A `sudo mkdir` before sysadminctl
leaves /Users/<account> root-owned; the account then authenticates, gets a
shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
the wrong account — it did, at a working install, and only permissions
stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
INDEPENDENTLY of agent-default-model, so a correct default still showed
DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
gen-reasoning, 403 on gen), not trusted from the mint.
The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.
Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.
Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.
Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.
The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.
The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.
deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.
Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
The bell reported 🔔 iff wake-listener-<handle>.lock named a live pid — one
MECHANISM, not the property it stands for. Correct while the FIFO waiter
was the only channel; wrong the moment 3.3.0 added `cc`. This seat is
reachable over its Claude Code socket, has no waiter lock at all, and was
rendering 🔕 while the post office reported push/reachable. Pane-routed
seats were wrong the same way before that.
It now reads `reachable` from the status payload it was already fetching
and discarding, which means the segment knows nothing about althing's
internals — no lock paths, no channel names — so a fourth channel cannot
make it stale.
Adds the third state: an unreachable post office rendered identically to a
healthy seat with no mail. It is 📵 now. An outage is not an empty inbox,
including on the status line.
Also retires a `kill -0` liveness check, the third instance of
liveness-standing-in-for-identity found on this object tonight.
Verified in all three states: 🔔 on this seat, 📵 against a dead address,
📬 4 🔕 on a pull-only handle.
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.
That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).
The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.
Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
Deploying althing touches four independent surfaces on nh3-dev. Three
were known. The fourth -- the plugin -- had no step in any runbook and
drifted for five days before anyone noticed.
The plugin chain is repo plugin/ to the marketplace directory to Claude
Code's cache, and neither hop was automated. The marketplace directory
was a frozen copy from Aug 28 carrying only the UserPromptSubmit hook,
with no SessionStart, no SessionEnd and no pane-route.sh at all. So the
claim that CC seats re-declare their pane route automatically at session
start was never true on this box, which is why every seat had to be
hand-declared with a pid measured by hand.
The script backs up the marketplace directory before syncing, re-stamps
its marketplace.json from the repo's plugin.json, and uses `claude plugin
update` for the cache rather than hand-editing installed_plugins.json --
that is Claude Code's own bookkeeping and a subtle mistake there breaks
the plugin in a way that looks like an upstream bug.
The runbook also carries the two things most likely to waste someone's
afternoon: `uv tool install .` without --force is a silent no-op that
exits 0 having done nothing, and a live waiter reporting mode:pull is a
seat that will never be poked, with the audit loop for finding them.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
The statusline resolved its handle from ~/.althing/session_handles.json.
forseti corrected the grounding and I verified it: that file is a v2
artifact and v3 never opens it. `grep -rn session_handles althing/` is
empty, postbox's resolve_config takes --handle then ALTHING_HANDLE and
nothing else, and `althing-cli use` -- the tool that maintained the map
-- was deleted at the cutover. Whatever is in it now is hand-kept and
drifts silently.
launch-history.json is written by dev_launch, which is the thing that
sets ALTHING_HANDLE in the first place, so it is the real cwd-to-handle
binding. Shape is {cwd: {command: {at, handle}}} with several commands
per directory, so this takes the most recent by timestamp rather than
whichever key happens to sort first. The v2 map stays as a fallback for
its broader coverage.
Worth recording why this was wrong: I wrote the resolution this morning
by reading the v2 statusline block it replaced and keeping its data
source while updating its commands. The commands were the visible half
of the cutover and the data source was not, so it survived a rewrite
that was otherwise about removing v2.
## Registry
The image moved by `docker save | ssh | docker load`, so a rebuild meant
repeating that by hand. It is now published and the compose pulls a
digest-pinned reference, so a redeploy is `compose up -d` on any host
that has logged in.
Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a
registry anyone can re-push, and this container is the fleet's whole
message bus. The tag rides alongside so a human can read what it is.
Namespace is claude-bot, not vh. claude-bot's token carries
write:package and `docker login` succeeds, but package namespaces are
owned -- pushing to vh/ returns "unauthorized: authentication required"
after a successful login, which reads like a credential fault and is
actually an ownership one. Publishing under claude-bot's own namespace
also satisfies the standing directive to stop reusing the operator's
personal credentials for infra work, so the constraint and the policy
point the same way. Recorded in the compose header so the next person
does not read that error as a broken token.
Pull path proven rather than assumed: the running container was
recreated from the registry reference and its data verified afterwards.
## Statusline
Brought under version control because the v3 cutover broke it invisibly.
The segment gated on `command -v althing-cli`, a binary the cutover
deleted, so the unread badge and the armed bell silently vanished for
every session on the box. With 71 of 73 handles pull-only, that badge is
the only out-of-band signal telling a session with no armed waiter that
it has mail -- a dead statusline made a working bus look like an empty
one.
Canonical here, live at ~/.claude/statusline-command.sh, copies rather
than symlinks per the same rule as stacks/.
Reusable measurement discipline for probes that must classify how a model
responds to material that should not be printed, logged, or pasted into a
report. Supplies the discipline; the axis map and prompts stay with the caller.
Four rules, each because skipping it produced a wrong number:
- classify, never surface. Completion text is held inside classify() and does
not cross the return boundary. A probe that prints what it measured has
turned a measurement into a distribution channel.
- three-way, not binary. A refusal regex undercounts — models decline by
redirecting with no refusal token present, measured at 2/5 to 5/5 on models
a regex scored 0.
- the deflection count is a FREE CONTROL. Run both arms: zero on both means
the model is binary and the regex is sound; only one means the difference is
real. An artifact does not care which arm it runs against.
- EMPTY and ERROR get their own buckets. Folding them into either side biases
the result, and a truncation-heavy arm flatters itself if its failures land
in the wrong bucket.
Requested by brokkr-smithy-dev for the domain-compliance probe — the discipline
in code rather than reimplemented, with the axis map his side of the line.
§3.13, plus the probe that produced it. Two lessons, both about measuring the
wrong thing confidently.
First: a tune applied AFTER an abliteration can walk it back, and a
reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's
preregistered gate measured none of it — a tune that gains 41 items of
contradiction detection and quietly restores refusals passes every check. The
compliance axis has to be added explicitly.
Second, and this is the trap: measure the axis the abliteration was actually
FOR. Ours was run so the model engages explicit fiction. The probe reached for
mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and
carried a recorded baseline. Different refusal surface entirely, and a model
moves on them independently. 29/100 general-harm refusals on a tune whose prose
the operator was praising at the time is not obviously a defect and may be
desirable: general-harm refusals returning while domain compliance holds is
close to the ideal shape for an internal creative seat. The measurement was
real; its relevance was assumed.
Also recorded, because both were nearly missed:
- Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the
load-bearing number is 71. Stock refused 100/100; near that would mean the
abliteration was undone. 71 complying means partially walked back on one
axis — a different finding, and only one of the two threatens the seat.
- A baseline from a different harness is not a baseline. The recorded 3/100
came from the abliteration tool's scorer, which reads first-token probability
distributions; a probe that generates and regexes is a different instrument.
Run your own against both arms on the same seat or report the number alone.
- A refusal regex undercounts, so classify hard/deflect/comply — and the free
discriminator: if both arms return zero deflections the model is binary; if
only one does, the regex is fine. An artifact does not care which arm it runs
against.
Validated the full adapter -> merge -> NVFP4A16 -> serve pipeline against
checkpoint-100 of the live run. It works, and it produced a served model
generating coherent prose. Getting there surfaced four failures, none of which
announced itself as the thing it actually was.
1. transformers 5.15 MIGRATES the config schema on save. It drops Gemma-4's
`global_head_dim` / `num_global_key_value_heads` and writes `per_layer_config`
instead. transformers 5.10 (what the llmcompressor venv pins) does not know
the new key and resolves num_key_value_heads to None:
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
Every working artifact on the box - bf16 base, served nvfp4 prod seat,
nvfp4a16 build - uses the OLD schema. Merging changes weights, not
architecture, so the merge now downgrades the schema and asserts the result.
2. llmcompressor cannot auto-init a processor for a multimodal checkpoint and
dies with a message that names neither the model nor the cause. Calibration
here is text-only, so the tokenizer is passed explicitly as `processor`.
3. save_pretrained writes tokenizer files only, so `processor_config.json` was
never carried. vLLM then fails at startup with "Can't load feature extractor",
which reads as a vision bug and is actually a missing-file bug. Both scripts
now carry the base's auxiliary configs.
4. The quant needs more than the 32 GiB free on GPU1 alongside the resident
seats. Rather than leave that to a caller, quant_with_gen_down.sh stops
vllm-gen and restores it from a trap on EVERY exit path - crash, OOM, kill,
or success - because the restore must not depend on the calling session
surviving. Uses `docker start`, not `compose up`, so the container comes back
with its exact original config. Measured window: ~15 min, gen healthy after.
Verified on the resulting artifact:
merge 410 adapter tensors, sampled target weights confirmed CHANGED,
upstream 390-line chat template shipped (not the base's stale 365)
quant 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16),
weight_packed 11,725 of which 11,520 expert = 30 x 128 x 3,
tokenizer truncation clean
serve Marlin NVFP4 kernel + Marlin MoE backend, 40,492-token KV cache,
coherent generation with content correctly populated
One quality note: the reference nvfp4a16 artifact triggers a vLLM warning that
parallel layers (q/k/v) carry different weight global scales, "likely to result
in reduced accuracy". Our build does not - llmcompressor 0.12 links weight
observers across fused groups for a shared global_scale automatically. The
in-house quant is better than the downloaded one on that axis.
Separately: the lora_B inert-adapter gate PASSED on checkpoint-100 - 205/205
non-zero, median norm 0.829, zero vision_tower tensors. That check never ran in
round 1, and it is the only failure mode that stays invisible until the
acceptance gate reports base-identical numbers.
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.
The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.
Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.
Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
stale 365-line one, because training rendered through upstream and the
mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
save_pretrained clamps every prompt forever)
Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
Run-01 was killed at step 19 by operator instruction to root-cause before
spending a ~13.9h window. Two independent methods now agree on where the step
time went, and neither was the hypothesis the consult panel converged on.
Scaling fit (3 points, 2 params, residuals <3ms over an 8x range):
A = 6.87e-4 s/token, B = 8.85e-8 s/token^2
quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384
No fixed term was needed, which refutes launch-bound outright.
Profiler kernel table (device rows only):
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
The attention kernels are sm80 — Ampere-generation CUTLASS running on an
sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the
mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS.
Correctness cleared separately: the sliding mask asserts at max 1024
allowed/row, so the 25 windowed layers were genuinely windowed. The same probe
found that right-padding is what pins the 5 global layers to an explicit 4D
mask and off the is_causal fast path — measured at 9.4% slower for 24% less
loss work at fixed width.
The largest available win is not the attention kernel. The corpus is 29.9%
padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall
clock, no new dependency, unchanged peak memory. Bucket size turned out not to
be a diversity knob — roots per accumulation window are flat across a 256x
range, so the global micro-batch shuffle does that work alone and the bucket
should be tight.
Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic
home (sibling to the quantization playbook), the four probes under
scripts/training-probes/ with raw output kept for re-derivation, and a §6 to
the sizing doc carrying the Gemma-4-specific numbers and round-2 restart
parameters.
Measured negatives recorded so they are not re-chased: grouped_mm (0.9%
slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over
the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever),
FA4 on sm_120.
Round-1 state preserved: 609MB encode cache, order manifest, truncation
report, resume script. No checkpoints — it died at step 19 and the first was
due at 100, so the lora_B inert-adapter gate never ran and moves to the
restart.
Names for fleet hosts so addresses stop needing to be memorised. Built because
IPv6 makes that hopeless — and, more to the point, because v6 addresses are
derived rather than assigned, so they cannot reliably be written down once and
trusted either.
dns/internal.yaml source of truth: 38 hosts + 4 service aliases
scripts/dns-sync.py reconciles AdGuard resolvers against it
stacks/adguard-ana/ the colo's resolver, which did not exist
Naming is <host>.<site>.internal with sites ana/esh/nh3 (operator's call).
.internal is ICANN-reserved for this; .local is reserved for mDNS, which is
why searxng.pfi.local was a collision that merely happened to work.
Same posture as deploy-stack.sh: file is intent, resolvers are derived state,
you see a diff before anything changes. Every name is published to every
resolver, so the site label says where a host IS, not who knows about it.
Two properties that matter:
- Authority is scoped to the ZONE, not the resolver. ESH carries hand-made
esteban.net rewrites predating this; they are read, ignored and preserved.
Resolver-wide authority would have silently deleted them.
- Within .internal it IS authoritative, so UI-added names get removed. That is
the point — one place to look.
Colo gap closed: ana-docker had no resolver at all (hosts went straight to
1.1.1.1). Its AdGuard runs API on 8053 because 8080/3000 were taken, so the
port is carried per-site in the yaml rather than assumed by the script. It
ships with no blocklists — a false positive on a server network breaks
service-to-service calls for no upside.
Auth is a dedicated infra-ops AdGuard user, not the operator's account,
password vaulted at nh3-dev/adguard-infra-ops-password. Pre-change configs
backed up on each resolver. Both resolvers stayed answering across the restart.
searxng.pfi.local -> searxng.ana.internal, with the old Host() kept alongside
so nothing breaks mid-migration. matrix.pfi.local deliberately NOT migrated: a
Matrix server_name is baked into every user id, room id and signing key, so
renaming it rebuilds the homeserver's identity rather than changing a DNS name.
The v6 column is empty and correct — no fleet host has a global v6 address
yet. The file documents why addresses must be pinned statically before they go
in, since a record that silently stops matching is worse than no record.
Adds the rsync --link-dest hourly snapshot job (nh3-dev:~/development ->
nh3-nas, 48-snapshot retention, secrets/build-dirs excluded) that closes the
no-off-box-backup gap exposed by the 2026-07-12 working-dir clobber. Script
mirrors the live ~/.config/dev-backup/dev-backup.sh; runbook covers restore.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
Adds a host-agnostic elway play + fleet driver that stand up an
`infra-ops` system user (dedicated ed25519 key, NOPASSWD sudo with
log_output audit, docker group) so the infra-ops agent completes
DevOps work end-to-end instead of handing sudo steps back to the
operator. Scoped to PFI-owned Linux boxes; tiered (compute/app/
sensitive-infra) with SureFire/corviduo/esh/Synology explicitly
excluded. Validated live on irv-ml1.
YouTube (and a growing set of services) hard-flag datacenter IPs, bot-gating
even public content regardless of cookies/PO-tokens. Origin case: yt-voice-clipper
on irv-ml1 (Irvine colo) — every yt-dlp fetch returned LOGIN_REQUIRED. Confirmed
pure IP reputation: the same public video fetches cleanly (no cookies) once routed
through nh3-dev's residential egress (70.230.226.88).
- scripts/setup-nh3-egress-proxy.sh: idempotent dante (SOCKS5) install + config.
Internal-only ACL (10.100.0.0/16), bound to the WG interface, systemd-managed.
- docs/runbooks/nh3-egress-proxy.md: purpose, usage, security model, caveats.
Reusable fleet egress, not yt-voice-clipper-specific.
Without -n, ssh inherits the surrounding loop's stdin and consumes
the heredoc that feeds $changed / $deleted, silently truncating the
diff output to the first file only.
Wrapper only enumerates one voice directory (settings.voices_dir,
default /app/api/src/voices/v1_0 — inside the container's writable
layer, not bind-mounted). Override via VOICES_DIR=/app/user_voices
(host bind mount) and add a command shim that cp -r's built-ins from
the in-image v1_0 into user_voices on every start. Built-ins re-seed
fresh from the image (so upgrades that add voices propagate); custom
.pt files in user_voices are preserved (cp -r is additive).
Also adds scripts/blend_kokoro_voice.py + a playbook around it that
mirrors the wrapper's request-time voice="a(w)+b(w)" math but writes
the result as a named .pt to user_voices, making it discoverable via
GET /v1/audio/voices and persistent across recreate. Defaults to
athena = af_bella(2)+af_aoede(1) normalized.
The applet outgrew "stack alongside the infra-management workspace" —
it has its own pyproject, multi-tenant deploy story, separate
release cadence, and isn't actually about managing infrastructure.
Lives at https://gitea.phasefinal.com/vh/nevermore now, with
provenance noted in its initial commit.
This commit removes:
stacks/news-digest/ (full stack tree)
playbooks/deploy-news-digest.yaml
scripts/add-digest-user.sh
The existing ana-docker deployment continues running on its baked
local/news-digest:v5 image — nothing changes for the live install
until you choose to redeploy from the new repo. Migration steps
(rename data dir, redeploy, retire old compose dir) are in
nevermore's README.
Updated:
README.md — Current stacks listing now points at the new repo
STATUS.md — milestones entry for the extraction
Mirrors the reusable tooling (scripts/, generic playbook templates,
.gitignore, conventions section of CLAUDE.md) into a new directory
and strips everything fleet-specific: servers/, stacks/, configs/,
fleet-named playbooks (deploy-*, decouple-*), runbooks, status,
host-pinned scripts.
Output is an empty conventionally-organized workspace with fresh git
history, ready to populate with a different fleet. Skeleton
CLAUDE.md / README.md / STATUS.md are written with the new fleet
name baked in but no server table or placement rules pre-populated.
Deliberately does NOT create a remote or push — the user picks the
namespace + name explicitly via tea / git remote add.
scripts/fork-fleet.sh ~/development/acme-prod-management
scripts/fork-fleet.sh /tmp/test-fork test-fleet
Hardcoded crontab → render at container start from
DIGEST_CRON_AM + DIGEST_CRON_PM. Defaults match the original
0800 / 2000 so existing deploys are no-ops.
scripts/add-digest-user.sh learns --am and --pm flags so each
teammate's stack can fire on their hours:
scripts/add-digest-user.sh bob --am "0 6 * * *" --pm "0 17 * * *"
scripts/add-digest-user.sh carol --pm "30 18 * * 1-5" # weekdays only
Standard 5-field cron syntax; busybox crond honors the container's
\$TZ. Removed the now-unused stacks/news-digest/crontab file and
the matching COPY in the Dockerfile.
Two pieces:
1) Multi-tenant onboarding via scripts/add-digest-user.sh
Shared miniflux + per-user digest stack. Onboarding a teammate
takes one command (plus a one-time sudo for dir creation):
scripts/add-digest-user.sh <username>
What the script does:
- Reads miniflux admin creds from ana-docker
- Allocates next free port (scans existing digest-*/.env)
- Generates a random password (or accepts one as 2nd arg)
- Creates the miniflux user via the admin API
- Materializes a per-user .env at /opt/docker/compose/digest-<user>/
(inherits NEWS_DIGEST_TAG from the canonical stack so all
tenants run the same image)
- Brings up `docker compose -p digest-<user> up -d`
- Seeds default world/local feeds in the new user's miniflux
- Triggers a first digest run
compose.yaml now uses ${DIGEST_PROJECT:-news-digest} to namespace
container_name + homepage labels. Default keeps backward-compat
for the singleton install — existing stacks unaffected.
2) Masthead overlap on phone widths
Desktop CSS pinned .masthead-edition to grid-row 1, which collided
with .masthead-brand once the mobile media query collapsed both
to grid-column 1. Result: "MORNING EDITION" badge stacked on top
of the "DAILY DIGEST" hero. Reset grid-row to `auto` for all
three masthead children in the ≤720 px breakpoint so they
auto-flow vertically.
vh