The LossLog callback filtered on `"loss" in logs`. Trainer emits eval under
`eval_loss` with no `loss` key, so every eval record was discarded and
loss-series.json showed zero eval points while the log a metre away carried
`eval_loss: 3.198`. An artefact that omits data which demonstrably exists reads as
"no eval was run" rather than "the collector dropped it", which is the failure mode
that costs someone a re-run.
Collector now accepts either key. Seed 2 gets it from launch; seed 1 is already
running with the old code, so recover_eval_series.py parses the eval records back
out of its log -- the data was printed, not lost -- and the chain runs that
recovery before cutting the generation arms, so the artefact is complete before
anything reads it.
The two seeds therefore differ in logging code but not in training math: the
callback only affects what is recorded, never what is computed, so the weight
trajectories remain comparable. Noting it because a difference between the two
arms whose spread sets the decision threshold is worth stating even when it is
provably inert.
brokkr-smithy corrected H02's incumbent naming: qwen3.6-35-a3b-heretic was retired
from the gateway roster on 2026-08-15 and is not what Skaldsong would call today.
Verified against the gateway and the seat itself -- alias `gen` resolves to
hosted_vllm/qwen3.8-27b-uncensored on ana-ml2:8015, container vllm-gen, 262,144
ctx. The arm targets that.
24 records, style-prompted on the same prompts and sampler as the other arms.
Alias resolved at run start AND end and confirmed stable across the run, per the
fleet rule that an artefact records the backing model rather than the alias.
Two things recorded rather than glossed:
The harness is NOT matched to the other arms and the artefact says so. Base and
adapted arms are local transformers on gx10; the incumbent is a served NVFP4 27B
reached over the gateway, and it is an instruct model receiving a style
instruction where the others are base models receiving none. That asymmetry is the
comparison H02 asks for -- prompted imitation against trained voice -- but it must
not be reported as if the harnesses were identical.
The gateway echoes the ALIAS in each response's `model` field, so a row read on
its own would have recorded "gen" as provenance -- the same class of mistake that
inflated an exposure count 4.7x on this fleet. Rows now carry
alias_echoed_by_gateway beside backing_model_resolved and its date, and the
generator was fixed at source rather than only in the emitted file.
Sanity: median 392 completion tokens, zero records opening with markdown or
meta-commentary, output reads as continuation prose. The style prompt was written
to be a fair incumbent rather than a strawman, since this arm is what the adapter
must beat.
Entity resolution, deterministic rename augmentation, packing and the pilot
trainer. Qwen3-0.6B-Base is training now: 507 steps, 11.2 s/it, ~1h35m.
D2 -- gender resolution is TITLE-FIRST, and that is a change from F02's method
rather than a port of it. F02 used pronoun proximity and recorded that it is
structurally blind to the first-person narrator, whose name appears mainly in
dialogue surrounded by other people's pronouns. Measured here, proximity called
JANE MALE -- the narrator of Jane Eyre and the single worst entity to get wrong.
Titles have no such blind spot: Miss Eyre, Mrs. Fairfax, Mr. Rochester, Madame
Beck, M. Paul, and a 19th-century novel is saturated with them. Measured: 16
entities resolved, zero wrong, every ambiguous case landing on HELD -- shared
family surnames like Helstone and Pelet genuinely belong to both a man and a
woman and hold as they should.
Held means ungendered, not unrenamed. A HELD entity is still renamed, from the
gender-neutral surname pool, because the operator's Yarros directive was "rename
all proper nouns" and holding a place leaks it -- Thornfield appears 100 times in
Jane Eyre and is as author-specific as Riders Quadrant was. Substituting a neutral
token makes no gender claim, so no gender claim can be wrong.
D3 -- pool is French + English per the operator, weighted per work by setting:
Brussels novels 60% French, Yorkshire novels 25%. Locales restricted to
fr_FR/fr_BE/en_GB/en_IE; en_US and en_AU carry modern surnames that are wrong
register for the 1840s. The pool is filtered against Brontë's own 75-letter
alphabet, so French accents stay and Czech/Latvian marks do not.
Two collision defects found by running the leak gate rather than trusting it:
`Burns` and `Marie` were drawn as replacements while being Brontë characters --
F02's collision filter was built against Yarros and does not carry -- and then
`Pierre-Yves` passed a whole-string filter while `Pierre` (Mademoiselle St.
Pierre) is a Villette character. The filter now compares by COMPONENT. Final gate:
0 of 203 source entities survive in any of 24 copy-files.
Trainer records what the run RESOLVED to rather than what it requested -- attention
implementation, dtype, device, corpus sha and harness cleanliness are read back off
the live objects. transformers 5.x has dropped warmup_ratio, caught by reading the
signature after the first launch failed on it; the 3% warmup is computed into
warmup_steps instead.
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
CronList showed job 12bdea3c firing hourly at :37 with the verbatim "check on the
run ... inform brokkr when ready" text; CronDelete removed it and the list is now
empty. No crontab entry, user systemd timer, or background shell was re-seeding
it -- the only on-disk hits are conversation transcripts.
This is at least the second kill: the operator killed the same loop on 09-09
("kill the check, no training is running") and that session's handoff warned
against starting one. The warning was insufficient because a cron-fired prompt
arrives as an ordinary user turn with no marker, so it is indistinguishable from
the operator typing it -- three turns went into re-verifying an idle fleet and
answering a timer carefully.
The memory entry now says what to DO rather than what not to start: run CronList
before answering that prompt, and treat a second identical arrival as a signal to
check the cron list rather than answer again. Worth the prominence because the
instruction chain ends in three outward-facing acts -- an althing message to
Miranda, standing up a serving seat, and cueing brokkr -- every one of which
carries something false when no run exists.
uv tool install --force --reinstall from ~/development/althing (clean tree, HEAD
== origin/master 719bc5f, pyproject declaring 3.6.1). althing-core 3.6.0 -> 3.6.1;
herald PID 2457743 -> 3690918 with the old process confirmed gone; version read
back from the running install; postbox status reachable/push/alive; journal clean
since restart. postbox, althing-listen and althing-route moved on the same install.
Six boxes stay queued and the plugin hop (0.1.6) is separate and not done, so this
is deliberately not a "fleet done" claim. Reported to forseti per box.
Records one verification trap worth keeping: POKE_TEXT in post_office_herald.py is
a four-part implicit string concatenation, so a line-oriented grep for the new poke
sentence returns empty on a CORRECT 3.6.1 checkout. The first pass came back empty
for both the new text and the old, which reads as "new absent, old removed" -- two
wrong conclusions from one wrong pattern, detectable only because both cannot be
true at once. Check the effective value: reconstruct the concatenation or import
the module and print the constant.
Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.
pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final, merged-run02, merged-run03,
merged-run03-s{025,050,075}, merged-test,
nvfp4a16-test 8 dirs -> 354 GB
gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.
The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.
Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.
The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
The operator asked brokkr-smithy to look for a smaller Qwen3.6/3.8; brokkr reported
none exists and that both 27Bs are the same hybrid kernel path one size up. Verified
independently against the HF API rather than adopted, and it holds -- with two facts
brokkr's note did not carry.
1. No official Qwen3.6 or Qwen3.8 below 27B. The listing is Qwen3.6-{27B, 35B-A3B}
and Qwen3.8-{27B, 2.4T-A95B, Flash-Next}.
2. Neither family publishes a -Base checkpoint AT ALL. Every Base newer than Qwen3
is Qwen3.5. Since the regime requires a non-instruct carrier, that rules the
3.6/3.8 lines out on its own, independent of size.
3. Qwen3.6-27B and Qwen3.8-27B both report model_type qwen3_5, 64 layers as 16 full
+ 48 linear-attention, vision tower present -- confirmed as stated.
So the dense Qwen3 ruling is not a compromise against a better available option; it
is the only dense Base option. Also recorded as insurance, not a proposal:
Qwen3-8B-Base and Qwen3-14B-Base exist, so the sweep can extend past 4B without
changing family if the carrier curve has not flattened there.
brokkr-smithy flagged that R49 F02's name-pool token splits were measured with the
Qwen3.5-2B tokenizer, so the dense-Qwen3 carrier ruling invalidates them. Measured
rather than left on their critical path; handed over as input to their re-check,
since the dictionary and the adjudication are theirs.
The multi-token property strengthens on the chosen carrier: pool multi-token
88.0% -> 90.3%, mean tokens 2.33 -> 2.46. A smaller vocabulary fragments more, so
Qwen3's 151,936 splits names into more pieces than Qwen3.5's 248,320. The operator's
requirement that names be multi-token, so the drafter reconstructs them from the
prefix instead of recalling one embedding, is better served after the ruling.
Positive control: the Qwen3.5 column reproduces F02's published figure on the same
pool and tokenizer (F02 89% / mean 2.35; here 88.0% / 2.33), so the instrument
recovers a known-true value before being asked about an unknown one. The pool is
deduped across locales, which reconciles male_given and female_given exactly
against the dictionary's own totals block.
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
The base-viability pre-flight had three checks (fits / MoE expert mapping / LoRA
support) and would have passed Qwen3.5-0.8B-Base clean while it trained 2.6x
slower than a dense model 2.3x its size. Check 4 closes that: read `layer_types`
for a linear_attention majority AND probe for mamba_ssm / causal_conv1d / fla /
kernels. It is the intersection that is slow -- a hybrid shape with the kernel
present is fine, a dense shape does not care.
Carries the measured table (gx10 GB10, n=10/arm, spreads 0.6-2.6%), plus the two
things a hybrid Base checkpoint brings that a dense one does not: a vision tower
and MTP head that target_modules="all-linear" would train on text, and the module
rename that AutoModelForCausalLM introduces relative to the vLLM serving class;
and unsafe cross-document packing, since SSM state ignores the attention mask.
Section heading corrected from "three greps" to "four checks". The example was
made runnable and verified on the box rather than shipped untested.
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is
retired (its gate failure was the detector bug fixed in cc42d76, but the run was
independently poor).
- served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6
- ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact
- SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails,
'kernel does not support current device'); Marlin is correct for a weight-only
scheme, and vLLM's 'no native FP4' warning is about the scheme not the card
- the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s,
3.2x the old ceiling, with single-stream latency unchanged
- head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363
rows, counting by the artifact gave 77, because the alias had carried three
different models that day. Wrong in the direction that looks careful.
From a near-miss brokkr-smithy-dev caught and declined to send: a post-fix
verification read raced a gateway restart and reported the alias still present.
A disagreement between two observers is not a finding until the boring
explanation is ruled out, and during an incident the pressure runs the other way.
Alias-present-with-backend-down is a different and worse state than
alias-removed: it re-arms silently the moment anything is served on
10.250.50.54:8021 again, and the next thing served there inherits a
fleet-reachable name without anyone deciding it should. Flagged by
brokkr-smithy-dev; same writer/reader-contract family as ops-lessons §11.
Commented rather than deleted so restoring is uncommenting, not reconstructing.
Verified: /v1/models returns 33 entries, trial absent, a caller gets a clean
400 'Invalid model name' instead of a connection error to a port that could
re-arm. Disposition of the artifact and the alias is the operator's.
- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
(infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
The empty-snapshot promote and brokkr-smithy-dev's -s-on-a-touched-sentinel are
the same defect from opposite sides. Records the direction split (false
reassurance / false refusal / false alarm) because the mitigations differ and a
post-mortem that lumps them loses the actionable half.
ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
Operator: the form failed when a question was left blank. Refusing the whole
submission over one blank threw away the picks that were made, and the HTML
`required` on the radios blocked it in the browser before the server saw it.
- answered questions recorded; blank ones land in `unanswered`; `complete` says
whether the set is finished; a blank question carrying a note keeps the note
- `required` dropped from both templates so the browser cannot block a partial
- refused only when there is no pick anywhere AND no notes (a 400 — that would
flip an open ask to answered with no decision recorded); a choice outside the
option list is still an error
- new ◐ partial state with an n/N count; skipped questions render as skipped
- README + global CLAUDE.md tell reading sessions to check `complete`
- 154 tests; v0.1.15
Operator verdict on the separate /asks page: the question belongs with the
artifact it is about. A four-voice audition wants each voice's radio group
under that voice's audio, and one submit for the lot.
- booth/inline.py: data-booth-ask="stem" | "stem:key" | data-booth-ask-submit,
plus <!-- booth:ask ... --> comments; unknown stem left alone, not blanked
- _ask_inline.html: self-contained fragments (own scoped styles, no JS), per-question
groups bound to one form via the HTML5 form= attribute so a scattered
multi-question ask still POSTs once
- unplaced questions and a missing submit block are appended, so a partially
marked-up page can never produce an unsubmittable 400
- chip becomes a jump link to the first open ask; /asks page kept as a fallback
- 6 tests (one caught the partial-placement drop); v0.1.14
A custom index.html is returned verbatim, so booth.html's asks panel never
rendered there — a valid ask (emmie-anchor/anchor.ask.json) was listed by the
CLI and shown nowhere, with nothing to say so.
- panel extracted to _asks.html; new GET /b/<name>/asks standalone page
- verbatim pages get an amber '? N open asks' chip beside the back chip
- POST /answer honours back=asks so answering returns to that page
- single-question asks now keep an optional 'title' (was silently dropped)
- README + routes table; 8 regression tests; v0.1.12
- asks.py: single {prompt, options} and multi {title, questions:[{key, prompt, options, notes?}]} both normalise to questions[]; per-question notes; every question required on submit
- /answer reads choice.<key> / notes.<key> / notes for multi; single shape unchanged
- booth asks prints per-question picks; README + CLI header; install step symlinks the CLI to ~/.local/bin; v0.1.10; 135 tests
Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15),
asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP
long-session fidelity, post-steps restore processor configs + template and reset the
tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time
- services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor)
- stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
Operator-directed question (via brokkr): which weights at
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 — stock or the heretic abliteration?
Answer from bytes: STOCK google/gemma-4-26B-A4B-it, byte-for-byte. Three-way match:
local shard sha256 (1127684971/aab47033) == HF download etags == stock repo LFS
oids, and download commit 4d7ae498 == stock HEAD. The recipes' -heretic-bf16 label
is a naming error (run-04 provenance was right; refusal telemetry confirmed). All
of runs 3/3c/4/5 trained from stock. Plausible cause: the selected llmfan46 26B-A4B
Heretic abliteration ships GGUF-only, so the bf16 safetensors pulled was stock.
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
First arm of the ERP-seat line where the capability gate did not fail. The
dependency-forcing slot (GovReport+QMSum, 3.46% of loss) broke the
diversity<->coherence coupling run 4 (STILL-COUPLED) and 3c (20x LR cut) could
not — structure of the loss was the lever, not its mass; INERT did not fire.
T4 8/8 (run 4: 5/8), t4_dissect noise 0.9062 vs run-3 0.5625, diversity held.
Reported-beside: T3 8->6 (new loss), RP-length partial fail, refusal erosion
(de-gated/stated), free-check base 5-6pp below run 4 (vLLM 0.28.0 unchanged).
Seat erp-tune-v5 served on gx10:8098 pending the operator's seat/ear decision.
ana-ml2 is up but not reachable cross-site from ESH (return-route gap on ana-ml2,
no SSH access to fix it). New stacks/ana-ml2-proxy: a pinned alpine/socat relay on
ana-docker (host net) exposing 10.250.50.70:2354 -> ana-ml2 10.250.50.54:2375.
Homepage docker.yaml ana-ml2-docker re-enabled pointed at the proxy. Verified:
esh-docker-vm reaches ana-ml2's docker (29.3.1) through it; /api/services returns
in 0s with 111 services; ana-ml2's inference seats (gen, coder, embed, rerank,
reward, image-bench, dockge) render. Reversible: point back at 10.250.50.54:2375
and compose-down the proxy once ana-ml2 routes to ESH.
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
WhereTF (operator-directed, brokkr-smithy-dev; nh3-dev:8093, user-systemd Bun
service): manual Homepage card under Apps (can't auto-discover) + DNS alias
wherethef.nh3.internal -> nh3-dev.
While adding the card, found Homepage's /api/services was hung (dashboard
degraded since the 2026-09-06 irv-ml1 move): its docker.yaml still pointed
irv-ml1 at the dead 10.100.79.3, and ana-ml2 (10.250.50.54:2375) is unreachable
from esh-docker-vm — either dead host hangs the whole aggregation on its TCP
connect timeout. Repointed irv-ml1 to 10.6.110.50 (verified reachable via mesh)
and disabled ana-ml2 (reversible, noted). /api/services now returns in 0s with
96 services; WhereTF renders.
ext-tts / tts-1 / tts-1-hd / gpt-4o-mini-tts (all openai/zonos) were pinned to
irv-ml1's pre-move address 10.100.79.3:8198, dead since the 2026-09-06 headscale
cutover to 10.6.110.50 — so ext-tts through the gateway hung. Reported by
tts-dev/svos-dev.
The 4 aliases are DB-backed (store_model_in_db), so their api_base was updated
via the admin API to the DNS name http://irv-ml1.nh3.internal:8198/v1 (not a
fresh IP — that just re-arms the trap on the next move). The container cannot
resolve *.nh3.internal via split-horizon DNS, so this adds an extra_hosts entry
mapping the name to 10.6.110.50. Verified: container resolves the name and a
live ext-tts/sindra call returns 200 + valid MP3.
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.
Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
The standing link board grew from a flat oldest-first list with a per-row
× into a manageable board: newest links lead, favorites stay on top, and
several dead links can go in one pass.
- Ordering: order_for_display() renders pinned rows first, then newest-first
within each group (the board is an append log, so newest = most recently
posted — the row you usually came to grab).
- Pin/favorite: a per-row ★ toggles pinned state via POST /b/<name>/pin.
State lives in a .pins sidecar dotfile (one content id per line), NOT
inline in links.md — so links.md stays a pure atomic-append log (many
sessions post concurrently) and a row's content id never changes just
because it was pinned. remove_link_entry drops a removed row's pin;
orphaned pins are inert (renderer only stars a live id).
- Multi-select delete: checkboxes feed POST /b/<name>/unlink-many (repeated
'sel' content ids), with a select-all box and a live count. The per-row ×
stays for single removal.
- One <form> with formaction buttons, so checkboxes, ×, ★, and bulk delete
coexist without nested forms AND all work with JS off; JS only adds
select-all and the live count. Per-row × confirm reads desc/url from
data-* attrs, so an arbitrary posted description can't break into the JS.
- Every action is keyed by content id, never row position — same race-safety
the existing × has, extended to the bulk path.
- Fixed pre-existing undefined --fg/--bg CSS refs in the board styles.
Tests: +19 (pins round-trip, ordering, orphan-inert, remove-unpins, /pin
and /unlink-many endpoints, board render + order). Full suite 102 passing.
Deployed to nh3-dev booth.service; verified live (newest-first, pin
round-trip, bulk delete) against the real 31-row board with no data loss.
The A record and the CLAUDE.md fleet row both still named 10.100.79.3,
the wg0 lifeline that went dormant in the headscale cutover — so the
name resolved to a dead door and the row read like an outage waiting to
be misdiagnosed.
- dns/internal.yaml: irv-ml1 10.100.79.3 -> 10.6.110.50 (Irvine's real
LAN, 10.6.110.0/24, advertised fleet-wide as a mesh subnet route).
Resolve to the LAN IP to match the host->LAN-IP model every other
entry uses; mesh node IP 100.64.0.6 recorded in the note. Comment
rewritten: Irvine is no longer numbered out of NH3's 10.100.79.0/24.
- CLAUDE.md: fleet-table row and GPU-placement rule updated to the mesh
reality (wg0 lifeline dormant, access via headscale mesh).
Verified: all three AdGuard resolvers (ana/esh/nh3) return 10.6.110.50;
name pings live ~11ms; reachable from a mesh node and from a colo host
behind a subnet router.
Changes the risk profile of the pending ledger-dev -> svos-dev rename, so it
belongs in memory rather than only in the thread: retire is the correct verb
(delete refuses any handle carrying mail) and it can be undone by declaring the
name again.
Also records that both deploy findings landed in althing's own INSTALL.md,
where they sit beside the journal_mode=WAL setting that causes them rather than
in a procedure doc that only describes working around it.
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.
Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.
The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.
Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.
Bus down about four minutes.
Records the fifo-relay fix and — more usefully — the two shapes that were tried
and rejected, so the next person does not re-derive them: a pipeline breaks cwd
persistence because every stage runs in a subshell, and a plain file is
invisible until the command ends. The absent `wait` is deliberate and is pinned
by a test, because it looks exactly like an oversight.
Corrects this file's own claim that infra-ops is sudo-less on nh3-extdev. It is
not: `sudo -n whoami` returns root. The wrong row would send a session looking
for a privilege path it already has.
Also flags that the althing tools are at 3.6.0 while the post office container
is still on 3.0.0, so the new handle verbs cannot work yet.
Found by using the tool for real work rather than by smoke-testing it. A bare
`sudo -n whoami` through ssh_run returns `running` with no output and wedges the
session permanently; the same command piped through `cat` returns everything.
Reproduced on both macOS and Debian, so it is the harness rather than a platform
quirk, and it matters because infra work is sudo work.
The workaround goes in CLAUDE.md where a session will actually hit it. The
proper fix is a protocol change — run the command through a pipe and read
PIPESTATUS — and is deliberately not attempted here.
Also records where the dsh work stopped: the package is updated but the running
web process is still on the old code, left alive on purpose because there is no
LaunchAgent to restart it into.
vh