Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and
create an alias for char-rp-fast."
G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under
its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone
from the gateway and now returns an explicit 400 rather than a substitution; 0 of
17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays
erp-seat because asset-engine derives seat liveness from it.
The first quant of that A4B served NaN and passed its healthcheck doing it. It was
built with the dense v2-31B recipe, whose ignore list has no router regex, so all
30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts
run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool,
correct served name, and every completion returned finish_reason=length with the
full token count and content: null. The model was emitting a full budget of tokens
that decoded to the empty string. Raw /v1/completions was empty too, ruling out the
chat template and the reasoning parser. The signal that named it was logprobs:
vLLM refused to serialize the response, "Out of range float values are not JSON
compliant: nan".
The lesson is about the control rather than the router. That tree had already been
structurally diffed and passed -- against a verified-good DENSE quant of the same
Gemma-4 family. A dense model has no routers, so the single thing that was wrong
was the single thing the control could not distinguish. Diffing instead against
Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222
ignore entries against 252, the 30 missing being layers.N.router.proj. A positive
control is only worth what it can distinguish, and "same family" is not "same
architecture class".
Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and
refuses a router in the quantize set before any GPU time. The live seat then passed
prose with no channel-prefix leak, a solid-colour image read correctly, an auto
tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from
Pfish-6 unchanged. The broken tree is parked on ana-ml2 as
...-NVFP4A16.BROKEN-routers-quantized-20260910.
Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus
multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's
six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder
profiling. The substitute was reversibility and ordering: named .env backup, prove
the seat on its real port while no alias points at it, move the alias last. That is
why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes
across two attempts; no consumer saw a broken alias.
Playbook gains the router-quant failure signature and the control-class rule in
3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6.
Quality is NOT established: no RP eval, no long-context check, no A/B against
Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G.
The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError`
on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a
`per_layer_config` key zerofata's canonical one lacks -- and the standing fix was
to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong.
`pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The
config was serialized by 5.16.1, which materializes `per_layer_config` from
`global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not
the gemma4 resolver. Under the image's own transformers the same config loads
fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the
toolchain moved mid-diagnosis. Two things separated "malformed upload" from
"moved toolchain": reproducing the real failing call (a bare AutoConfig load does
not reproduce it; the trigger is reached through AutoTokenizer) and keeping
zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control.
The fix drops `per_layer_config` rather than forcing global access. It is exactly
redundant -- keys are precisely the ten full_attention layer indices, sole value
(512, 4), verbatim the global fields -- and forcing instead would make
`config.head_dim` answer 256 to the callers building the 512-wide layers.
patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever
stops holding.
Verified on the tensor table rather than the exit code: the output is identical
family-for-family and count-for-count to the August canonical quant, with 356
BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load
leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test
has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a
live seat displaced.
Also fixes the A4B output, which had a truncation cap baked into its tokenizer
(max_length 8192) from being quantized with the calibration corpus.
Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16
to say drop the dataset outright for any A16 scheme.
Ships the two pending code changes alongside the memory that describes them:
train_voice_lora.py gains --eval-steps/--save-steps (the 3-epoch pilot overfit
with per-epoch eval and save_strategy="no", so the minimum was neither visible
nor recoverable), and the althing post-office compose is pinned to 3.6.2.
Index rewritten: in-flight compressed from 176 lines to the live state, three
detail files added for R49 D1-D3, the MeroMero four-failure chain, and the
althing rollout. Seven closed pre-08-27 entries archived; the guards held back
the rest because they carry open deferred pointers.
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
CronList showed job 12bdea3c firing hourly at :37 with the verbatim "check on the
run ... inform brokkr when ready" text; CronDelete removed it and the list is now
empty. No crontab entry, user systemd timer, or background shell was re-seeding
it -- the only on-disk hits are conversation transcripts.
This is at least the second kill: the operator killed the same loop on 09-09
("kill the check, no training is running") and that session's handoff warned
against starting one. The warning was insufficient because a cron-fired prompt
arrives as an ordinary user turn with no marker, so it is indistinguishable from
the operator typing it -- three turns went into re-verifying an idle fleet and
answering a timer carefully.
The memory entry now says what to DO rather than what not to start: run CronList
before answering that prompt, and treat a second identical arrival as a signal to
check the cron list rather than answer again. Worth the prominence because the
instruction chain ends in three outward-facing acts -- an althing message to
Miranda, standing up a serving seat, and cueing brokkr -- every one of which
carries something false when no run exists.
uv tool install --force --reinstall from ~/development/althing (clean tree, HEAD
== origin/master 719bc5f, pyproject declaring 3.6.1). althing-core 3.6.0 -> 3.6.1;
herald PID 2457743 -> 3690918 with the old process confirmed gone; version read
back from the running install; postbox status reachable/push/alive; journal clean
since restart. postbox, althing-listen and althing-route moved on the same install.
Six boxes stay queued and the plugin hop (0.1.6) is separate and not done, so this
is deliberately not a "fleet done" claim. Reported to forseti per box.
Records one verification trap worth keeping: POKE_TEXT in post_office_herald.py is
a four-part implicit string concatenation, so a line-oriented grep for the new poke
sentence returns empty on a CORRECT 3.6.1 checkout. The first pass came back empty
for both the new text and the old, which reads as "new absent, old removed" -- two
wrong conclusions from one wrong pattern, detectable only because both cannot be
true at once. Check the effective value: reconstruct the concatenation or import
the module and print the constant.
Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.
pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final, merged-run02, merged-run03,
merged-run03-s{025,050,075}, merged-test,
nvfp4a16-test 8 dirs -> 354 GB
gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.
The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.
Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.
The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
(infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
Operator-directed question (via brokkr): which weights at
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 — stock or the heretic abliteration?
Answer from bytes: STOCK google/gemma-4-26B-A4B-it, byte-for-byte. Three-way match:
local shard sha256 (1127684971/aab47033) == HF download etags == stock repo LFS
oids, and download commit 4d7ae498 == stock HEAD. The recipes' -heretic-bf16 label
is a naming error (run-04 provenance was right; refusal telemetry confirmed). All
of runs 3/3c/4/5 trained from stock. Plausible cause: the selected llmfan46 26B-A4B
Heretic abliteration ships GGUF-only, so the bf16 safetensors pulled was stock.
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
First arm of the ERP-seat line where the capability gate did not fail. The
dependency-forcing slot (GovReport+QMSum, 3.46% of loss) broke the
diversity<->coherence coupling run 4 (STILL-COUPLED) and 3c (20x LR cut) could
not — structure of the loss was the lever, not its mass; INERT did not fire.
T4 8/8 (run 4: 5/8), t4_dissect noise 0.9062 vs run-3 0.5625, diversity held.
Reported-beside: T3 8->6 (new loss), RP-length partial fail, refusal erosion
(de-gated/stated), free-check base 5-6pp below run 4 (vLLM 0.28.0 unchanged).
Seat erp-tune-v5 served on gx10:8098 pending the operator's seat/ear decision.
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.
Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
vh