CronList showed job 12bdea3c firing hourly at :37 with the verbatim "check on the
run ... inform brokkr when ready" text; CronDelete removed it and the list is now
empty. No crontab entry, user systemd timer, or background shell was re-seeding
it -- the only on-disk hits are conversation transcripts.
This is at least the second kill: the operator killed the same loop on 09-09
("kill the check, no training is running") and that session's handoff warned
against starting one. The warning was insufficient because a cron-fired prompt
arrives as an ordinary user turn with no marker, so it is indistinguishable from
the operator typing it -- three turns went into re-verifying an idle fleet and
answering a timer carefully.
The memory entry now says what to DO rather than what not to start: run CronList
before answering that prompt, and treat a second identical arrival as a signal to
check the cron list rather than answer again. Worth the prominence because the
instruction chain ends in three outward-facing acts -- an althing message to
Miranda, standing up a serving seat, and cueing brokkr -- every one of which
carries something false when no run exists.
uv tool install --force --reinstall from ~/development/althing (clean tree, HEAD
== origin/master 719bc5f, pyproject declaring 3.6.1). althing-core 3.6.0 -> 3.6.1;
herald PID 2457743 -> 3690918 with the old process confirmed gone; version read
back from the running install; postbox status reachable/push/alive; journal clean
since restart. postbox, althing-listen and althing-route moved on the same install.
Six boxes stay queued and the plugin hop (0.1.6) is separate and not done, so this
is deliberately not a "fleet done" claim. Reported to forseti per box.
Records one verification trap worth keeping: POKE_TEXT in post_office_herald.py is
a four-part implicit string concatenation, so a line-oriented grep for the new poke
sentence returns empty on a CORRECT 3.6.1 checkout. The first pass came back empty
for both the new text and the old, which reads as "new absent, old removed" -- two
wrong conclusions from one wrong pattern, detectable only because both cannot be
true at once. Check the effective value: reconstruct the concatenation or import
the module and print the constant.
Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.
pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final, merged-run02, merged-run03,
merged-run03-s{025,050,075}, merged-test,
nvfp4a16-test 8 dirs -> 354 GB
gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.
The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.
Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.
The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
The operator asked brokkr-smithy to look for a smaller Qwen3.6/3.8; brokkr reported
none exists and that both 27Bs are the same hybrid kernel path one size up. Verified
independently against the HF API rather than adopted, and it holds -- with two facts
brokkr's note did not carry.
1. No official Qwen3.6 or Qwen3.8 below 27B. The listing is Qwen3.6-{27B, 35B-A3B}
and Qwen3.8-{27B, 2.4T-A95B, Flash-Next}.
2. Neither family publishes a -Base checkpoint AT ALL. Every Base newer than Qwen3
is Qwen3.5. Since the regime requires a non-instruct carrier, that rules the
3.6/3.8 lines out on its own, independent of size.
3. Qwen3.6-27B and Qwen3.8-27B both report model_type qwen3_5, 64 layers as 16 full
+ 48 linear-attention, vision tower present -- confirmed as stated.
So the dense Qwen3 ruling is not a compromise against a better available option; it
is the only dense Base option. Also recorded as insurance, not a proposal:
Qwen3-8B-Base and Qwen3-14B-Base exist, so the sweep can extend past 4B without
changing family if the carrier curve has not flattened there.
brokkr-smithy flagged that R49 F02's name-pool token splits were measured with the
Qwen3.5-2B tokenizer, so the dense-Qwen3 carrier ruling invalidates them. Measured
rather than left on their critical path; handed over as input to their re-check,
since the dictionary and the adjudication are theirs.
The multi-token property strengthens on the chosen carrier: pool multi-token
88.0% -> 90.3%, mean tokens 2.33 -> 2.46. A smaller vocabulary fragments more, so
Qwen3's 151,936 splits names into more pieces than Qwen3.5's 248,320. The operator's
requirement that names be multi-token, so the drafter reconstructs them from the
prefix instead of recalling one embedding, is better served after the ruling.
Positive control: the Qwen3.5 column reproduces F02's published figure on the same
pool and tokenizer (F02 89% / mean 2.35; here 88.0% / 2.33), so the instrument
recovers a known-true value before being asked about an unknown one. The pool is
deduped across locales, which reconciles male_given and female_given exactly
against the dictionary's own totals block.
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
The base-viability pre-flight had three checks (fits / MoE expert mapping / LoRA
support) and would have passed Qwen3.5-0.8B-Base clean while it trained 2.6x
slower than a dense model 2.3x its size. Check 4 closes that: read `layer_types`
for a linear_attention majority AND probe for mamba_ssm / causal_conv1d / fla /
kernels. It is the intersection that is slow -- a hybrid shape with the kernel
present is fine, a dense shape does not care.
Carries the measured table (gx10 GB10, n=10/arm, spreads 0.6-2.6%), plus the two
things a hybrid Base checkpoint brings that a dense one does not: a vision tower
and MTP head that target_modules="all-linear" would train on text, and the module
rename that AutoModelForCausalLM introduces relative to the vLLM serving class;
and unsafe cross-document packing, since SSM state ignores the attention mask.
Section heading corrected from "three greps" to "four checks". The example was
made runnable and verified on the box rather than shipped untested.
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is
retired (its gate failure was the detector bug fixed in cc42d76, but the run was
independently poor).
- served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6
- ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact
- SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails,
'kernel does not support current device'); Marlin is correct for a weight-only
scheme, and vLLM's 'no native FP4' warning is about the scheme not the card
- the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s,
3.2x the old ceiling, with single-stream latency unchanged
- head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363
rows, counting by the artifact gave 77, because the alias had carried three
different models that day. Wrong in the direction that looks careful.
From a near-miss brokkr-smithy-dev caught and declined to send: a post-fix
verification read raced a gateway restart and reported the alias still present.
A disagreement between two observers is not a finding until the boring
explanation is ruled out, and during an incident the pressure runs the other way.
Alias-present-with-backend-down is a different and worse state than
alias-removed: it re-arms silently the moment anything is served on
10.250.50.54:8021 again, and the next thing served there inherits a
fleet-reachable name without anyone deciding it should. Flagged by
brokkr-smithy-dev; same writer/reader-contract family as ops-lessons §11.
Commented rather than deleted so restoring is uncommenting, not reconstructing.
Verified: /v1/models returns 33 entries, trial absent, a caller gets a clean
400 'Invalid model name' instead of a connection error to a port that could
re-arm. Disposition of the artifact and the alias is the operator's.
- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
(infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
The empty-snapshot promote and brokkr-smithy-dev's -s-on-a-touched-sentinel are
the same defect from opposite sides. Records the direction split (false
reassurance / false refusal / false alarm) because the mitigations differ and a
post-mortem that lumps them loses the actionable half.
ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
Operator: the form failed when a question was left blank. Refusing the whole
submission over one blank threw away the picks that were made, and the HTML
`required` on the radios blocked it in the browser before the server saw it.
- answered questions recorded; blank ones land in `unanswered`; `complete` says
whether the set is finished; a blank question carrying a note keeps the note
- `required` dropped from both templates so the browser cannot block a partial
- refused only when there is no pick anywhere AND no notes (a 400 — that would
flip an open ask to answered with no decision recorded); a choice outside the
option list is still an error
- new ◐ partial state with an n/N count; skipped questions render as skipped
- README + global CLAUDE.md tell reading sessions to check `complete`
- 154 tests; v0.1.15
Operator verdict on the separate /asks page: the question belongs with the
artifact it is about. A four-voice audition wants each voice's radio group
under that voice's audio, and one submit for the lot.
- booth/inline.py: data-booth-ask="stem" | "stem:key" | data-booth-ask-submit,
plus <!-- booth:ask ... --> comments; unknown stem left alone, not blanked
- _ask_inline.html: self-contained fragments (own scoped styles, no JS), per-question
groups bound to one form via the HTML5 form= attribute so a scattered
multi-question ask still POSTs once
- unplaced questions and a missing submit block are appended, so a partially
marked-up page can never produce an unsubmittable 400
- chip becomes a jump link to the first open ask; /asks page kept as a fallback
- 6 tests (one caught the partial-placement drop); v0.1.14
A custom index.html is returned verbatim, so booth.html's asks panel never
rendered there — a valid ask (emmie-anchor/anchor.ask.json) was listed by the
CLI and shown nowhere, with nothing to say so.
- panel extracted to _asks.html; new GET /b/<name>/asks standalone page
- verbatim pages get an amber '? N open asks' chip beside the back chip
- POST /answer honours back=asks so answering returns to that page
- single-question asks now keep an optional 'title' (was silently dropped)
- README + routes table; 8 regression tests; v0.1.12
- asks.py: single {prompt, options} and multi {title, questions:[{key, prompt, options, notes?}]} both normalise to questions[]; per-question notes; every question required on submit
- /answer reads choice.<key> / notes.<key> / notes for multi; single shape unchanged
- booth asks prints per-question picks; README + CLI header; install step symlinks the CLI to ~/.local/bin; v0.1.10; 135 tests
Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15),
asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP
long-session fidelity, post-steps restore processor configs + template and reset the
tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time
- services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor)
- stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
vh