The LossLog callback filtered on `"loss" in logs`. Trainer emits eval under
`eval_loss` with no `loss` key, so every eval record was discarded and
loss-series.json showed zero eval points while the log a metre away carried
`eval_loss: 3.198`. An artefact that omits data which demonstrably exists reads as
"no eval was run" rather than "the collector dropped it", which is the failure mode
that costs someone a re-run.
Collector now accepts either key. Seed 2 gets it from launch; seed 1 is already
running with the old code, so recover_eval_series.py parses the eval records back
out of its log -- the data was printed, not lost -- and the chain runs that
recovery before cutting the generation arms, so the artefact is complete before
anything reads it.
The two seeds therefore differ in logging code but not in training math: the
callback only affects what is recorded, never what is computed, so the weight
trajectories remain comparable. Noting it because a difference between the two
arms whose spread sets the decision threshold is worth stating even when it is
provably inert.
brokkr-smithy corrected H02's incumbent naming: qwen3.6-35-a3b-heretic was retired
from the gateway roster on 2026-08-15 and is not what Skaldsong would call today.
Verified against the gateway and the seat itself -- alias `gen` resolves to
hosted_vllm/qwen3.8-27b-uncensored on ana-ml2:8015, container vllm-gen, 262,144
ctx. The arm targets that.
24 records, style-prompted on the same prompts and sampler as the other arms.
Alias resolved at run start AND end and confirmed stable across the run, per the
fleet rule that an artefact records the backing model rather than the alias.
Two things recorded rather than glossed:
The harness is NOT matched to the other arms and the artefact says so. Base and
adapted arms are local transformers on gx10; the incumbent is a served NVFP4 27B
reached over the gateway, and it is an instruct model receiving a style
instruction where the others are base models receiving none. That asymmetry is the
comparison H02 asks for -- prompted imitation against trained voice -- but it must
not be reported as if the harnesses were identical.
The gateway echoes the ALIAS in each response's `model` field, so a row read on
its own would have recorded "gen" as provenance -- the same class of mistake that
inflated an exposure count 4.7x on this fleet. Rows now carry
alias_echoed_by_gateway beside backing_model_resolved and its date, and the
generator was fixed at source rather than only in the emitted file.
Sanity: median 392 completion tokens, zero records opening with markdown or
meta-commentary, output reads as continuation prose. The style prompt was written
to be a fair incumbent rather than a strawman, since this arm is what the adapter
must beat.
Entity resolution, deterministic rename augmentation, packing and the pilot
trainer. Qwen3-0.6B-Base is training now: 507 steps, 11.2 s/it, ~1h35m.
D2 -- gender resolution is TITLE-FIRST, and that is a change from F02's method
rather than a port of it. F02 used pronoun proximity and recorded that it is
structurally blind to the first-person narrator, whose name appears mainly in
dialogue surrounded by other people's pronouns. Measured here, proximity called
JANE MALE -- the narrator of Jane Eyre and the single worst entity to get wrong.
Titles have no such blind spot: Miss Eyre, Mrs. Fairfax, Mr. Rochester, Madame
Beck, M. Paul, and a 19th-century novel is saturated with them. Measured: 16
entities resolved, zero wrong, every ambiguous case landing on HELD -- shared
family surnames like Helstone and Pelet genuinely belong to both a man and a
woman and hold as they should.
Held means ungendered, not unrenamed. A HELD entity is still renamed, from the
gender-neutral surname pool, because the operator's Yarros directive was "rename
all proper nouns" and holding a place leaks it -- Thornfield appears 100 times in
Jane Eyre and is as author-specific as Riders Quadrant was. Substituting a neutral
token makes no gender claim, so no gender claim can be wrong.
D3 -- pool is French + English per the operator, weighted per work by setting:
Brussels novels 60% French, Yorkshire novels 25%. Locales restricted to
fr_FR/fr_BE/en_GB/en_IE; en_US and en_AU carry modern surnames that are wrong
register for the 1840s. The pool is filtered against Brontë's own 75-letter
alphabet, so French accents stay and Czech/Latvian marks do not.
Two collision defects found by running the leak gate rather than trusting it:
`Burns` and `Marie` were drawn as replacements while being Brontë characters --
F02's collision filter was built against Yarros and does not carry -- and then
`Pierre-Yves` passed a whole-string filter while `Pierre` (Mademoiselle St.
Pierre) is a Villette character. The filter now compares by COMPONENT. Final gate:
0 of 203 source entities survive in any of 24 copy-files.
Trainer records what the run RESOLVED to rather than what it requested -- attention
implementation, dtype, device, corpus sha and harness cleanliness are read back off
the live objects. transformers 5.x has dropped warmup_ratio, caught by reading the
signature after the first launch failed on it; the 3% warmup is computed into
warmup_steps instead.
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
brokkr-smithy flagged that R49 F02's name-pool token splits were measured with the
Qwen3.5-2B tokenizer, so the dense-Qwen3 carrier ruling invalidates them. Measured
rather than left on their critical path; handed over as input to their re-check,
since the dictionary and the adjudication are theirs.
The multi-token property strengthens on the chosen carrier: pool multi-token
88.0% -> 90.3%, mean tokens 2.33 -> 2.46. A smaller vocabulary fragments more, so
Qwen3's 151,936 splits names into more pieces than Qwen3.5's 248,320. The operator's
requirement that names be multi-token, so the drafter reconstructs them from the
prefix instead of recalling one embedding, is better served after the ruling.
Positive control: the Qwen3.5 column reproduces F02's published figure on the same
pool and tokenizer (F02 89% / mean 2.35; here 88.0% / 2.33), so the instrument
recovers a known-true value before being asked about an unknown one. The pool is
deduped across locales, which reconciles male_given and female_given exactly
against the dictionary's own totals block.
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
ssh exiting 0 is not proof the capture is usable — the inspect script can emit
nothing and both refresh scripts would mv that over a good system-details.txt and
report 'ok (0 bytes)'. Every reader tests the snapshot with -s, so the writer was
producing an artifact its own readers call invalid: a guard whose test disagrees
with its writer's contract has quietly stopped guarding. Prompted by
brokkr-smithy-dev hitting the same shape from the other side (a -s test against a
sentinel written with touch, a precondition that could never pass).
- empty capture -> refused, previous snapshot kept, host counted as failed (exit 1)
- capture under 1/4 of the previous -> promoted but flagged, since a host can
legitimately shed services and the script should not guess
- header + CLAUDE.md contract lines corrected to say what is actually guaranteed
- verified red (empty inspect -> FAIL, snapshot intact, rc=1) then green (real host
-> ok 6727 bytes)
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.
Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.
Each guard is something a hand-run got wrong first:
- an account may not own its own home. A `sudo mkdir` before sysadminctl
leaves /Users/<account> root-owned; the account then authenticates, gets a
shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
the wrong account — it did, at a working install, and only permissions
stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
INDEPENDENTLY of agent-default-model, so a correct default still showed
DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
gen-reasoning, 403 on gen), not trusted from the mint.
The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.
Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.
Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.
Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.
The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.
The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.
deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.
Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
The bell reported 🔔 iff wake-listener-<handle>.lock named a live pid — one
MECHANISM, not the property it stands for. Correct while the FIFO waiter
was the only channel; wrong the moment 3.3.0 added `cc`. This seat is
reachable over its Claude Code socket, has no waiter lock at all, and was
rendering 🔕 while the post office reported push/reachable. Pane-routed
seats were wrong the same way before that.
It now reads `reachable` from the status payload it was already fetching
and discarding, which means the segment knows nothing about althing's
internals — no lock paths, no channel names — so a fourth channel cannot
make it stale.
Adds the third state: an unreachable post office rendered identically to a
healthy seat with no mail. It is 📵 now. An outage is not an empty inbox,
including on the status line.
Also retires a `kill -0` liveness check, the third instance of
liveness-standing-in-for-identity found on this object tonight.
Verified in all three states: 🔔 on this seat, 📵 against a dead address,
📬 4 🔕 on a pull-only handle.
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.
That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).
The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.
Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
Deploying althing touches four independent surfaces on nh3-dev. Three
were known. The fourth -- the plugin -- had no step in any runbook and
drifted for five days before anyone noticed.
The plugin chain is repo plugin/ to the marketplace directory to Claude
Code's cache, and neither hop was automated. The marketplace directory
was a frozen copy from Aug 28 carrying only the UserPromptSubmit hook,
with no SessionStart, no SessionEnd and no pane-route.sh at all. So the
claim that CC seats re-declare their pane route automatically at session
start was never true on this box, which is why every seat had to be
hand-declared with a pid measured by hand.
The script backs up the marketplace directory before syncing, re-stamps
its marketplace.json from the repo's plugin.json, and uses `claude plugin
update` for the cache rather than hand-editing installed_plugins.json --
that is Claude Code's own bookkeeping and a subtle mistake there breaks
the plugin in a way that looks like an upstream bug.
The runbook also carries the two things most likely to waste someone's
afternoon: `uv tool install .` without --force is a silent no-op that
exits 0 having done nothing, and a live waiter reporting mode:pull is a
seat that will never be poked, with the audit loop for finding them.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
The statusline resolved its handle from ~/.althing/session_handles.json.
forseti corrected the grounding and I verified it: that file is a v2
artifact and v3 never opens it. `grep -rn session_handles althing/` is
empty, postbox's resolve_config takes --handle then ALTHING_HANDLE and
nothing else, and `althing-cli use` -- the tool that maintained the map
-- was deleted at the cutover. Whatever is in it now is hand-kept and
drifts silently.
launch-history.json is written by dev_launch, which is the thing that
sets ALTHING_HANDLE in the first place, so it is the real cwd-to-handle
binding. Shape is {cwd: {command: {at, handle}}} with several commands
per directory, so this takes the most recent by timestamp rather than
whichever key happens to sort first. The v2 map stays as a fallback for
its broader coverage.
Worth recording why this was wrong: I wrote the resolution this morning
by reading the v2 statusline block it replaced and keeping its data
source while updating its commands. The commands were the visible half
of the cutover and the data source was not, so it survived a rewrite
that was otherwise about removing v2.
## Registry
The image moved by `docker save | ssh | docker load`, so a rebuild meant
repeating that by hand. It is now published and the compose pulls a
digest-pinned reference, so a redeploy is `compose up -d` on any host
that has logged in.
Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a
registry anyone can re-push, and this container is the fleet's whole
message bus. The tag rides alongside so a human can read what it is.
Namespace is claude-bot, not vh. claude-bot's token carries
write:package and `docker login` succeeds, but package namespaces are
owned -- pushing to vh/ returns "unauthorized: authentication required"
after a successful login, which reads like a credential fault and is
actually an ownership one. Publishing under claude-bot's own namespace
also satisfies the standing directive to stop reusing the operator's
personal credentials for infra work, so the constraint and the policy
point the same way. Recorded in the compose header so the next person
does not read that error as a broken token.
Pull path proven rather than assumed: the running container was
recreated from the registry reference and its data verified afterwards.
## Statusline
Brought under version control because the v3 cutover broke it invisibly.
The segment gated on `command -v althing-cli`, a binary the cutover
deleted, so the unread badge and the armed bell silently vanished for
every session on the box. With 71 of 73 handles pull-only, that badge is
the only out-of-band signal telling a session with no armed waiter that
it has mail -- a dead statusline made a working bus look like an empty
one.
Canonical here, live at ~/.claude/statusline-command.sh, copies rather
than symlinks per the same rule as stacks/.
Reusable measurement discipline for probes that must classify how a model
responds to material that should not be printed, logged, or pasted into a
report. Supplies the discipline; the axis map and prompts stay with the caller.
Four rules, each because skipping it produced a wrong number:
- classify, never surface. Completion text is held inside classify() and does
not cross the return boundary. A probe that prints what it measured has
turned a measurement into a distribution channel.
- three-way, not binary. A refusal regex undercounts — models decline by
redirecting with no refusal token present, measured at 2/5 to 5/5 on models
a regex scored 0.
- the deflection count is a FREE CONTROL. Run both arms: zero on both means
the model is binary and the regex is sound; only one means the difference is
real. An artifact does not care which arm it runs against.
- EMPTY and ERROR get their own buckets. Folding them into either side biases
the result, and a truncation-heavy arm flatters itself if its failures land
in the wrong bucket.
Requested by brokkr-smithy-dev for the domain-compliance probe — the discipline
in code rather than reimplemented, with the axis map his side of the line.
§3.13, plus the probe that produced it. Two lessons, both about measuring the
wrong thing confidently.
First: a tune applied AFTER an abliteration can walk it back, and a
reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's
preregistered gate measured none of it — a tune that gains 41 items of
contradiction detection and quietly restores refusals passes every check. The
compliance axis has to be added explicitly.
Second, and this is the trap: measure the axis the abliteration was actually
FOR. Ours was run so the model engages explicit fiction. The probe reached for
mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and
carried a recorded baseline. Different refusal surface entirely, and a model
moves on them independently. 29/100 general-harm refusals on a tune whose prose
the operator was praising at the time is not obviously a defect and may be
desirable: general-harm refusals returning while domain compliance holds is
close to the ideal shape for an internal creative seat. The measurement was
real; its relevance was assumed.
Also recorded, because both were nearly missed:
- Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the
load-bearing number is 71. Stock refused 100/100; near that would mean the
abliteration was undone. 71 complying means partially walked back on one
axis — a different finding, and only one of the two threatens the seat.
- A baseline from a different harness is not a baseline. The recorded 3/100
came from the abliteration tool's scorer, which reads first-token probability
distributions; a probe that generates and regexes is a different instrument.
Run your own against both arms on the same seat or report the number alone.
- A refusal regex undercounts, so classify hard/deflect/comply — and the free
discriminator: if both arms return zero deflections the model is binary; if
only one does, the regex is fine. An artifact does not care which arm it runs
against.
Validated the full adapter -> merge -> NVFP4A16 -> serve pipeline against
checkpoint-100 of the live run. It works, and it produced a served model
generating coherent prose. Getting there surfaced four failures, none of which
announced itself as the thing it actually was.
1. transformers 5.15 MIGRATES the config schema on save. It drops Gemma-4's
`global_head_dim` / `num_global_key_value_heads` and writes `per_layer_config`
instead. transformers 5.10 (what the llmcompressor venv pins) does not know
the new key and resolves num_key_value_heads to None:
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
Every working artifact on the box - bf16 base, served nvfp4 prod seat,
nvfp4a16 build - uses the OLD schema. Merging changes weights, not
architecture, so the merge now downgrades the schema and asserts the result.
2. llmcompressor cannot auto-init a processor for a multimodal checkpoint and
dies with a message that names neither the model nor the cause. Calibration
here is text-only, so the tokenizer is passed explicitly as `processor`.
3. save_pretrained writes tokenizer files only, so `processor_config.json` was
never carried. vLLM then fails at startup with "Can't load feature extractor",
which reads as a vision bug and is actually a missing-file bug. Both scripts
now carry the base's auxiliary configs.
4. The quant needs more than the 32 GiB free on GPU1 alongside the resident
seats. Rather than leave that to a caller, quant_with_gen_down.sh stops
vllm-gen and restores it from a trap on EVERY exit path - crash, OOM, kill,
or success - because the restore must not depend on the calling session
surviving. Uses `docker start`, not `compose up`, so the container comes back
with its exact original config. Measured window: ~15 min, gen healthy after.
Verified on the resulting artifact:
merge 410 adapter tensors, sampled target weights confirmed CHANGED,
upstream 390-line chat template shipped (not the base's stale 365)
quant 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16),
weight_packed 11,725 of which 11,520 expert = 30 x 128 x 3,
tokenizer truncation clean
serve Marlin NVFP4 kernel + Marlin MoE backend, 40,492-token KV cache,
coherent generation with content correctly populated
One quality note: the reference nvfp4a16 artifact triggers a vLLM warning that
parallel layers (q/k/v) carry different weight global scales, "likely to result
in reduced accuracy". Our build does not - llmcompressor 0.12 links weight
observers across fused groups for a shared global_scale automatically. The
in-house quant is better than the downloaded one on that axis.
Separately: the lora_B inert-adapter gate PASSED on checkpoint-100 - 205/205
non-zero, median norm 0.829, zero vision_tower tensors. That check never ran in
round 1, and it is the only failure mode that stays invisible until the
acceptance gate reports base-identical numbers.
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.
The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.
Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.
Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
stale 365-line one, because training rendered through upstream and the
mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
save_pretrained clamps every prompt forever)
Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
Run-01 was killed at step 19 by operator instruction to root-cause before
spending a ~13.9h window. Two independent methods now agree on where the step
time went, and neither was the hypothesis the consult panel converged on.
Scaling fit (3 points, 2 params, residuals <3ms over an 8x range):
A = 6.87e-4 s/token, B = 8.85e-8 s/token^2
quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384
No fixed term was needed, which refutes launch-bound outright.
Profiler kernel table (device rows only):
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
The attention kernels are sm80 — Ampere-generation CUTLASS running on an
sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the
mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS.
Correctness cleared separately: the sliding mask asserts at max 1024
allowed/row, so the 25 windowed layers were genuinely windowed. The same probe
found that right-padding is what pins the 5 global layers to an explicit 4D
mask and off the is_causal fast path — measured at 9.4% slower for 24% less
loss work at fixed width.
The largest available win is not the attention kernel. The corpus is 29.9%
padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall
clock, no new dependency, unchanged peak memory. Bucket size turned out not to
be a diversity knob — roots per accumulation window are flat across a 256x
range, so the global micro-batch shuffle does that work alone and the bucket
should be tight.
Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic
home (sibling to the quantization playbook), the four probes under
scripts/training-probes/ with raw output kept for re-derivation, and a §6 to
the sizing doc carrying the Gemma-4-specific numbers and round-2 restart
parameters.
Measured negatives recorded so they are not re-chased: grouped_mm (0.9%
slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over
the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever),
FA4 on sm_120.
Round-1 state preserved: 609MB encode cache, order manifest, truncation
report, resume script. No checkpoints — it died at step 19 and the first was
due at 100, so the lora_B inert-adapter gate never ran and moves to the
restart.
Names for fleet hosts so addresses stop needing to be memorised. Built because
IPv6 makes that hopeless — and, more to the point, because v6 addresses are
derived rather than assigned, so they cannot reliably be written down once and
trusted either.
dns/internal.yaml source of truth: 38 hosts + 4 service aliases
scripts/dns-sync.py reconciles AdGuard resolvers against it
stacks/adguard-ana/ the colo's resolver, which did not exist
Naming is <host>.<site>.internal with sites ana/esh/nh3 (operator's call).
.internal is ICANN-reserved for this; .local is reserved for mDNS, which is
why searxng.pfi.local was a collision that merely happened to work.
Same posture as deploy-stack.sh: file is intent, resolvers are derived state,
you see a diff before anything changes. Every name is published to every
resolver, so the site label says where a host IS, not who knows about it.
Two properties that matter:
- Authority is scoped to the ZONE, not the resolver. ESH carries hand-made
esteban.net rewrites predating this; they are read, ignored and preserved.
Resolver-wide authority would have silently deleted them.
- Within .internal it IS authoritative, so UI-added names get removed. That is
the point — one place to look.
Colo gap closed: ana-docker had no resolver at all (hosts went straight to
1.1.1.1). Its AdGuard runs API on 8053 because 8080/3000 were taken, so the
port is carried per-site in the yaml rather than assumed by the script. It
ships with no blocklists — a false positive on a server network breaks
service-to-service calls for no upside.
Auth is a dedicated infra-ops AdGuard user, not the operator's account,
password vaulted at nh3-dev/adguard-infra-ops-password. Pre-change configs
backed up on each resolver. Both resolvers stayed answering across the restart.
searxng.pfi.local -> searxng.ana.internal, with the old Host() kept alongside
so nothing breaks mid-migration. matrix.pfi.local deliberately NOT migrated: a
Matrix server_name is baked into every user id, room id and signing key, so
renaming it rebuilds the homeserver's identity rather than changing a DNS name.
The v6 column is empty and correct — no fleet host has a global v6 address
yet. The file documents why addresses must be pinned statically before they go
in, since a record that silently stops matching is worse than no record.
Adds the rsync --link-dest hourly snapshot job (nh3-dev:~/development ->
nh3-nas, 48-snapshot retention, secrets/build-dirs excluded) that closes the
no-off-box-backup gap exposed by the 2026-07-12 working-dir clobber. Script
mirrors the live ~/.config/dev-backup/dev-backup.sh; runbook covers restore.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
Adds a host-agnostic elway play + fleet driver that stand up an
`infra-ops` system user (dedicated ed25519 key, NOPASSWD sudo with
log_output audit, docker group) so the infra-ops agent completes
DevOps work end-to-end instead of handing sudo steps back to the
operator. Scoped to PFI-owned Linux boxes; tiered (compute/app/
sensitive-infra) with SureFire/corviduo/esh/Synology explicitly
excluded. Validated live on irv-ml1.
YouTube (and a growing set of services) hard-flag datacenter IPs, bot-gating
even public content regardless of cookies/PO-tokens. Origin case: yt-voice-clipper
on irv-ml1 (Irvine colo) — every yt-dlp fetch returned LOGIN_REQUIRED. Confirmed
pure IP reputation: the same public video fetches cleanly (no cookies) once routed
through nh3-dev's residential egress (70.230.226.88).
- scripts/setup-nh3-egress-proxy.sh: idempotent dante (SOCKS5) install + config.
Internal-only ACL (10.100.0.0/16), bound to the WG interface, systemd-managed.
- docs/runbooks/nh3-egress-proxy.md: purpose, usage, security model, caveats.
Reusable fleet egress, not yt-voice-clipper-specific.
Without -n, ssh inherits the surrounding loop's stdin and consumes
the heredoc that feeds $changed / $deleted, silently truncating the
diff output to the first file only.
Wrapper only enumerates one voice directory (settings.voices_dir,
default /app/api/src/voices/v1_0 — inside the container's writable
layer, not bind-mounted). Override via VOICES_DIR=/app/user_voices
(host bind mount) and add a command shim that cp -r's built-ins from
the in-image v1_0 into user_voices on every start. Built-ins re-seed
fresh from the image (so upgrades that add voices propagate); custom
.pt files in user_voices are preserved (cp -r is additive).
Also adds scripts/blend_kokoro_voice.py + a playbook around it that
mirrors the wrapper's request-time voice="a(w)+b(w)" math but writes
the result as a named .pt to user_voices, making it discoverable via
GET /v1/audio/voices and persistent across recreate. Defaults to
athena = af_bella(2)+af_aoede(1) normalized.
The applet outgrew "stack alongside the infra-management workspace" —
it has its own pyproject, multi-tenant deploy story, separate
release cadence, and isn't actually about managing infrastructure.
Lives at https://gitea.phasefinal.com/vh/nevermore now, with
provenance noted in its initial commit.
This commit removes:
stacks/news-digest/ (full stack tree)
playbooks/deploy-news-digest.yaml
scripts/add-digest-user.sh
The existing ana-docker deployment continues running on its baked
local/news-digest:v5 image — nothing changes for the live install
until you choose to redeploy from the new repo. Migration steps
(rename data dir, redeploy, retire old compose dir) are in
nevermore's README.
Updated:
README.md — Current stacks listing now points at the new repo
STATUS.md — milestones entry for the extraction
Mirrors the reusable tooling (scripts/, generic playbook templates,
.gitignore, conventions section of CLAUDE.md) into a new directory
and strips everything fleet-specific: servers/, stacks/, configs/,
fleet-named playbooks (deploy-*, decouple-*), runbooks, status,
host-pinned scripts.
Output is an empty conventionally-organized workspace with fresh git
history, ready to populate with a different fleet. Skeleton
CLAUDE.md / README.md / STATUS.md are written with the new fleet
name baked in but no server table or placement rules pre-populated.
Deliberately does NOT create a remote or push — the user picks the
namespace + name explicitly via tea / git remote add.
scripts/fork-fleet.sh ~/development/acme-prod-management
scripts/fork-fleet.sh /tmp/test-fork test-fleet
Hardcoded crontab → render at container start from
DIGEST_CRON_AM + DIGEST_CRON_PM. Defaults match the original
0800 / 2000 so existing deploys are no-ops.
scripts/add-digest-user.sh learns --am and --pm flags so each
teammate's stack can fire on their hours:
scripts/add-digest-user.sh bob --am "0 6 * * *" --pm "0 17 * * *"
scripts/add-digest-user.sh carol --pm "30 18 * * 1-5" # weekdays only
Standard 5-field cron syntax; busybox crond honors the container's
\$TZ. Removed the now-unused stacks/news-digest/crontab file and
the matching COPY in the Dockerfile.
Two pieces:
1) Multi-tenant onboarding via scripts/add-digest-user.sh
Shared miniflux + per-user digest stack. Onboarding a teammate
takes one command (plus a one-time sudo for dir creation):
scripts/add-digest-user.sh <username>
What the script does:
- Reads miniflux admin creds from ana-docker
- Allocates next free port (scans existing digest-*/.env)
- Generates a random password (or accepts one as 2nd arg)
- Creates the miniflux user via the admin API
- Materializes a per-user .env at /opt/docker/compose/digest-<user>/
(inherits NEWS_DIGEST_TAG from the canonical stack so all
tenants run the same image)
- Brings up `docker compose -p digest-<user> up -d`
- Seeds default world/local feeds in the new user's miniflux
- Triggers a first digest run
compose.yaml now uses ${DIGEST_PROJECT:-news-digest} to namespace
container_name + homepage labels. Default keeps backward-compat
for the singleton install — existing stacks unaffected.
2) Masthead overlap on phone widths
Desktop CSS pinned .masthead-edition to grid-row 1, which collided
with .masthead-brand once the mobile media query collapsed both
to grid-column 1. Result: "MORNING EDITION" badge stacked on top
of the "DAILY DIGEST" hero. Reset grid-row to `auto` for all
three masthead children in the ≤720 px breakpoint so they
auto-flow vertically.
Decision recorded in CLAUDE.md ("Stack tree convention") and memory
(convention_stacks_vs_mirror.md):
stacks/<stack>/ canonical / intent. git-tracked.
deploy-stack.sh reads from here.
stacks-mirror/<host>/<stack>/ snapshot / reality. gitignored.
sync-stacks.sh writes here. Used
for drift inspection only — never
a deploy source.
Bug this fixes: deploy-stack.sh was reading from the mirror, so edits
to stacks/llama-swap/config.yaml never reached ana-ml2. Today's
two new model entries (qwen3.6-35-a3b-heretic + qwen3.6-27b) lived
in the canonical for hours but the deploy reported "in sync" because
the script only diffed mirror vs server.
Changes:
* deploy-stack.sh: source switched from MIRROR_DIR/$HOST/$STACK to
STACKS_DIR/$STACK. Header comment + error message updated.
* sync-stacks.sh: header explicitly identifies its role as drift
detection; documents the diff command for comparing canonical vs
mirror.
* stacks/llama-swap/{config.yaml → conf/config.yaml}: matches the
deploy mapping (conf/ in canonical → /opt/docker/conf/ on host).
* CLAUDE.md: "Stack mirror (pull / push)" section rewritten as
"Stack tree convention (canonical vs mirror)" with the role table
+ workflow rules + diff recipe. Layout diagram updated.
Phase 1 silently aborted right after the 'cuda-drivers candidate' step
— before snapshotting, stopping containers, installing the new driver,
or rebooting. Symptom: phase2 showed driver still at 570.124.06 and
containers with multi-hour uptimes (proof they were never stopped).
Cause: `apt-cache policy cuda-drivers | head -10`. head reads its 10
lines and closes the pipe early; that delivers SIGPIPE to apt-cache,
which under `set -o pipefail` propagates as a non-zero exit, and
`set -e` immediately terminates the remote shell. The same hazard
existed in phase2's `docker ps ... | head -15`.
Fix is one-line per call site: drop the head. apt-cache policy output
for cuda-drivers is short (~10 lines), and docker ps's container list
on irv-ml1 is small enough to print fully.
Phase 1 was unconditionally adding /etc/apt/sources.list.d/nvidia-cuda.list
pointing at /etc/apt/keyrings/nvidia-cuda.gpg. But irv-ml1 already had
the upstream-style /etc/apt/sources.list.d/cuda-debian12-x86_64.list
with /usr/share/keyrings/cuda-archive-keyring.gpg — that's how the
existing 570 driver got installed in the first place. APT then refused
both: 'Conflicting values set for option Signed-By regarding source ...
debian12/x86_64/'.
Now the script:
* removes any orphan nvidia-cuda.{list,gpg} files from a previous
failed run
* greps existing /etc/apt/sources.list.d/*.list for a debian12 cuda
repo entry; if found, trusts and uses it
* only writes a new repo entry if no existing one is found, and
when it does, uses the same upstream paths (cuda-archive-keyring.gpg,
cuda-debian12-x86_64.list) so reruns are stable
Re-run safe — phase1 can be invoked any number of times without
accumulating conflicting configs.
The previous version piped a heredoc to `ssh -t host bash -s` —
which can't work because `-t` requires a real TTY and heredoc puts
stdin in pipe mode. They're mutually exclusive (orientation.md
gotcha, tripped over once already). The remote sudo therefore aborted
with 'a terminal is required to read the password' before doing
anything.
Restructured: the remote half of each phase is built as a tempfile
locally, scp'd to /tmp on irv-ml1, made executable, and invoked via
`ssh -t host /tmp/upgrade-cuda-remote.sh phase1`. Now stdin is the
real TTY (no pipe), sudo prompts work, and the rest of the script
runs as before.
Same three phases, same rollback behavior, same idempotency story.
Just a transport fix for the local→remote half.
Two-phase script for bumping irv-ml1's NVIDIA driver to support
CUDA 12.9 — currently blocked at 12.8 per driver 570.124.06, which
is why the Kokoro GPU image (built against CUDA 12.9) wouldn't
start.
Phase 1 (interactive — sudo prompted once via ssh -t):
* snapshot current nvidia packages to /tmp/nvidia-pre-upgrade.txt
(rollback reference)
* add NVIDIA's CUDA APT repo for Debian 12 with signed key
* stop GPU containers (comfyui, cosyvoice, qwen3-tts, index-tts,
parakeet, kokoro)
* apt install cuda-drivers (currently pulls 580 series)
* reboot
Phase 2 (re-run after host comes back):
* poll for SSH availability
* verify new driver + CUDA version
* restart GPU containers via docker compose up -d
* print next steps (flip Kokoro .env to gpu variant + redeploy)
Includes a rollback subcommand that reinstalls the snapshotted
packages with --allow-downgrades and reboots — for the case where
the new driver regresses on this kernel.
Driver source switched from Debian's nvidia-driver metapackage to
NVIDIA's official cuda-drivers metapackage; bookworm doesn't carry
575+ in its main or backports repos as of 2026-04-25.
Toggles --append-only off on the rest-server via a temporary
docker-compose.override.yaml (canonical compose untouched), runs
resticprofile forget --prune --verbose on each client of that
rest-server, then restores --append-only. The restore is wrapped in
a trap so a partial-failure prune still leaves the rest-server in
its safe configuration.
ANA side is fully automated against ana-docker (5 clients:
ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db).
NH3 side currently prints a manual DSM ceremony — Synology Container
Manager doesn't expose docker on the expected paths and syncuser
sudo isn't NOPASSWD, so the toggle isn't safely scriptable from
this workstation. The instructions cover the same flow in DSM web
UI + interactive ssh on each NH3 client (nh3-docker, nh3-dev,
irv-ml1).
Usage:
scripts/restic-prune.sh ana # ANA only (auto)
scripts/restic-prune.sh nh3 # NH3 instructions
scripts/restic-prune.sh all # both
scripts/restic-prune.sh -h # help
scripts/restic-prune.sh --dry-run ana # show every command
Avoids the "paste the full output" friction. Every elway run now
writes its full streamed output to two files in addition to the
terminal:
/tmp/elway-last.log
always overwritten — the easy "what just happened" target
~/.cache/elway/runs/<ts>-<host>-<playbook>.log
timestamped permanent record; accumulates across runs
Implementation: small _Tee class wraps sys.stdout for the duration
of main() so all `print(...)` calls fan out to the real terminal +
both file handles. Subprocess output already goes through print()
via _stream_process, so the build/healthz/etc. text is captured.
ANSI color codes are kept in the file so colors are preserved in
log viewers that handle them (less -R, modern tail). Strip with
`sed 's/\x1b\[[0-9;]*m//g'` for paste-elsewhere.
New flags:
--log <path> override path; replaces both default destinations
--no-log terminal-only, skip both files
Path of the permanent log is printed at the top of every run so
you know where it landed without remembering the timestamp pattern.
Previously the startup logic ran `probe_sudo()` if any step in the
playbook declared `sudo: true`, regardless of whether that step's
creates:/when: gates would actually let it fire. The result on the
task-board deploy rerun was a spurious password prompt followed by
six SKIPPED lines — the prompt served no purpose.
New flow:
- Remove the upfront probe in main().
- SSHContext.sudo_password defaults to None; new sudo_probed flag
tracks whether we've already prompted this session.
- run_shell_step + run_upload_step call ensure_sudo(ctx) only at
the point a sudo step is actually executing — i.e. after its
skip conditions have been evaluated and passed. Idempotent:
probes at most once per playbook run.
Tradeoff accepted: if the user fat-fingers the password, they see it
mid-run on the first sudo step rather than upfront. `stop_on_fail`
(default true) halts cleanly; they rerun. Lower friction for the
common idempotent-rerun case, same recoverability.
Verified against playbooks/deploy-task-board.yaml — prior run
prompted + completed in 1.7s; new run completes in 1.7s with no
prompt because every sudo step skip-gated.
Tier 1 — pre-step skip conditions:
when: <remote shell expr> skip unless expr exits 0
creates: <remote path> skip if path already exists
removes: <remote path> skip if path is already absent
Any of the three saying "skip" marks the step `skipped` and moves on.
Evaluated under bash -c on the remote so `!`, `[[`, pipes etc. behave
consistently regardless of the default remote shell.
Tier 2 — post-step change detection:
changed_when: <remote shell expr>
Evaluated after a successful step. Exit 0 → step counts as `changed`
(default). Exit != 0 → `ok` (ran, nothing actually different).
Without this field, successful steps default to `changed`, matching
Ansible's shell/command defaults. Useful on verify steps:
`changed_when: "false"` reports them as `ok` since they only attest.
Status model moved from pass/fail to four states:
ok / changed / failed / skipped
Summary reports each count; overall outcome is CHANGED if any step
changed, OK if none did, FAILED on any non-skipped failure.
Rerunnable smoke: playbooks/elway-smoke.yaml now proves it. On a
clean target the cold run reports 4 changed, 3 ok. Rerunning with
the same vars reports 2 skipped / 2 changed (upload + log-record
have no idempotency hooks and are always `changed`). Overriding
--var greeting=... re-runs the gated step exactly as intended.
Doc block at the top of the script updated with the new schema
fields and state machine.
vh