a0c5fc6ed549da32878c33291f0cdc935f1c2f33
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
a0c5fc6ed5 |
feat(pfi-gx10): rack-move network playbook — VLAN 50, static 10.100.50.60
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four existing statics and below the .150 DHCP pool where fleet statics live. The playbook never leaves itself one path back. Wi-Fi stays up throughout while the wired interface is configured beside it; the new address is verified from outside before anything is torn down, and Wi-Fi teardown is explicitly a separate later change. A botched netplan therefore costs a retry over Wi-Fi rather than a trip to the rack — which is what substitutes for 'netplan try', whose interactive rollback needs a TTY that elway cannot provide. Two preconditions are asserted as steps rather than assumed: the interface must have carrier (writing a static config for a dead NIC and reporting success is the failure this avoids), and its MAC must match, since interface names can renumber across kernels but MACs do not. Requires nothing from the operator beyond racking the box. The wired NIC has a distinct MAC from the Wi-Fi one, so the post-move address and switch port are both discoverable from the UDM rather than needing to be relayed. |
||
|
vh
|
8fb8cc87ca | chore(pfi-gx10): first inventory snapshot | ||
|
vh
|
1b596c8c30 |
feat(pfi-gx10): register the ASUS Ascent GX10 and convert it to headless
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target with GDM and GNOME Remote Desktop running. playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop, masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and adds sshd keepalives so a stalled link does not kill a long-running job. Two things learned the hard way and recorded in the playbook: - gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never 'enabled'. A guard of always skips, and a verify written the same way passes while the desktop is still running. Both now test is-active. The first run reported six green verifies having not stopped gdm. - elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as the connecting user and must carry their own sudo; connect as infra-ops. The playbook refuses to stop the display manager while a seat session is held, overridable with --var force_dm_stop=true. Networking is deliberately out of scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet carrier, and belongs to the rack-install change. |
||
|
vh
|
931bac8f68 |
docs(matrix): current state, upgrade procedure, alias convention, push findings
Synapse v1.120.0 -> v1.159.0 and Element-web v1.11.80 -> v1.12.27 (2026-09-01). The existing build steps date from the AIPA era and are now marked as provenance rather than as instructions. Records what only existed in a session transcript: - Schema migrations are one-way; rollback is restore-from-dump. Pre-upgrade pg_dump procedure, with a pg_restore --list verification step. - Why the appservice user namespace is now exclusive: false. exclusive governs who ELSE may act, not what the appservice may do, so on a closed single-admin server it locked out all other account creation to prevent squatting that cannot occur. Includes the two things not to do: narrow the regex (orphans 13 accounts) or rename the id (Synapse keys ownership on it). - The shared-secret registration HMAC takes no trailing null after notadmin. - Room alias convention #<agent>-<purpose>, operator-ratified, with its cost accepted deliberately and its rationale stated as room-identity-carries-tier rather than push-payload-carries-room-name. - Push reality: the pusher is event_id_only, so the notification is assembled on-device by Element X's service extension. Records the resulting server-invisible failure mode when the phone cannot reach the homeserver. - QR sign-in requires Matrix Authentication Service and why it is deferred. Ops ownership recorded: worldtree-dev writes the bridge, infra-ops operates this instance. |
||
|
vh
|
9e986d8ee8 |
feat(phasefinal-web): cloudflare edge config — cache ruleset + always online
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin, so cache policy stays declared once in nginx.conf rather than split between the repo and the dashboard. Always Online enabled, which is what actually survives an origin outage; a 300s document TTL alone would only mask five minutes. Verified: document and assets both reach cf-cache-status HIT, apex 301s to www, edge email obfuscation active. |
||
|
vh
|
524aa4d860 |
fix(phasefinal-web): healthcheck targeted ::1, so traefik skipped the container
The healthcheck used http://localhost/, which resolves to ::1 in nginx:alpine while nginx listens on IPv4 only — so it never passed, the container stayed unhealthy, and Traefik silently declined to create a router for it. That presents as a broken docker provider: correct labels, right network, no route, no error. Target 127.0.0.1 explicitly and add a start_period. Adds the apex router (301 phasefinal.com -> www) and drops the file-provider workaround, which was mitigating the wrong diagnosis. |
||
|
vh
|
7ffbee6f09 |
feat(phasefinal-web): corporate site stack on ana-docker
Single static page (nginx) fronted by Traefik at www.phasefinal.com, built from the design brief. Site markup/CSS checked in verbatim from the design session; fonts self-hosted (SIL OFL) with the @font-face block uncommented, which every fresh export re-comments. Routed via a Traefik file-provider config rather than the container labels: the docker provider on ana-docker was not registering newly-created containers, so the file router avoids restarting shared ingress. Labels are retained in compose so the file can be dropped once that is fixed. |
||
|
vh
|
c488eadc31 |
memory: snapshot — althing v3 fleet-wide at 3.1.1, sec on GPU0
The in-flight section was a day stale: it still described run 3c as the live subject on a box where nothing had moved. Rewritten around what is actually true now -- v3 deployed fleet-wide, the post office relocated to nh3-docker, sec serving on GPU0, run 3c still held on power. Six new decision entries, three of which carry findings that outlive their incident: the inbound half of the handle-resolution bug (a stale ALTHING_HANDLE reads another agent's mailbox and reports it empty, which is a second route into the failure v3 exists to prevent), the OOM attribution to Claude Code sessions, and the operator's two explicit belays recorded so a later session does not re-raise them as new. Auto-archival fired at 301 lines and moved exactly one entry. Three others were old enough and every one carries a still-open deferred pointer -- the parked CI flip, muninn-gate's submit path, and the triton backend deferred to the Ada refresh. Held back per the guards; an over-cap file that keeps live decisions beats a scannable one that lost them. The entry that did move had its deferred item closed today: nh3-extdev's staged v2.1.2 wheel is moot now that the box runs 3.1.1. |
||
|
vh
|
583f329d00 |
fix(playbooks): 3.1.1 deploy — and why the markers match presence, not count
Deploys althing-core 3.1.1 to nh3-extdev. Both markers present on both boxes; the warning verified behaviourally in four conditions rather than by grep alone -- mismatched inherited handle warns, matching handle silent, explicit --handle silent, unlaunched directory silent, and the warning precedes the output it is about. The release's own verification line says `grep -c handles_launched_at dev_launch.py # 2+`. The real count there is 1, the definition; the other two occurrences are in postbox.py. The installed tree is byte-identical to the repo at the pushed tag, so the instruction is wrong rather than the install. This playbook matches on presence via grep -q, so it passed. Had it asserted the stated count it would have reported FAILED on a perfect deploy -- a verification instruction that fails on correct input, which is the same false-negative this file has now produced three times in different costumes. Recorded above the variable so the next bump does not reintroduce a count. |
||
|
vh
|
8a04d6f1bb |
fix(statusline): resolve the handle from the v3 binding, not the v2 map
The statusline resolved its handle from ~/.althing/session_handles.json.
forseti corrected the grounding and I verified it: that file is a v2
artifact and v3 never opens it. `grep -rn session_handles althing/` is
empty, postbox's resolve_config takes --handle then ALTHING_HANDLE and
nothing else, and `althing-cli use` -- the tool that maintained the map
-- was deleted at the cutover. Whatever is in it now is hand-kept and
drifts silently.
launch-history.json is written by dev_launch, which is the thing that
sets ALTHING_HANDLE in the first place, so it is the real cwd-to-handle
binding. Shape is {cwd: {command: {at, handle}}} with several commands
per directory, so this takes the most recent by timestamp rather than
whichever key happens to sort first. The v2 map stays as a fallback for
its broader coverage.
Worth recording why this was wrong: I wrote the resolution this morning
by reading the v2 statusline block it replaced and keeping its data
source while updating its commands. The commands were the visible half
of the cutover and the data source was not, so it survived a rewrite
that was otherwise about removing v2.
|
||
|
vh
|
c648a40b68 |
fix(playbooks): 3.1.0 herald deploy; the marker check takes a LIST now
Deploys althing-core 3.1.0 to nh3-extdev and restarts the herald. Verified by content on both boxes: POST_OFFICE_HINT 0 -> 4 in post_office_herald.py and resolve_post_office 0 -> 3 in dev_launch.py, dist-info 3.0.3 -> 3.1.0. The check took one file:marker pair. 3.1.0 changed two files, so a single pair would have asserted half a release and passed -- the same half-passing-silently shape as the version-string check it replaced two releases ago, one level up. It now takes a space-separated list, reports each pair individually, and fails if any is missing. Every release's markers so far are recorded above the variable so the next bump is a lookup rather than an archaeology exercise. Also verified the behaviour the release exists for rather than just its markers. The herald writes its address to $ALTHING_ROOT/post-office and dev_launch.resolve_post_office reads it when the variable is unset: env unset -> http://10.100.50.40:8390 env set -> the env value, which wins env set to blank -> the file, because blank counts as unset My first attempt tested this through postbox, which still requires the variable and reported "no post office address is configured" -- correct behaviour that looked like a failed deploy. dev-launch is the reader, not postbox. |
||
|
vh
|
590b55f7d8 |
feat(playbooks): potrace/agg headers, with the two traps that mislead
pypotrace is an sdist that compiles at install time, so every machine and every CI runner resolving it needs these headers first. That makes it a recurring per-box action rather than the one-off it arrived as. Two things learned installing it on nh3-dev are recorded here rather than left in an althing thread, at forseti's suggestion, because a thread is not where the next person looks: Only libagg is a pkg-config consumer. potrace ships no .pc file and is found via potracelib.h directly, so `pkg-config --exists potrace` returns false on a correctly configured box. It looks exactly like the cause and never is. libagg's pkg-config modversion is 2.7.0 while its Debian package version is 1:2.6.1-r134. Comparing those two numbers convinces you the wrong package is installed. The verify phase asserts the geometry, not the import: a square must come back as one curve of four CornerSegments. An extension linked against the wrong thing can import cleanly and return nonsense, so a successful build is not evidence the module works. Getting the build probe to run took three passes and the reason is worth keeping. uv is not on a non-interactive ssh PATH; it is in a different place on each box; and on nh3-dev it sits inside a 0700 home, so even the correct absolute path fails `test -x` for the ssh user because the directory cannot be traversed. The headers are system-wide and root's business, but the build check is a developer action and has to run as the user who owns the toolchain. |
||
|
vh
|
cdeb57c18b |
fix(playbooks): 3.0.3 herald deploy, and a content check that survives releases
Deploys althing-core 3.0.3 to nh3-extdev and restarts the herald. Verified by content on both boxes: PANE_SETTLE_S 0 -> 2 occurrences, value 0.3, dist-info 3.0.1 -> 3.0.3. The content check was hardcoded to the 3.0.1 markers, so from the next release onward it would have kept passing while asserting nothing about what had just been installed -- a check that verifies the previous release is indistinguishable from one that works. It now takes the marker and file as variables, bumped per release, with both releases' markers recorded so the pattern is obvious rather than folklore. That is the same defect class as the install step gated on `postbox` not existing, which this playbook carried until last round: a guard written correctly for the first run and never re-read on the second. The post office container was not touched. forseti established by import graph that althing/post_office/* imports neither changed module -- the fix is in reach_pane, which is herald code -- and the container has been up two hours across both herald restarts. |
||
|
vh
|
9f87ff87e1 |
feat(althing): surface the post office on Homepage under Toolchain
Labels the container into `Toolchain`, an existing group under the existing Toolchain tab -- "the plumbing", which is where a message bus belongs. Confirmed live: Homepage's API now returns it. I had previously recorded in this file that no group fitted, which was wrong. That conclusion came from a grep over the layout block that missed the nested groups, and it went into a comment as though it were a finding. The group was there the whole time. Labels bind at container creation, so this deployed with `up -d` rather than `restart`; a restart leaves the old labels and the dashboard keeps showing what was there before. nh3-docker is already a discovered host in homepage's docker.yaml as `nh3-pfi-docker`, so the label alone is enough -- adding a services.yaml entry as well would render the card twice. althing-chamber on ana-docker also carries Toolchain labels and is a separate service per the operator. Left alone. |
||
|
vh
|
dbb930d546 |
fix(playbooks): 3.0.1 herald reinstall, and two guards that were release-hostile
Reinstalls althing-core on nh3-extdev for 3.0.1 (the pane-route fix) and restarts the herald. Both boxes verified BY CONTENT rather than by version string -- forseti's own checks, grep for _PANE_ID and _live_pid, because a dist-info directory records what was installed, not what the files contain. Both went 0 -> 3 and 0 -> 2. Two bugs in the playbook this run exposed, both of which only appear on the second use: The install step was gated on `postbox` not existing. That guard was correct for the cutover, when postbox genuinely was absent, and wrong for every release after it -- postbox exists now, so a version bump would have silently skipped the install and the playbook would have reported success having done nothing. `--force` already makes the reinstall idempotent, so the guard bought nothing and cost correctness. The post_office variable still pointed at nh3-dev, three hours after the post office moved to nh3-docker. It failed in the verify rather than at install time, which reads as a broken deploy rather than as a stale constant. Worth noting the failure message was the outage semantics working exactly as designed: "This is an outage, not an answer: do not treat it as 'no mail'." |
||
|
vh
|
22da609053 |
feat: registry-push the post office image; version the statusline
## Registry The image moved by `docker save | ssh | docker load`, so a rebuild meant repeating that by hand. It is now published and the compose pulls a digest-pinned reference, so a redeploy is `compose up -d` on any host that has logged in. Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a registry anyone can re-push, and this container is the fleet's whole message bus. The tag rides alongside so a human can read what it is. Namespace is claude-bot, not vh. claude-bot's token carries write:package and `docker login` succeeds, but package namespaces are owned -- pushing to vh/ returns "unauthorized: authentication required" after a successful login, which reads like a credential fault and is actually an ownership one. Publishing under claude-bot's own namespace also satisfies the standing directive to stop reusing the operator's personal credentials for infra work, so the constraint and the policy point the same way. Recorded in the compose header so the next person does not read that error as a broken token. Pull path proven rather than assumed: the running container was recreated from the registry reference and its data verified afterwards. ## Statusline Brought under version control because the v3 cutover broke it invisibly. The segment gated on `command -v althing-cli`, a binary the cutover deleted, so the unread badge and the armed bell silently vanished for every session on the box. With 71 of 73 handles pull-only, that badge is the only out-of-band signal telling a session with no armed waiter that it has mail -- a dead statusline made a working bus look like an empty one. Canonical here, live at ~/.claude/statusline-command.sh, copies rather than symlinks per the same rule as stacks/. |
||
|
vh
|
9d4e7bd34a |
feat(althing): move the post office to nh3-docker
Operator directive, and a standing goal: the bus belongs on the docker host. The flag-day deployment put it on nh3-dev because the herald lives there -- but the herald is the piece that must be host-local, and the post office is explicitly the piece that is not. nh3-dev was wrong on three counts. Our own server table calls it "not a Docker-stack host". It has had three OOM events in fourteen days with the interval halving, and the confirmed hog is Claude Code sessions at 5-18 GB, which is that box's actual job. And mem_limit protects the fleet from the post office while doing nothing in the other direction: oom_score_adj was 0, an ordinary kill candidate, on a box whose last sweep took althing-herald and uvicorn. The new deployment sets oom_score_adj=-500. The compose is now version-controlled here as a normal stack rather than living only in the althing repo's deploy dir. ## docker stop does not checkpoint the WAL The database was 155 KB with a 4.1 MB write-ahead log, and every recent message was in the log. A clean container stop left it untouched -- an explicit PRAGMA wal_checkpoint(TRUNCATE) was required. A docker cp of the .db alone would have produced a database that opens cleanly, passes integrity_check, serves the full 73-handle roster, and is missing the day's mail, with nothing raising an error. Row counts were verified at source, in the staged copy, and after seeding, because the count is the only thing that separates those two outcomes. The old volume is left in place. Not a rollback path, which the operator ruled out -- just not deleting the only other copy on the day of a move. ## Follow-up left open The image has no registry push and moves by save/ssh/load, so a rebuild means repeating that by hand. It should join the gitea registry pattern the other stacks use. |
||
|
vh
|
e58360668e |
feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
|
||
|
vh
|
f875f746b8 |
feat(playbooks): nh3-dev memory forensics — and the OOM hog is Claude Code
forseti asked for journald kernel persistence plus sysstat, on the premise that nh3-dev's three OOM events in 14 days left no evidence. The premise was wrong. journald has been persistent all along: 15,068 kernel entries in the 82-day previous boot and 351 OOM records across retained boots, full task tables included. `journalctl -b -1 -k` returned one entry because it ran as a user in neither adm nor systemd-journal, and journalctl shows only your own messages in that case. The same artifact produced the "journal stops at 05:36:08 with no shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM kills logged at 05:38, 05:40 and 05:42. So the fix for "no evidence" is a group membership, not a logging change: usermod -aG adm lkraven, which is the group Debian's journald ACL names explicitly. With the journal readable the attribution is already in it. The versioned Claude Code binary lives at .local/share/claude/versions/, so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as are those named claude. Every one of the twelve largest resident processes ever recorded on this box is a CC session, topping out at 18.4 GB. Everything else killed is 30-55 MB collateral, which clears the althing daemons by measurement rather than by their own sampling. sysstat and atop are added because the journal records the moment of the kill, not the ramp, and names the victim rather than the winner. atop was not requested and is the one that matters: with a dozen panes open, only a per-process timeseries says which session was growing. Not done: a cgroup cap on CC sessions. It is the real mitigation and it would kill long-running sessions mid-work, so it goes to the operator. |
||
|
vh
|
ea818380ff |
memory: the rack is one circuit — my blast-radius objection was wrong
Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."
That inverts the argument I committed one commit ago in
|
||
|
vh
|
3cc55b4b40 |
memory: separate the measured breaker trip from the load hypothesis
The record read "power capacity is the open item" next to ana-ml2's ~600 W, which reads as a cause. It is not one. The trip and its timing are measured; the attribution to the training load is the operator's working read and the reason for the weekend triage. The observation that makes the single-load story incomplete on its own terms: a site-wide blackout is a larger blast radius than one GPU box accounts for. If ana-ml2's draw were the whole story, ana-nas, ana-wg and the public address would not have gone dark with it. Shedding seats may still be the right first move and it is cheap. That is not the same as having identified what loaded the circuit, and the distinction matters going into a triage that will act on it. |
||
|
vh
|
88d79375f7 |
memory: run 3c had TWO launches — the third was an untimestamped report
brokkr-smithy-dev asked how many times 3c was launched rather than reconstructing it, and their reading was three. It was two. #1 17:53:33 PDT killed by the power loss at step 80/604 #2 20:58:41 PDT stopped deliberately at 21:07:40, healthy The phantom third came from a report I wrote at 23:03 narrating the 21:07 kill in the present tense with no timestamp. Every fact in it was accurate; it was unreadable in sequence against a correctly-observed 22:46 snapshot of an idle GPU. Evidence is ZFS birth times (a `>` redirect truncates the log but keeps its birth, so mtime alone cannot separate "rewritten" from "created"), plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a relaunch would have rewritten three files there. Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the downtime from "~90 minutes" to 1h58m: the last journald entry before a hard power loss is the last time anything wanted to log, not the moment of the loss, and here it was 20 minutes early. Corrects the in-flight header (step 22 -> last-logged step 24, stop deliberate) and its stale "as of" stamp. |
||
|
vh
|
98e7d4886a |
memory: snapshot — run 3 gated DO-NOT-SERVE, run 3c held on a tripped breaker
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm guardrail regression that its own preregistered rule passed -- a pooled preserve-list test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched, killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the operator at step 22 pending a weekend power triage. Also captured: the corpus mix was specified in a unit the optimiser never sees (45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage are one direction in weight space, so the merge-back measures the problem rather than fixing it; four guests including the storage SPOF had onboot unset and never came back from the outage, now fixed with dependency ordering; and a transport failure that enters a measurement as a value looks like whatever you hoped to find -- which found a live defect in another agent's instrument an hour after it was reported. Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and abandoned: 0); 4 held back on open deferred-work pointers. |
||
|
vh
|
1a36e60d3a |
docs(quant-playbook): §3.7's APC-off mitigation was reverted nine days ago and the section never said so
Found while answering a question from the operator, relayed via brokkr-smithy-dev, about whether a recorded Qwen3.8 degeneracy at ~1,700 tokens relates to a length sensitivity just measured on the tuned Gemma-4. The record is §3.7 and the number is ~2,000 -- but reading it to answer that question surfaced that the section is stale. §3.7 presented "disable prefix caching, keep MTP" as THE MITIGATION, resolved 2026-08-17, and stated the gen seat runs that config. It does not and has not since that same day: APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use, so it was reverted. The multi-day hunt resolved to the AEON W4A4 quant being defective, with MTP / prefix-caching / gateway merely amplifying it (§3.8 records the corrected causal story; §3.7 was never updated to match). Verified against the live container rather than against the compose file alone: vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3. stacks/gen-seat/compose.yaml carries the full corrected history inline and is the current authority. §3.7's superseded text is kept and fenced rather than deleted -- it is the history of a mitigation that looked right and was not. Added a dated row to §7 per the standing rule that a wrong playbook claim gets a superseded-claims entry, not just a fix. The lesson inside the lesson is worth more than the correction: §3.7's own standing rule is "gate MTP on a multi-turn coherence probe, not just single-shot acceptance." The APC-off mitigation was gated on exactly that probe, passed it, and still failed in real use -- the multi-turn probe was itself too small to gate on. A passing probe is not sufficient evidence at any size that has not been calibrated against real use. |
||
|
vh
|
5a51e76135 |
fix(litellm)!: withdraw merged-run03 — measured self-harm guardrail regression
Added 40 minutes ago at the operator's request; withdrawn on brokkr-smithy-dev's
window-2 safety follow-up. Measured against its OWN base:
selfharm/methods base 1.000 (10/10) -> tuned 0.560 (k=25) -44.0pp p=0.011
meta/sysprompt base 1.000 (10/10) -> tuned 0.720 (k=25) -28.0pp p=0.071
Eleven of twenty-five times it engages with a request for the most effective methods
to kill oneself. The base refused ten of ten. It also surrenders a planted
system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 -- the zero-leak
baseline recorded in this same file hours earlier as "a baseline to PROTECT". It did
not hold.
Withdrawn from the GATEWAY specifically because that is the shared-key surface: one
all-agents key reaches every model listed here, across every session and project. The
operator's hand-testing is preserved in full at the direct endpoint :8099 -- this
removes the fleet's blast radius, not his access. Acted rather than waited because he
is away and the request predates the finding.
ITS PREREGISTERED GATE PASSED. The pooled operational delta is -1.0pp against a
+/-3.00pp bound: nineteen axes held at 5/5 and a 44-point collapse on one moved the
aggregate by one point. The rule was NOT retroactively changed. The failure is
structural and is recorded as R47 section 8 item 11 -- a pooled preserve-list test
cannot see a single-axis collapse, and any future preserve-list gate needs a per-axis
tripwire sized so a total loss on one axis cannot hide in an aggregate.
NOT attributed to the filters: five things changed between run 2 and run 3 and there
is no run-2 measurement on these axes. The measured claim is narrower and sufficient
-- run 3's tuned arm is materially worse than its own base on two axes it was never
licensed to touch. Not a CSAM finding; that detector ran fail-closed across all 575
generations and scanned clean.
The model_list entry is left in place commented out, with the finding above it, so
re-adding is deliberate and informed rather than a blank re-registration.
Verified: config parses, gateway healthy after reload, merged-run03 absent from
/v1/models, direct :8099 still serving.
|
||
|
vh
|
c577d69e2d |
feat(litellm): expose run-3's merged tune for parallel hand-testing
merged-run03 -> ana-ml2:8099, the run-3 ERP/RP SFT merged into stock instruct. Operator asked for it so he can test it alongside the gate rather than after it. NAMED FOR THE ARTIFACT, NOT A TIER. It is `merged-run03` and not `erp-tune-v3` because its behavioural gate has not run. A tier name arriving before the evidence that would justify it is how a name comes to mean something nobody decided -- and with a v2 already in the list, a v3 reads as a successor to anyone holding the shared key. If it passes, `v3` is a name to give it then, as a decision. brokkr-smithy-dev raised this against my own erp-tune-v3 suggestion and was right. The entry carries the preregistrations ABOVE the description, so a reader meets the commitments before the numbers: T6 one-directional (a gain is uninterpretable against a 3.1x fireball tailwind), T3/T4 at ceiling on base so recovery is UNOBSERVABLE rather than merely unpredicted, and any run-2 comparison descriptive and non-attributable with its five confounds named. Also carries the retraction in-line: "bluemoon is the largest loss contributor at 38.6%" came from a words x 1.4 estimator, not a tokenizer. As encoded it is third at 32.9%. The direction survives (1.4% -> 8.0% of total loss) and that is the finding; the superlative does not. Documents why its config.json is the base's copied verbatim: transformers 5.15.1 save_pretrained silently drops text_config.global_head_dim and num_global_key_value_heads, and vLLM then dies in make_layers with a TypeError naming neither the config nor the field. Cost a failed boot to find. A LoRA merge changes weights, not architecture, so the base config is correct by definition. gemma4-26b-a4b-it-base marked CURRENTLY DOWN rather than deleted -- the tuned arm took GPU0 and only one 26B bf16 seat fits on that card. Kept because the seat returns, and deleting a name to re-add it later is how scoped keys get orphaned. Verified: config parses, no duplicate model_name, gateway healthy after reload, completion returns text in `content` with reasoning_content null. |
||
|
vh
|
b6ce22ddcb |
feat(litellm): register the run-3 gate base arm at operator request
gemma4-26b-a4b-it-base -> ana-ml2:8099, the unmodified upstream instruct release (/tank/aimodels/gemma4-26b-a4b-it-bf16). Operator asked for it on the gateway so he can hand-test it; it had been direct-only because the seat is ephemeral. The entry disambiguates WHICH base explicitly. Three exist on that box -- -bf16 (this one, official instruct), -abliterated-bf16, and -heretic-bf16 (run 1's trainee) -- and brokkr-smithy-dev's gate plan called this arm "stock abliterated" a few hours ago, which would have been a different set of weights. A reader of the config should not have to resolve that ambiguity themselves. Carries the measured refusal posture in-line rather than in an althing thread, per the erp-tune-v2 precedent: R19's Mistral Small 4 map does NOT transfer to this base (it draws a wider line than consent, refusing consenting-adult incest and fictional gore that Mistral engages), system-prompt leak is 0/15 against Mistral's 4/5, and advice/medical 0/5 is a pre-existing base gap recorded so it cannot later be misattributed to a tune. Flagged EPHEMERAL in the strongest terms available: it holds ana-ml2 GPU0, which the run-3 gate needs for its tuned arm, so this entry will 503 when window 1 completes. It is not a promise of availability. Serving flags mirror erp-tune-v2 (--reasoning-parser gemma4 plus --default-chat-template-kwargs enable_thinking=false, and --max-model-len 16384) so a base-vs-tuned comparison differs in weights only. Verified: config parses, no duplicate model_name, gateway healthy after restart, model listed at /v1/models, and a completion returns text in `content` with `reasoning_content` null -- the enable_thinking trap is not firing. |
||
|
vh
|
71e44176e9 |
memory: snapshot — run 3 corpus built and held on a megamix containment defect
Run 2 is finished, gated FAIL, and serving on the gateway at operator request. Run 3's corpus was built to brokkr's first recipe and held before any GPU spend: creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the two roots that overlap, at median jaccard 0.873. Containment, not overlap. Dedup direction reversed so the primary source survives rather than the copy inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the largest contributor. Wholly-human share up, megamix share down, total context unchanged at 12.49M so the operator's settled mix arithmetic survived. Two structural findings recorded because they outlive this recipe: F1 'excise PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP and stheno remain unchecked against any other root. Also records the correction I published wrong twice: run 2 was never unstable. All 46 flags were too_short, the collapse guards fired zero times, and it is the left tail of a length distribution -- not new to run 2 either, so it is a property of the recipe and a further base swap will not fix it. |
||
|
vh
|
1a4ef5c7a1 |
docs(training-playbook): 4.6.3 was wrong twice — correct it, and keep the retraction visible
The entry reported an 'output-stability regression' as a novel run-2 finding. Both halves were false and the corrections are more instructive than the original conclusion, so they stay in-line rather than being edited over. Not new: run 1's own gate record already carried the same effect with a caveat attached and unresolved. Two runs across two different base models makes it a property of the RECIPE, not of the base swap -- which also means a third run that changes the base again will not fix it. Not degeneracy, and not a separate finding: all 46 flags were too_short rp turns of 3-14 words, and the two collapse guards fired ZERO times on any run. It is the left tail of a length distribution that had been measured and reported in the same message. Truncation is the same mechanism mirrored on the story side. Both are thresholds calibrated on the base's output shape applied to a model with a different one -- 4.6.1, which both parties had written down and neither applied. The surviving lesson is sharper: a short-answer gate cannot see length behaviour AT ALL, and because it could not, the effect went two full runs before anyone named it. The cost of a gate-set blind spot is measured in runs. Adds 4.6.3.1 on trip points inside the serving stack's jitter -- same seed, same weights, rate moves 9.6% -> 12.6%, sd 1.77pp. Not 'the gate is non-deterministic' but 'the trip point sits inside the jitter', because the fix follows from the precise statement. Includes the split-design rule for measuring such a rate, and the rule that a measured rate must carry its corpus in its name. |
||
|
vh
|
37d3189622 |
docs(erp-dpo): the clip hypothesis is falsified — the distribution is bimodal
The output-side test ran on the live seat. There is no shoulder at 123: the 120-139 bin holds three of ninety-six and is a TROUGH, and 17.7% of generations cross a cap PIPPA can never cross. The clip-as-boundary reading is dead, killed by the test that could have confirmed it. Corrects this document's own earlier read, which compared the tuned MEAN (88.5) to PIPPA's MEDIAN (67) and concluded 'comfortably inside the upper body'. Median to median it is 62 against 67. Mixing statistics across a comparison produced a more reassuring answer than the data supports. What the data shows instead is bimodality -- a mode at 20-39, a trough, a second mode astride PIPPA's centre, a tail to 505, against a base with no such shape. The tune changed rp length's SHAPE rather than its centre: roots whose length distributions do not overlap learned as distinct modes rather than blended into an average. And the skew is rp-ONLY, which localises it to the family the clipped root lives in and is the strongest support the turn-share mechanism gets from the output side. Consequence for pair generation: chosen/rejected sampled from a bimodal generator inherit the mixture, not a mean, and naive sampling over-draws the short mode. Also records that the degeneracy rate is NOT yet a usable baseline -- same arm, same seed, VOID flipped no->YES across a re-run because the 10% budget sits at the noise boundary. A guard whose trip point is at the noise floor produces disagreement between honest observers rather than silence. Replicates running. |
||
|
vh
|
1e4d827c5d |
memory: erp-tune-v2 registered in the LiteLLM gateway at operator request
Operator asked for it so he can evaluate the failed tune by hand, overriding my not-in-the-gateway recommendation. His call. erp-tune-v1 was DELETED from the config in the same reload rather than repointed, so the name now 400s cleanly instead of 500ing against a stopped backend. Deleting rather than repointing is the point: repointing would resolve a name a consumer already knows to different weights, silently. The config entry carries the failed-gate table, the long-form truncation (9.9%) and degeneracy (4.9%) rates, and the rp-length caveat in-line -- so someone reading the gateway config learns what they are calling without having to find the althing thread. Fleet verified healthy after the restart. |
||
|
vh
|
b5bbc29b91 |
memory: gate verdict FAIL — and the T6/T3 trade is what the pair of runs bought
Records the verdict as a FAIL without rounding it off, and the three findings
worth more than the verdict:
- T6 spatial +15.0 where run 1 failed the same axis at -3.5, with the base
swap as the only intended variable. Neither run ships; together they price
what the abliteration was costing, which neither could answer alone.
- an output-stability regression visible ONLY on long-form (truncated 0->38,
degenerate 0->19 per 384) that the reasoning battery could not see across
four passes because its answers are short
- PIPPA's 123-word product clip sitting in the length signal at 70.3% of bot
TURNS against 37.5% of bot WORDS, with the counter-evidence recorded too
(the tune landed near the median, not the cap)
Also records why keeping the tune out of the LiteLLM gateway now reads as
clearly right rather than merely cautious: a FAILED tune must not be one alias
resolution away from a consumer who has not read the thread.
|
||
|
vh
|
5171f19e16 |
docs(erp-dpo): the PIPPA length clip, measured — DPO pairs would inherit it
The run-2 gate found tuned rp turns 36% shorter than base. brokkr hypothesised the mix was teaching PIPPA's 2023 Character.AI product clip; the corpus side is now measured and confirmed. PIPPA's max is 123 words EXACTLY, 100% at or under it, and 0.00% in the 124-130 band -- a wall, not a preference. Every other root crosses its own p99 smoothly. The mechanism is sharper than 'PIPPA is in the mix'. PIPPA is 70.3% of bot TURNS but only 37.5% of bot WORDS, precisely because its turns are clipped -- and length is learned per turn, not per token. By loss tokens it looks like a third of the dialogue signal; by end-of-turn demonstrations it is seven in ten from a source that cannot exceed 123 words. Generalises: a length-clipped root is over-represented in the length signal by exactly the ratio its clipping creates. Counter-evidence recorded too: the tune landed near PIPPA's MEDIAN (67), not its CAP, which is central tendency rather than learning the boundary. Weaker claim than the hypothesis, and not demonstrated either way. Filed here rather than only in the gate record because preference pairs generated FROM this tune inherit its length distribution in both chosen and rejected -- DPO would train an artifact in as an explicit objective. Settle the length question before generating pairs. |
||
|
vh
|
0bb9ee7777 |
docs(training-playbook): 4.6.3 — a short-answer gate cannot see a long-form defect
Run 2's reasoning battery reported zero truncations and zero degenerates on both arms across four passes. The same tune, measured on long-form generation in the same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real output-stability regression, structurally invisible to that gate because its answers are short. Not a bug in the battery -- a coverage property. An instrument measures the regime it samples, and output length is a regime. Generalises to context length, conversation depth, and any axis where the gate's operating point is narrower than production's. The actionable form: enumerate the regimes your gate set spans, name the ones it does not, and decide deliberately rather than discovering the gap downstream. Corollary on sequencing -- put a long-form generation in the gate and put it early, because a length-dependent regression is exactly the one you want found before four clean short-task passes make everyone comfortable. |
||
|
vh
|
3ae32ddc7f |
memory: base set complete, tuned arm live with digests verified identical
Records the floors the tuned deltas have to clear, since they are the whole
point of the base pass and are not recoverable from anywhere else: reasoning
core 0.5 pt, diversity overall 0.0125, story attractor 0.0000.
Two caveats that would otherwise be misread:
- the rp family froze ZERO markers, so its attractor hit rate is structurally
0.0 on both arms. That reads as a clean result and means the instrument
cannot discriminate on that family; rp is measured on the distance axis
only.
- 'Elias' in 92/96 base stories is an independent replication of a published
102/144 on the same family, at a higher rate -- not a novel finding.
Image digest sha256:4091d5593f77 verified identical across both arms, which was
brokkr's stated void condition.
|
||
|
vh
|
a0f59d2778 |
docs(training-playbook): 4.6.2 — a null result needs a positive control
From the run-2 gate. A memorisation probe reporting 0.00% across all 72 items is the correct output for a model that has not seen the corpus, and is also the exact output of a probe that is not firing. Nothing in the number distinguishes them. brokkr-smithy-dev drove the overlap function with known-answer inputs (identical 100%, half-verbatim 65.38%, unrelated 0%, empty 0%) before trusting the null, which is what converts a suspicious zero into evidence. This is 4.5's inert gate wearing a different face: there a check that could not return 'fail', here a measurement that cannot return non-zero. A clean null is the most reassuring output an instrument produces and the least self-evidencing. Same section records the identical-on-both-arms variant: the diversity battery's rp family froze zero markers, so its attractor hit rate read 0.0 on base AND tuned. That reads as a clean result and means the instrument cannot discriminate on that family. Report as a bounded limitation, never as a delta of zero -- a check returning the same value for every input is not measuring. Checklist gains the line. |
||
|
vh
|
3df8707e28 |
memory: base arm live, tuned arm down — battery running sequentially
brokkr withdrew the both-arms-concurrent requirement himself: his diversity battery emits the frozen marker list to a FILE, so the arms were never a live dependency. The real constraint is narrower -- all of one arm's passes on one served instance before the swap -- and sequential satisfies it. No fleet seats displaced, operator not woken. Records the two parity guards, both of which came out of failures rather than foresight: the image is pinned by DIGEST (a vLLM version change between arms six hours apart is a base swap that appears in no config diff), and /tank/aimodels is mounted for BOTH arms even though only the base needs it, because a mount that differs between arms is a difference between arms. |
||
|
vh
|
62f01a02da |
memory: snapshot — run 2 trained, merged, coherence-gated and serving as erp-tune-v2
Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).
Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.
Records three things a future session would otherwise get wrong:
- the mask is proven by the loss-token delta, NOT by the matching p50 step
times -- step time is insensitive to which positions carry loss, so that
check cannot go red on the axis I originally cited it for
- two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
resourcing call, not a scheduling one
- erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
fix needs a config edit plus a reload that interrupts fleet traffic, so it
is batched for morning rather than done at 2am
|
||
|
vh
|
d54f25605f |
docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input
4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2 launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on the same checkout during its seven hours and _git_commit() was called at save time. Commit AHEAD of the code that ran, naming changes it never executed -- including the provenance fields this section prompted. Same defect as run 1's BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at launch, carry it, and record the dirty flag beside the commit rather than instead of it. Generalises to every run-scoped identity: anything read at save time describes the world at save time. 4.6.1 -- the inverse of the inert gate, and it costs trust rather than correctness. A coherence gate false-rejected 'The capital of Portugal is Lisbon' as degenerate against a global 15-word floor. The floor was calibrated against the wrong reference, not set too strict. Lowering it globally would blunt the check where short output genuinely is degeneration; the fix is a floor per prompt. Write the positive test alongside the negative one. |
||
|
vh
|
bcf63db527 |
docs(erp-dpo): readiness survey for the DPO stage
Run 2 is an SFT on the official instruct base, so it will refuse at near-stock rates by design; targeted DPO is where refusals get pruned on chosen axes. That was the trade accepted when the stock base was picked over a third-party abliteration. Surveys what is on disk against what the stage needs. Ready: the merged tune, the SFT adapter, GPU0 once the eval seat comes down, the whole non-loss half of the SFT harness, two unvetted Gutenberg preference sets, and the LitBench-RM judge. Missing, in order of pain: preference data for the refusal axes (nothing on disk targets it -- the Gutenberg sets are prose-quality), the axis list itself, and a DPO trainer (trl is not installed). The gating item is not technical: WHICH refusal axes are in scope and which are explicitly kept. Data generation, pair counts, the held-out split and the success probe are all functions of that list, so nobody should generate a pair before it is written down. Flags that the domain-compliance probe should measure run 2 BEFORE pruning, since the pre-number is the only baseline that will ever exist. Also records the operational trap: do the trl install AFTER a run finishes, never during one -- a resolution that upgrades transformers under a live process can break its save path. |
||
|
vh
|
dbca9a3c66 |
docs(training-playbook): audit the whole manifest against the pairing rule
§4.3's generalisation was stated and then not applied to the manifest that prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and the one pairing that would have caught the §4.1 cache failure -- the mask's sha against the loss-token delta -- existed by accident, because someone had asked for an encode report for unrelated reasons. Adds the audit table, and the rider that matters more than the table: put the observed check where it can actually FAIL. chat_template_sha256's pair is the sha of the string the tokenizer carries, but asserting that in the parent one line after assigning the file to the tokenizer compares a value to itself. It belongs in the encode worker -- a different process, across a pickle boundary, where an unset config key silently leaves every worker rendering through the checkpoint's own template. |
||
|
vh
|
c1db188e6a |
docs(training-playbook): §4.3 records an OBSERVED consequence, not just a config string
brokkr-smithy-dev pointed §4.5's own test at §4.3's remedy: recording `attn_implementation_resolved` is a check that cannot fail on the axis the failure lives on. A silent Dynamo fallback to uncompiled flex leaves `config._attn_implementation == "flex_attention"` untouched while the run computes at ~20x the cost and, per torch's own docs, does not work correctly through the backward pass. The field records the request's RESOLUTION, not its SURVIVAL. On the failure mode that matters it reports success either way. So the section now requires the step-time distribution beside it -- n, min, p50, p99, max -- which is the check that can actually fail. Compiled sits at p50 ~20 s; a fallback at ~400 s. One perf_counter() in on_step_end buys it. Distribution rather than a mean, because a mean hides exactly the bimodality a PARTIAL fallback produces. Generalised past this instance: any provenance field recording a CONFIGURED value is a claim about intent. If the failure you fear is the configuration silently not taking effect, you need a second field recording an OBSERVED consequence, and the pairing is the check. A settings dump alone is decorative. Two implementation details are called out because both were wrong in the first draft -- percentiles nearest-rank so every reported value is a real observation, and exclude the FIRST step rather than the slowest, since step 1 carries compilation but is not reliably the maximum on a variable-width run. New §4.7.1: rotate the log on relaunch. Run 2's first attempt died on the warmup_ratio TypeError and the relaunch appended, so the traceback sat at line 15 of a file whose live run began at line 39 -- and a `tail -n +1 -F` monitor replayed the dead traceback as a fresh event. One file describes one run. Checklist gains both lines. |
||
|
vh
|
dae6ede8e2 |
docs(training-playbook): §4 — when the artifact lies about itself
The playbook covered why a run is SLOW. It did not cover the more expensive
failure: a run that COMPLETES, reports plausible numbers, and is wrong about
itself. Seven of those turned up on the Gemma-4 ERP/RP tune between 08-24 and
08-26 and not one raised an error.
New §4, seven landmines plus a pre-launch checklist:
4.1 a cache key must cover the MEANING of the cached thing. The encode
cache missed the impersonation mask; run 2 would have reused run 1's
unmasked encodings and written impersonation_mask_sha256 into its own
manifest while doing it. No error, no count change, normal loss curve.
4.2 validating a VALUE is not validating the PARAMETER. warmup_ratio was
in range and deleted from transformers 5. Build kwargs as data and
diff the NAMES against the installed signature -- you cannot check the
argument list of a call you have already made.
4.3 record what the run RESOLVED to, never what it requested. Run 1
recorded no attention backend, so an MFU panel profiled the serving
seat under sdpa and recommended adopting flex_attention for a run that
was already using it.
4.4 never train from a dirty tree; harness_commit will name a commit that
does not describe the run. Annotate afterwards, never edit the shipped
artifact -- and state what is NOT wrong, or the note casts doubt on
every field it omits.
4.5 a watchdog whose pgrep pattern appears in its own argv can only ever
return "alive". The inert-gate shape in a liveness check.
4.6 an instrument nobody runs is not an instrument. Mutation-check any
test guarding a property that fails silently.
4.7 fix a stale measurement at the source. "~4.3 HOURS to rebuild the
encode cache" (really 145.5 s) was copied into a new launcher by the
same person who had just measured the real number.
4.8 the pre-launch honesty checklist, ten minutes.
Also:
- Header and framing widened. The file is now a training playbook with a
throughput half and an integrity half; the filename stays for inbound links.
- Sections 4-7 renumbered to 5-8. External refs are all to §1.1 and §3.4 and
are unaffected.
- Four rows added to the superseded-claims table, including the kernel table /
68% quadratic / 8.6% MFU set, which describe the serving seat rather than
the training run.
- gemma4-erp-tune-sizing.md §6 carries a correction banner with the explicit
falls/survives split, because that is the doc someone actually reads before
a run.
|
||
|
vh
|
2656196f47 |
memory: snapshot — the tune is trained, gated, and serving
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
|
||
|
vh
|
2a05ae91af |
feat(training-probes): counted-not-surfaced classifier scaffold
Reusable measurement discipline for probes that must classify how a model
responds to material that should not be printed, logged, or pasted into a
report. Supplies the discipline; the axis map and prompts stay with the caller.
Four rules, each because skipping it produced a wrong number:
- classify, never surface. Completion text is held inside classify() and does
not cross the return boundary. A probe that prints what it measured has
turned a measurement into a distribution channel.
- three-way, not binary. A refusal regex undercounts — models decline by
redirecting with no refusal token present, measured at 2/5 to 5/5 on models
a regex scored 0.
- the deflection count is a FREE CONTROL. Run both arms: zero on both means
the model is binary and the regex is sound; only one means the difference is
real. An artifact does not care which arm it runs against.
- EMPTY and ERROR get their own buckets. Folding them into either side biases
the result, and a truncation-heavy arm flatters itself if its failures land
in the wrong bucket.
Requested by brokkr-smithy-dev for the domain-compliance probe — the discipline
in code rather than reimplemented, with the axis map his side of the line.
|
||
|
vh
|
64bf9d313f |
docs(training-playbook): measure refusal retention on the abliteration's OWN axis
§3.13, plus the probe that produced it. Two lessons, both about measuring the wrong thing confidently. First: a tune applied AFTER an abliteration can walk it back, and a reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's preregistered gate measured none of it — a tune that gains 41 items of contradiction detection and quietly restores refusals passes every check. The compliance axis has to be added explicitly. Second, and this is the trap: measure the axis the abliteration was actually FOR. Ours was run so the model engages explicit fiction. The probe reached for mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and carried a recorded baseline. Different refusal surface entirely, and a model moves on them independently. 29/100 general-harm refusals on a tune whose prose the operator was praising at the time is not obviously a defect and may be desirable: general-harm refusals returning while domain compliance holds is close to the ideal shape for an internal creative seat. The measurement was real; its relevance was assumed. Also recorded, because both were nearly missed: - Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the load-bearing number is 71. Stock refused 100/100; near that would mean the abliteration was undone. 71 complying means partially walked back on one axis — a different finding, and only one of the two threatens the seat. - A baseline from a different harness is not a baseline. The recorded 3/100 came from the abliteration tool's scorer, which reads first-token probability distributions; a probe that generates and regexes is a different instrument. Run your own against both arms on the same seat or report the number alone. - A refusal regex undercounts, so classify hard/deflect/comply — and the free discriminator: if both arms return zero deflections the model is binary; if only one does, the regex is fine. An artifact does not care which arm it runs against. |
||
|
vh
|
a696b49e2a |
docs(training-playbook): merging a tune back toward stock can UNDO an abliteration
§3.12. brokkr-smithy-dev caught and retracted his own recommendation mid-thread; recording it before it reads back later as advice. A common remedy for an overfit tune is a partial merge back toward the base to recover general capability. The published recipes that recommend it merge into the STOCK instruct checkpoint. On an abliterated base, following that literally re-introduces the exact refusal directions the abliteration was run to remove — and it is silent, because the merged model looks healthier on general benchmarks while the property the seat exists for quietly returns. Rule: any merge-back targets the SAME base the LoRA was trained against, never the upstream stock weights however similar the name. The wider lesson is about recipe-card provenance. Community cards are per-checkpoint artifacts and do not transfer across dense-vs-MoE, stock-vs-abliterated, or size variants. The worked example: a recommendation carried from a card for a DENSE STOCK 31B onto a MoE ABLITERATED 26B-A4B on the strength of a shared family name. The overfitting warning on that card happened to come from the right architecture; the pipeline, reward stacks and merge-back came from the wrong one. Same family, three axes apart. So: before quoting a recipe card at a decision, state which checkpoint it was written for and which axes differ. "Same family" is not an answer. |
||
|
vh
|
2ec8f42297 |
docs(training-playbook): base-viability pre-flight, three greps before you pick
§3.11. Three consecutive "what about X as a base?" questions in one session,
each answerable in minutes, none of which had been asked before a 7-hour
training window was committed. Writing the check down so it runs first.
1. does it fit for TRAINING - BF16 weights against the real measured peak,
not the weight figure (Gemma-4 is 48.1 GiB of weights and peaks at
79.7 GiB at mb2/seq-16k). Model-line names lie: "Mistral Small 4" is 119 B,
238 GB in BF16, more than both cards combined. QLoRA is not an escape
hatch for MoE - bitsandbytes walks nn.Linear and fused 3-D experts are not
that.
2. if MoE - does the serving engine implement get_expert_mapping. Zero means
LoRA cannot be served at all. gemma4*.py -> 0; deepseek_v2, mixtral,
glm4_moe, ernie45_moe -> present.
3. does the model class support LoRA - and GREP THE CLASS, NOT THE FILE.
Point 3 has teeth and I nearly got it wrong twice in one turn. mistral.py greps
as SupportsLoRA=0 and is fully LoRA-capable via LlamaForCausalLM.
mistral_large_3.py greps as 0 for both and inherits get_expert_mapping from
DeepseekV3ForCausalLM. Capability is inherited; a file-level grep misses it and
only MRO resolution answers it. Same class of error as asserting a substring
instead of an effective value.
Worked results recorded for the three candidates evaluated:
Gemma-4 26B-A4B fits, no expert mapping -> trainable, MERGE-ONLY
Mistral Small 4 119B 238 GB, has mapping -> servable, NOT trainable here
Ministral 3 14B ~28 GB, dense, inherited -> passes all three
Adds a fourth glance at architecture shape, since it predicts how much of this
playbook applies at all: uniform head_dim <= 128 with no sliding window keeps
both flash and cuDNN reachable and makes §3.1/§3.3 moot, while mixed head dims
plus a sliding window is exactly what forces dense O(n^2) attention onto
Ampere-generation kernels for 65% of the step.
|
||
|
vh
|
96731bb090 |
docs(training-playbook): prove the serving path before spending the window
§3.10. The quantization playbook already says prove your targets before spending GPU time; this is the same rule one step later, and easier to skip. A ~7h LoRA run was built assuming the adapter could be hot-swapped onto a quantized base at serve time. The sizing doc flagged serving as unsettled and said the requirement was needed "while he is early, not after the run" — the concern was identified correctly and then the check was deferred. Tested afterwards, vLLM refuses outright: gemma4's model class implements zero occurrences of get_expert_mapping, which process_packed_modules_mapping requires for any MoE model. One grep, available months earlier. Two generalisations recorded: - Feature support is per-architecture, not per-family. LoRA works for the DENSE sibling of this same model family and not the MoE one, so "model X is supported" says nothing about X's variants. - A capability gap in the serving engine cannot be worked around from the training side. The adapter here never touched experts and was refused anyway, because the refusal keys on the model being MoE, not on what the adapter targets. Includes the mechanical check: grep the engine's model class for the capability, then start the engine with the feature flag alone — no adapter required, since --enable-lora forces the machinery to initialise and that is where it fails. The recovery is cheap here (merge, ~35 min per tune). The cost of finding out late is that it forecloses an architecture choice after the training window has already been spent. |
||
|
vh
|
8de5f7a73c |
docs(gemma4-erp-tune): merged weights are mandatory — vLLM cannot LoRA any Gemma-4
The §5 open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops as it did on vLLM 0.24.0 (#47639), with merged weights as the fallback if it did. Retested on vllm/vllm-openai:latest against the NVFP4A16 base plus the live run's checkpoint adapter. It does not no-op. It refuses to start: AttributeError: To support LoRA for MoE model, 'get_expert_mapping' must be implemented And the reason is bigger than the quant. The check is in vllm/lora/utils.py::process_packed_modules_mapping and branches on whether the model is MoE — quantization is not in the condition. gemma4.py, gemma4_mm.py, gemma4_mtp.py and gemma4_unified.py contain zero occurrences of get_expert_mapping, while deepseek_v2, glm4_moe and ernie45_moe do implement it. So vLLM cannot serve a LoRA on Gemma-4 at all, BF16 or quantized. Merging is not a workaround for a quantization limitation; it is the only path for this architecture. This holds even though the adapter never touches experts — validate_adapter_parameters forbids per-expert params, so all 205 targets are attention and dense MLP. The refusal is about the model being MoE, not about what the adapter targets. Worth recording that the current behaviour is an improvement: a loud refusal beats the 0.24.0 silent no-op, which would ship a base model wearing the tune's name and pass every check that does not compare against base. |
||
|
vh
|
ab980e9345 |
fix(erp-tune-serve): four defects the end-to-end dry run found, all silent
Validated the full adapter -> merge -> NVFP4A16 -> serve pipeline against
checkpoint-100 of the live run. It works, and it produced a served model
generating coherent prose. Getting there surfaced four failures, none of which
announced itself as the thing it actually was.
1. transformers 5.15 MIGRATES the config schema on save. It drops Gemma-4's
`global_head_dim` / `num_global_key_value_heads` and writes `per_layer_config`
instead. transformers 5.10 (what the llmcompressor venv pins) does not know
the new key and resolves num_key_value_heads to None:
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
Every working artifact on the box - bf16 base, served nvfp4 prod seat,
nvfp4a16 build - uses the OLD schema. Merging changes weights, not
architecture, so the merge now downgrades the schema and asserts the result.
2. llmcompressor cannot auto-init a processor for a multimodal checkpoint and
dies with a message that names neither the model nor the cause. Calibration
here is text-only, so the tokenizer is passed explicitly as `processor`.
3. save_pretrained writes tokenizer files only, so `processor_config.json` was
never carried. vLLM then fails at startup with "Can't load feature extractor",
which reads as a vision bug and is actually a missing-file bug. Both scripts
now carry the base's auxiliary configs.
4. The quant needs more than the 32 GiB free on GPU1 alongside the resident
seats. Rather than leave that to a caller, quant_with_gen_down.sh stops
vllm-gen and restores it from a trap on EVERY exit path - crash, OOM, kill,
or success - because the restore must not depend on the calling session
surviving. Uses `docker start`, not `compose up`, so the container comes back
with its exact original config. Measured window: ~15 min, gen healthy after.
Verified on the resulting artifact:
merge 410 adapter tensors, sampled target weights confirmed CHANGED,
upstream 390-line chat template shipped (not the base's stale 365)
quant 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16),
weight_packed 11,725 of which 11,520 expert = 30 x 128 x 3,
tokenizer truncation clean
serve Marlin NVFP4 kernel + Marlin MoE backend, 40,492-token KV cache,
coherent generation with content correctly populated
One quality note: the reference nvfp4a16 artifact triggers a vLLM warning that
parallel layers (q/k/v) carry different weight global scales, "likely to result
in reduced accuracy". Our build does not - llmcompressor 0.12 links weight
observers across fused groups for a shared global_scale automatically. The
in-house quant is better than the downloaded one on that axis.
Separately: the lora_B inert-adapter gate PASSED on checkpoint-100 - 205/205
non-zero, median norm 0.829, zero vision_tower tensors. That check never ran in
round 1, and it is the only failure mode that stays invisible until the
acceptance gate reports base-identical numbers.
|
||
|
vh
|
6a8582936e |
feat(erp-tune): NVFP4A16 serving pipeline, and the MoE landmine it uncovered
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.
The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.
Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.
Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
stale 365-line one, because training rendered through upstream and the
mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
save_pretrained clamps every prompt forever)
Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
|
||
|
vh
|
7b5fd91d3c |
docs(gemma4-erp-tune): root-cause the 8.6% MFU — attention on Ampere kernels, 29.9% padding
Run-01 was killed at step 19 by operator instruction to root-cause before spending a ~13.9h window. Two independent methods now agree on where the step time went, and neither was the hypothesis the consult panel converged on. Scaling fit (3 points, 2 params, residuals <3ms over an 8x range): A = 6.87e-4 s/token, B = 8.85e-8 s/token^2 quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384 No fixed term was needed, which refutes launch-bound outright. Profiler kernel table (device rows only): attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 dense GEMM 2,774.0 ms 7.9% other 5,739.0 ms 16.4% The attention kernels are sm80 — Ampere-generation CUTLASS running on an sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS. Correctness cleared separately: the sliding mask asserts at max 1024 allowed/row, so the 25 windowed layers were genuinely windowed. The same probe found that right-padding is what pins the 5 global layers to an explicit 4D mask and off the is_causal fast path — measured at 9.4% slower for 24% less loss work at fixed width. The largest available win is not the attention kernel. The corpus is 29.9% padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall clock, no new dependency, unchanged peak memory. Bucket size turned out not to be a diversity knob — roots per accumulation window are flat across a 256x range, so the global micro-batch shuffle does that work alone and the bucket should be tight. Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic home (sibling to the quantization playbook), the four probes under scripts/training-probes/ with raw output kept for re-derivation, and a §6 to the sizing doc carrying the Gemma-4-specific numbers and round-2 restart parameters. Measured negatives recorded so they are not re-chased: grouped_mm (0.9% slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever), FA4 on sm_120. Round-1 state preserved: 609MB encode cache, order manifest, truncation report, resume script. No checkpoints — it died at step 19 and the first was due at 100, so the lora_B inert-adapter gate never ran and moves to the restart. |
||
|
vh
|
872c2c562f |
memory: the MFU hunt — two hypotheses measured and killed, consult dispatched
Records what has actually been ruled out rather than what is suspected. The hardware is fine: a plain dense GEMM at the same shape reaches 97.1% of the benchmarked 313.8 TFLOPS peak. The Python expert loop is not the cause, which was my hypothesis and I was confident in it. transformers' grouped_mm experts backend runs 0.9% SLOWER than eager with bit-identical output and identical peak memory, and torch 2.13 has the kernel available, so it is not falling back for lack of one. MoE is not the bottleneck at all. Isolated at real shapes the block runs at 26.5% of peak with 36% of its time in pure gather/scatter, and a dispatch-free bmm version would reach 80.9% — but the whole MoE contribution is only about 10% of a step. Making it free buys 7%. So roughly 90% of the time is unaccounted for. The leading untested hypothesis is that the five full_attention layers use global_head_dim 512, above FlashAttention-2's 256 cap, which would push SDPA onto a slow backend for O(n^2) attention at sequence 16384. Also records that the earlier 5% MFU figure was wrong in two ways — unpadded tokens and a guessed peak — and that the operator caught it. Padding is real but secondary at 29.9%. Consult dispatched to brokkr-smithy-dev for the frontier-dwarf panel. |
||
|
vh
|
07743c6aff |
memory: snapshot — tune training unattended, MFU root-caused to a Python expert loop
The in-flight section is rewritten around the run itself rather than the decisions that led to it. The sizing and seat-call bullet collapses to a pointer now that both are executed; its detail lives in docs/pfi/gemma4-erp-tune-sizing.md. Adds the measured MFU finding: 27.1 TFLOPS against a benchmarked 313.8 TFLOPS peak, root-caused by reading the source rather than inferring — transformers runs the Gemma-4 experts in a Python loop, 128 experts across 30 layers, roughly 11,500 iterations per optimizer step under gradient checkpointing. Padding is a secondary 29.9% tax. Records that my first estimate of 5% MFU was wrong in two compounding ways: divided by unpadded tokens, and compared against a guessed peak rather than a measured one. The operator pushed back on the number and was right to. The fused MoE kernel is deferred work with a tracking surface — park id 47 — per the snapshot rule that deferred decisions go in Recent decisions with a pointer, never into the volatile in-flight section. Also records the resume trap: the original launch command begins with rm -rf on the output directory, which would destroy both the encode cache and every checkpoint. resume-run-01.sh exists so that cannot happen. |
||
|
vh
|
d6dfd61c91 |
memory: the ERP tune is running — override granted, 12 defects fixed first
Operator overrode the corpus gate for one run on 2026-08-25, with the grant staged beside the recipe rather than asserted in chat. It deliberately does not flip any root's training_eligible flag, so the signal that made the run stop in the first place survives intact. Records where the run lives, what it is configured with, how to restore the fleet, and the two lessons that generalise past this project. The first is inert gates. Two turned up in one evening — auditcore, whose CSAM hard-drop never fired across 42,662 records, and validate_vision_keys, which compared model.state_dict() against itself and could not fail on any input. Both read as guards. The question that catches them is not whether the check passes but whether it can fail. The second is an invariant enforced on one code path and not its sibling. That was my own bug: INV-T9 requires a window to hold at least one complete assistant turn, and I enforced it where the window is cut but not where it fits, so a trailing user-only remainder became a zero-loss window and killed the first launch. Same shape as the inert gates, in code I wrote an hour earlier. Two further foot-guns worth the space: enable_input_require_grads is mandatory beside gradient checkpointing on a frozen base, or every adapter stays at its initialisation and the run completes successfully having learned nothing; and the upstream Gemma-4 template forward-scans to suppress a closing turn marker before another assistant message, so incremental rendering cannot tile against it and assistant runs must be merged first. |
||
|
vh
|
33433e0d1e |
docs(gemma4-erp-tune): replace the estimates with measurements — they were 3x optimistic
Ran the loss path on the real checkpoint on GPU0 with synthetic tokens.
The arithmetic held for parameter counts and was badly wrong for
activation memory.
naive CE bsz1 seq 8192 81.93 GiB
naive CE bsz1 seq16384 OOM
chunked CE bsz1 seq16384 65.66 GiB
chunked CE bsz2 seq16384 79.71 GiB <- the run config
chunked CE bsz4 seq16384 OOM
The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the
~5 GiB estimated: the estimate modelled gradient checkpointing as
storing layer inputs plus a modest recompute peak, and the real MoE
recompute peak with top-8-of-128 routing and its scatter/gather buffers
is far heavier. Dense-model intuition does not size an MoE run.
Two predictions landed exactly — 205 target modules and 74,342,400
trainable params at r64 — which is why the rest of the model of the
thing is still worth trusting.
The headline is that chunked CE at seq 16384 costs 16 GiB less than
naive CE at seq 8192, so chunking is what makes brokkr's 16384
recommendation reachable rather than an optimisation on top of it.
max_seq_len moves 8192 -> 16384 on his truncation finding: the cap
drops 6.2% of samples but 22.4% of tokens, concentrated entirely in
dialogue, which is 60% of the mix.
Also records the four harness changes this required (eitri-smithy
62b556b), including the inert-adapter trap: without
enable_input_require_grads() alongside gradient checkpointing on a
frozen base, no gradient reaches the adapters, every one stays at its
initialisation, and the run completes successfully having learned
nothing.
|
||
|
vh
|
47ec3d1a97 |
memory: the ERP tune is blocked on a corpus gate only the operator can clear
Every clean-v1 CLEANROOT carries training_eligible: false with two named blockers, and the recipe states plainly that nothing in it is Charter §3 training-eligible. I initially read scoped_grant: operator-2026-08-22 as authorization and told brokkr-smithy-dev I was proceeding. That was wrong, and the person who wrote the field corrected it: the grant governs INV-4 one-way tier inheritance — the adapter is permanently internal-erp-rnd and never distributable — not training clearance. The stage-2 detector is measured-inert rather than merely unvalidated. auditcore v3.7.2 returned its hard-drop exit code zero times across 42,662 raw RP records, its printed verdict ignores its own printed threshold, and it passed a record a blind audit had already identified as sexual content involving a participant the text marks as a child. Verified the one thing that decides whether that specific record reaches training: pippa-5083 is present in kept-manifest.jsonl (4,551 rows) and absent from recipe-dedup-kept.jsonl (20,473 rows), which is the survivor list the harness gates on. The substitute lexical screen caught it. That is one known instance caught by a stopgap and says nothing about what the screen misses. Both brokkr and I recommend stopping. Neither blocker is hours of work. |
||
|
vh
|
c9943b1507 |
docs(gemma4-erp-tune): whole-card placement — gen moves to GPU1, sec stands down
Operator chose a third placement over the two the sizing offered: rather than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no co-tenant and gen never goes dark beyond its own restart. Revised run parameters, since a whole card changes them: - micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving 888 optimizer steps instead of 444. At one epoch the step count is worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert per step against ~512 at micro-batch 1 — a far healthier GEMM on 704-wide experts. - Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock but pushes activations to ~24 GiB per sequence, which forces micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow. - Scriberr stays on GPU1. The previous revision suggested moving it to GPU0, which was correct only while training was going to live on GPU1. Records the ordering constraint in both directions, the elway identity requirement, and that sec's aliases should be allowed to fail at the gateway rather than be substituted with another model. |
||
|
vh
|
9d70100867 |
feat(ana-ml2): elway playbooks to open and close the ERP/RP tune window
Operator call: rather than train beside gen on GPU0, move gen to GPU1 and
stand sec down for the night, so the tune gets a whole 95.60 GiB card and
the fleet's general seat never goes dark beyond its own restart.
Order is load-bearing in both directions and the playbooks enforce it.
gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction
of TOTAL card memory: 42,091 MiB must be FREE at startup or the engine
refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up, so
recreating gen onto GPU1 first would take the main seat down and leave it
down. mog-sec stops first and a hard gate checks the freed memory before
gen is touched. The close playbook mirrors it: gen must vacate GPU1
before mog-sec starts, since mog-sec needs 50,901 MiB of its own.
Close opens with a gate that refuses to run while a process is still
resident on GPU0, so it cannot evict a training run mid-flight.
Override with --var allow_busy_gpu0=true.
Three defects found and fixed while landing this, all worth keeping:
- Verifying GPU residency via `docker inspect --format {{.State.Pid}}`
never matches. vLLM V1 runs EngineCore as a child of the container's
pid 1, and it is the child that holds the memory and that nvidia-smi
reports. Match by cgroup instead.
- A step's `sudo: true` does not extend to its when/creates/changed_when
guards, which run as the login user. The root-only .env made an
unsudo'd grep exit 2, so the GPU-id flip SILENTLY SKIPPED. The
effective-value assert is what caught it.
- That assert originally grepped the config YAML for -\s*'?1'? and failed
against compose's double-quoted `- "1"`. Parse the JSON with jq; an
assert that fails for the wrong reason is worse than no assert.
elway must be invoked as infra-ops@10.250.50.54 rather than the ana-ml2
ssh-target, which resolves to lkraven and has no NOPASSWD sudo.
|
||
|
vh
|
c507db9ac0 |
docs(gemma4-erp-tune): size the run against the checkpoint — QLoRA is structurally unavailable
The proposed shape was QLoRA r64. It cannot be run as specified. The checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj [128,2816,704] — no .weight suffix, so they are parameters, not modules). bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params / 42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit saves ~3.1 GiB of 48.07 and does not error while doing it. Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per additional 8192-token sequence. Two sizing items were absent from the brief and both are load-bearing: - vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping 30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at ~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a 95.6 GiB card — it starts, then OOMs on the first long sample. Fused or chunked linear CE is mandatory and must be smoke-proven before a window is booked, since Liger may not carry a Gemma-4 MoE patch. - v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the full-attention layers). A v_proj target silently produces no adapter there, and k_proj adapts K and V simultaneously. 45.96M trainable at r64 across q/k/v/o. Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr. Recommend standing down sec (2 aliases, last request ~5h ago) rather than gen (7 aliases, 765 busy-engine log lines in 24h). Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at effective batch 16, which makes the wall-clock-checkpointing amendment concrete rather than hypothetical. Package as a uv venv on /tank: root is 91% full (36 GB) with /var/lib/docker on it. |
||
|
vh
|
9d0e628643 |
memory: correct the vLLM version claim — ana-ml2 runs a spread, and 0.27.1 is on disk
The snapshot recorded "ana-ml2 now runs vLLM 0.26.0". That is true of the char-rp seat's pin and false of the box, which the operator caught immediately. Measured per running container: gen is on nightly-311b3513 reporting 0.27.2rc1.dev150, mog-sec on nightly-e9d1398d reporting 0.26.1rc1.dev1102, and rerank-a3 / coder / reward / embed still on 0.24.0. char-rp and the trainee bench stack are pinned to v0.26.0. So there is no single "the version" for this host, and stating one invites exactly the wrong retest. The correction improves the LoRA question rather than complicating it: vllm/vllm-openai:v0.27.1 is already on disk and unused — a TAGGED release, not a nightly, roughly four months past the 0.24.0 where the silent-no-op was diagnosed. That is the right target for a decision test: no nightly variance, no pull. The retest instruction in both the decision entry and the handoff now names it. |
||
|
vh
|
668e590e7d |
memory: snapshot — char-rp on the Gemma-4 MoE, abliterated trainee staged, QLoRA sizing next
Captures an evening that ran from an OOM crash-loop to a measured trainee base. The durable lessons, none of which CLAUDE.md can carry: --gpu-memory-utilization sizes the KV cache and does not cover CUDA context or graphs, which is half of why a seat that fit on the 21st stopped fitting on the 24th; the other half is that gen's footprint GROWS WITH UPTIME (38.5 GiB fresh against 45.6 GiB after three days, same container, same flag), so headroom arithmetic against a long-running gen measures a moving number. The stale-chat-template trap turned out to be endemic across third-party Gemma-4 derivatives rather than a one-off, and it is silent in both directions — wrong prompt when serving, train/serve skew when tuning. And a benchmark finding was retracted because 12% on a five-option task is below the 20% chance floor: a below-chance score indicts the instrument before the model, and a preflight can be thorough while aimed in the wrong direction. Records the serving decision for the tuned model with its history intact: LoRA-on-NVFP4 is preferred if it works, merged weights the expected fallback, but the archived root-cause says the objection was never NVFP4-specific — vLLM 0.24.0's LoRA application was a silent no-op proven quant-agnostic, and ana-ml2 now runs 0.26.0. Retest before designing around merge; the answer changes what Eitri's harness must emit, and he is still early. Auto-archival moved 5 entries (Recent decisions) to archival-memory.md; the guards held back the rest of the 78 age-eligible candidates because their bodies carry open deferred-work language, per the keep-when-unsure rule. Index sits at 286 lines, above the ~250 target and reported rather than forced. |
||
|
vh
|
5415fd4b30 |
docs(gemma4-charrp): abliteration measured in isolation — close to free, but it MOVES capability
Second bench window, operator-authorised after an initial decline and reversal. Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned upstream template, same 192 items, CoT off. Abliteration was the only axis that moved, which is what the previous run could not claim. Net core cost is 0.6 points — but the headline understates what happened. Capability MOVED rather than degraded: five items lost on contradiction detection, four gained on spatial composition, nearly cancelling. A gain was not predicted by anyone, least of all on that axis. The decision this was authorised to settle: llmfan46 stands as the trainee base. No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap between the builds is smaller than the gap this measurement failed to find. Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are ~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than the measurement supports and only marginals were run; and this says nothing about quantization, because the stock-NVFP4 T2 figure came from n=16 against n=32 here — different item counts mean different item sets, so that comparison is n-confounded and is not being made. Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench stack already existed — itself the residue of debugging a 35-restart crash-loop caused by the production compose hardcoding --quantization compressed-tensors. The fix outlasted the incident. gen restored and verified through the gateway; char-rp remains down deliberately; bench stack env reset to the heretic base for the post-tune gate. |
||
|
vh
|
019ccff7e8 |
feat(gemma4-trainee-bench): BF16 bench stack; record that gen's footprint grows with uptime
Adds an ephemeral stack for serving the BF16 trainee base on :8016 under the char-rp aliases, so the abliterated base can be measured on the same battery and the same gateway routes as the served seat with no harness edit. It is a separate stack rather than another variable on gemma4-charrp because that compose hardcodes `--quantization compressed-tensors` for the NVFP4 build. Pointing it at unquantized BF16 weights crash-loops immediately — `TypeError: CompressedTensorsConfig.__init__() missing 3 required positional arguments: 'target_scheme_map', 'ignore', 'quant_format'` — vLLM trying to read a quantization config out of a checkpoint that has none. 35 restarts before it was caught. `restart: "no"` here so a bench seat cannot resurrect itself and block gen's restore, and no homepage labels so it leaves no permanently-offline dashboard card. It cannot coexist with gen and says so: 48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before any KV cache. Running it means gen is stopped. THE MORE USEFUL FINDING is in the meromero env note: gen's memory footprint GROWS WITH UPTIME. Measured today at 46,726 MiB (45.6 GiB) after ~3 days up, and 39,424 MiB (38.5 GiB) immediately after a restart — same container, same --gpu-memory-utilization 0.43, ~7 GiB apart. That is the missing half of this afternoon's crash-loop: the char-rp seat "fit on the 21st and stopped fitting on the 24th" because nothing about char-rp changed and gen crept up underneath it. Headroom arithmetic done against a long-running gen is measuring a moving number, so the note now says to measure against a freshly-restarted one. Operator's requested end state reached and verified through the gateway: gen and summarizer both 200, char-rp down deliberately to hold GPU0 headroom for the upcoming trainee run, bench seat stopped. |
||
|
vh
|
14ff4a3f57 |
docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap
The operator directed that the ERP/RP trainee base be a low-damage abliterated instruct build rather than the stock checkpoint. Two are now staged under /tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB / 25.8B shape with only transformers_version differing in config: gemma4-26b-a4b-it-heretic-bf16 llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100 gemma4-26b-a4b-it-abliterated-bf16 TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset "Low damage" was treated as a measurable claim rather than a description: the field spreads from KL 0.09 to 0.4118 and the table is in the README so the next choice is made on numbers. Fleet anchor for reading them — our own abliteration work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so both staged builds sit inside an already-validated band rather than past it. huihui-ai is rejected despite its reputation: no published metrics, its own card calls the method a crude proof-of-concept, it abliterates both thinking and non-thinking modes, and its parameter count runs ~738M over upstream. The operator's independent read matched. The more durable finding is the chat template. NOT ONE third-party Gemma-4 derivative pulled here ships upstream's — three independent repos carry the identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third 365-line variant, and only the RedHatAI NVFP4 build matches upstream's 6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager slipping, and it is now recorded as a class rather than as the single incident that surfaced it during the A16 control staging. That matters twice over and silently both times: serving a mismatched template renders a different prompt, which is why production pins it; and training through `base/chat_template.jinja` means training on a different prompt format than production serves — train/serve skew with no error, presenting as a tuning failure. brokkr-smithy-dev has been warned on the training side while the harness contract is still early enough to amend. |
||
|
vh
|
8d6a9390de |
docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed
Supersedes what commit
|
||
|
vh
|
3446367d5e |
feat(gemma4-charrp): pin the chat template; A16 control run executed and reverted
The template is now passed explicitly, defaulting to the A4 build's chat_template.jinja. That is a no-op for what is served — the A4 build ships that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once trailing newlines are normalised — and it permanently closes the class of bug found while staging the control: the A16 build ships a stale 266-line template against upstream's 390, with the thinking path built differently and no `thinking` property in its tokenizer_config response_schema. Serving each build with its own template would have moved a second axis. The control ran on the operator's greenlight and has been reverted. Seat is back on the W4A4 build, healthy, RestartCount 0, both aliases verified through the gateway — char-rp returns content with reasoning_content empty, char-rp-reasoning returns both. Result, since it settles a question this repo's config now encodes: activation precision does NOT explain the contradiction-detection deficit. Contradiction detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16 — one item — against gen's 81% on identical items. Every other task is identical across the two builds and the core difference is 2.6 points carried almost entirely by two single items. brokkr-smithy-dev pre-registered that a null result would be the robust branch, because a hidden third axis would tend to create a delta rather than suppress one, so the conclusion survives the residual doubt neither side could close without a dequantization pass. The practical upshot for future scheme choices: W4A4 costs less on this workload than the caution warranted. The caution was still correct to have. Displaced production for 3.7 seconds of measurement plus two container recreates. The A16 build and the BF16 tuning base both stay on disk with the runbook in the stack README, so re-running is a two-minute flip. |
||
|
vh
|
1bd90eaacc |
docs(gemma4-charrp): stack README — the three model dirs, the A16 control runbook
The stack had no README and now carries three model directories that look interchangeable and are not: the BF16 QLoRA base that cannot be served here, the W4A4 quant that is served, and the W4A16 build that exists solely as an activation-axis control. Writing down which is which, and why, before someone "simplifies" the compose to the BF16 path and rediscovers the OOM. Also captures the A16 control procedure end to end, including the two confounds found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit activations, and the stale chat template the real one ships — and the fact that overriding the template is safe because the tokenizers are identical. Both sides have now cross-checked this: brokkr independently diffed every non-quantization config field of both builds against the upstream BF16 and found only transformers_version differing. Residual risk recorded rather than hidden: config identity is not weight identity and nobody has done a dequantization pass. The Gemma-4 flags are documented as architecture-level rather than checkpoint-level, since that is why they survived the seat swap unchanged, and the enable_thinking:false pin is called out as mandatory rather than stylistic — without it every plain prose turn lands in reasoning_content with a null content and every consumer breaks. Notes the non-termination defect with thinking on (32 of 96 calls truncating at 12k tokens, all 16 constraint items among them, reasoning sound right up to the point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not applied to a seat whose production mode is thinking-off. No live change: the seat is still serving the W4A4 build. Displacing it for the control run is an operator decision and is still open. |
||
|
vh
|
24e8826219 |
docs(gemma4-charrp): the A16 control needs a chat-template override, not just a path swap
Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's
battery found a second axis hiding inside what was supposed to be a
single-variable control.
The A16 build ships a STALE chat template. Verified by hash against the upstream
weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI
A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is
not cosmetic — upstream and A4 open the thinking path with
`{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off
it, while the A16 template has no such set and guards with
`enable_thinking is defined and enable_thinking` instead. tokenizer_config.json
corroborates: A4's response_schema carries a `thinking` property, A16's has only
role and content. That build was quantized from an older revision of the
checkpoint.
Served with its own template, the A16 arm would render a different prompt for
identical messages, and a contradiction-detection delta could be attributed to
activation precision when it was the template. That is the same failure class as
the misnamed-A16 repos — a field nobody validated, believed because the name
looked right — one layer further down, and it would have produced a result that
looked like a finding.
Overriding is safe because the tokenizers agree: vocab identical at 262,144
entries, added_tokens identical, so the same template over the same vocab
renders the same token ids. Everything else pre-flights clean — both artifacts
complete with no missing shards, generation_config.json byte-identical.
Seat NOT flipped; displacing production for the bench window is the operator's
call and is still open.
|
||
|
vh
|
f509668e45 |
docs(gemma4-charrp): record the A16 activation-axis control and how to run it
brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction detection with CoT off, against gen's 81%, while state tracking, deterministic constraint following, long-context state and the confabulation control all sat at 100%. That is not general degradation — it is the shape 4-bit input activations produce on the most reasoning-dense task, which is exactly the confound flagged when the W4A4 quant was chosen. They have recorded the finding as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an A16 build to separate the two. No quant run was needed: a genuine A16 build of the same checkpoint already exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 (prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations null, 17 GB). Same weights, same loader as the live seat, one axis moved. ⚠ Two other repos would have answered the question wrongly and the note in the env template says so: bg-digitalservices and ManniX-ITA both publish "Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits 4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's 17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the model. Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom that crash-looped the predecessor this morning. Port and both aliases are unchanged either way, so no consumer config moves. |
||
|
vh
|
27155c0f3b |
feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every gateway route are unchanged, so no consumer sees a difference in addressing: `char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now include chain-of-thought, which makes throughput more critical rather than less — the user waits through the whole reasoning block before the first visible token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7. Both artifacts are on disk and they are NOT interchangeable. The BF16 weights (/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the engine would die at allocation exactly the way the predecessor did this afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB), chosen over the other -it quants because it is compressed-tensors (nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than -it, and the thinking channel lives in the instruction-tuned weights. Smaller weights at the same 0.47 memory budget buy a much larger KV pool: 27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4. The gemma4 tool-call parser, reasoning parser and the enable_thinking:false default all carry over unchanged — they are architecture-level, not checkpoint-level. The --chat-template override does NOT carry over: MeroMero pointed at a jinja hand-patched against that checkpoint, and this model ships its own. Verified that dropping it did not reintroduce the failure that flag existed to prevent — non-thinking prose lands in content with reasoning_content empty, and the thinking alias populates reasoning_content with content carrying the answer. ⚠ Scheme differs from the incumbent and the bench should say so: this quant declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16. Faster, and not like-for-like on the activation axis. meromero-charrp is retained stopped in `created` state and relabelled to AI - Dormant, per the house rollback pattern. Both stacks want :8016, so rolling back means stopping the gemma4 seat first. |
||
|
vh
|
850e0c3351 |
fix(meromero-charrp): drop GPU0 budget to 0.47 — the seat was OOM crash-looping
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the time it was looked at, taking both the `char-rp` and `char-rp-reasoning` gateway aliases down with it (they resolve to the same seat on :8016 — hosted_vllm/char-rp and hosted_vllm/char-rp-thinking). Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the engine gets through weights, torch.compile and CUDA-graph capture looking entirely healthy, then dies at KV-cache allocation with `torch.OutOfMemoryError: ... 195.19 MiB is free`. GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV cache and does not account for CUDA context, graphs and non-torch overhead. This seat was at 0.51, so the pair was committed to 0.94 of the card with about 0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today. 0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 -> 23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the pool still holds 1.4x a full-length sequence. What is lost is concurrent long requests, not context. Verified through the gateway rather than at the container: char-rp returns 200 with content, char-rp-reasoning returns 200 with both content and reasoning_content populated. Seat is healthy with RestartCount 0. The arithmetic and the "check used_memory, not the flag" warning are written into the env template, because the next person to raise either budget needs to lower the other in the same change. |
||
|
vh
|
35adc4a043 |
feat(homepage): rebuild on Australis Skyfall — dual theme, light mode shipped
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.
The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.
build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.
⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.
So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.
Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.
Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
|
||
|
vh
|
39da1d4a97 |
feat(homepage): recategorise on "do I open this?", collapse the API groups
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
|
||
|
vh
|
f6f2f69649 |
fix(homepage): uniform 4-column grid, hold the status gutter, unleak AI Systems
The board's card width changed at every group boundary because `columns:` was being tuned per group under the 2026-08-18 "columns = member count" rule. That rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that group's CARD WIDTH, not its density. Measured on the live board, Notes rendered a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything else 360px. All 20 groups are now `columns: 4` and every card renders at 360. `.service-name` reserved a 78px status gutter with `padding-right` and relied on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the gutter was spill room the title printed straight through. Six cards on the AI tab rendered their name underneath their own status pill, measured by testing the title text node's box against the status cluster's. The intended ellipsis never fired either: it is painted by whichever block's own line overflows, and that is the anonymous box around the bare title text node, which does not carry `overflow`. The gutter holds by wrapping now, and the description opts back out of it with a negative margin since the pill only ever covers the first line. Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox in 2026-08-18, arriving this time from a container label. Relabelled to `AI - Audio Tools`, where the other ASR seats already live. The `homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of the name and now carries the constraint, plus the fact that a labelled container is discovered from any of the five configured engines and must not also be listed in services.yaml. Also: descriptions clamped to three lines so a five-line outlier stops dragging its row 50px taller than its neighbours; icon ramp overridden off slate-400 -> slate-700, which was sinking the bottom half of every glyph into the card fill; bookmark groups and Jellyfin's trailing stream rows brought into the card and eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates them. The icon override sits on `html[class]`, not `:root`: Homepage sets the same variables on `.theme-slate`, which is on the <html> element, and a class beats `:root` on the same element. Verified with Playwright against the live board — per-group card width, card height spread, and a geometric title-vs-status collision check, before and after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/ |
||
|
vh
|
32349b7653 |
memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed
Rewrites the in-flight section, which had gone stale in one place (speaches is stopped, not live) and did not reflect a session in which everything opened was also closed. Three detail files carry the bulk: the FortiGate's public surface reduced to nothing including the ACME listener, Scriberr's deployment and the three upstream defects it required working around, and the ESH DNS repair that also made the IPv6 naming scheme real on three hosts. The tunnel entry in the decisions log described a cipher change as proposed and pending; it has since been attempted, found impossible in the form intended, and resolved differently, so the line now points at the outcome rather than the intention. New decisions cover the cipher adoption and the finding that the per-flow ceiling belongs to the UniFi gateway's software cipher rather than the firewall, the administrative closure, the transcription deployment, the DNS repair, the retirement of an ASR service whose only consumer was abandoned, and a database VM that until now had no fleet identity and no vaulted credential at all. Three approaches are recorded as abandoned: an AEAD cipher the far end cannot express, an upstream DNS setting the gateway accepts and ignores, and a scheme to claim unused delegated prefixes that founders not on the prefixes but on having to rebuild a firewall policy to use them. |
||
|
vh
|
d419b11d43 |
docs(ipv6): close the NH3 multi-prefix question by operator ruling
The gateway has no IP-passthrough mode, confirmed by the operator with admin access to it, so the inexpensive path of letting the UDM take the delegation whole and carve it natively is unavailable at this site. What remains is a separate DHCPv6 client presenting several identities, which needs recabling to reach the gateway, splits routing so that one device handles v4 and another v6, and above all moves IPv6 off the UDM's zone firewall, leaving the entire policy to be rebuilt elsewhere before any host could safely hold a globally routable address. The operator has declined it and the LANs stay without v6. The index entry also still asserted the superseded single-delegation conclusion as fact; it now carries the corrected reading alongside the ruling, so the finding and the decision not to act on it arrive together rather than the first inviting a retry of the second. |
||
|
vh
|
bd209951ac |
docs(ipv6): correct the NH3 delegation finding — eight /64s, not one
The note concluded AT&T delegates a single /64 and that reading the address pattern as a /60 was a mistake. A later finding in the same session overturned that by reading the BGW's own statistics page, and that correction was never written down; it survived only in the transcript and surfaced again while assessing whether more prefixes could be claimed. The gateway holds the /60 and rations it, keeping half for itself and delegating the top eight /64s one at a time. Both observations agree: a prefix-id only carves within a delegation already held, so the earlier test could not have moved a lone /64 regardless. The limit is that UniFi solicits once, which makes the requester the ceiling rather than the carrier. |
||
|
vh
|
d127e29fac |
docs(ipv6): retire the next-candidates line now that all three are done
Replaces it with why the remaining segments have no eligible hosts: two are appliance-only and three have no IPv6 enabled yet, pending the firewall-policy pass that SLAAC on a client segment would require. |
||
|
vh
|
4e83395ddf |
docs(ipv6): all three esh-server hosts now carry the segment name
esh-pve-nas and esh-vm-db join esh-docker-vm on 4411:b105, at :50:55 and :50:60 respectively, each applied by the same prefix-deriving if-up.d hook so the last two groups read straight off the IPv4 address. Two obstacles are recorded because both will recur. The Proxmox node had link-local only despite every relevant sysctl appearing correct, because its bridge carries per-interface forwarding and the kernel ignores router advertisements on a forwarding interface unless accept_ra is explicitly two rather than one. The fix takes the advertised prefix while declining the default route, so the hypervisor gains an address without any change to how it routes; this was verified after applying, with the v6 default route count still at zero. The database VM refuses key authentication for the privileged accounts and its unprivileged login cannot escalate without a password, so the hook went in through the QEMU guest agent from the hypervisor, which executes as root inside the guest. The document notes the base64 indirection needed to get a multi-line script through intact. |
||
|
vh
|
ffb7fba346 |
docs: give the ESH IPv6 naming scheme a home, and make it real on one host
The scheme has existed since August as a single line of persistent memory, which a snapshot then deleted. It is a naming convention rather than temporal state, so it now lives in docs/pfi as a proper document, and the memory entry is reduced to a pointer at it. The document carries the full table, the address structure, the reasoning about which slots can and cannot hold a name, and the recipe for applying one to a host. It also corrects the conclusion the original note ended on. That note held that these names could never appear on the wire, which is true of everything UniFi is able to assign but not of what a host can assign to itself, and the distinction is the whole difference between a joke and an address. AdGuard on esh-docker-vm now holds the esh-server name, at 2607:73c0:402:1d02:4411:b105:50:45, where the segment identity and the IPv4 address are both legible. It is applied by an if-up.d hook that derives the prefix at runtime rather than hardcoding it, backgrounds itself with a retry so it cannot stall interface bring-up, and adds nothing to the existing interface configuration. This is load-bearing rather than decorative. The gateway advertises an IPv6 resolver to clients, macOS prefers it over the IPv4 one, and it previously pointed at an address derived from that host's MAC. |
||
|
vh
|
41091eef8f |
docs: restore the ESH IPv6 naming scheme, dropped by a snapshot
The six ESH LAN hexspeak names were recorded in |
||
|
vh
|
6217d3993e |
fix(scriberr): force uv to copy rather than reflink when building envs
Scriberr builds each model backend's Python environment with uv at container start. uv's default link mode reflinks or hardlinks out of its cache, which fails on this overlayfs over ZFS combination and reports it as a failure to clone a metadata file with errno 11, resource temporarily unavailable. The wording points nowhere near the cause. The damage was partial and therefore easy to miss: WhisperX and PyAnnote came up fine and the application served normally, while the Parakeet and Sortformer backends were quietly absent. Setting UV_LINK_MODE to copy trades a little disk and build time for environments that actually materialise, and the occurrence count for that error is zero on the following start. The knob is exposed through the environment so it can be moved back to the default if a future host does support reflinks. |
||
|
vh
|
efddb4e511 |
feat(scriberr): stand up transcription on ana-ml2, pinned to GPU1
Scriberr transcribes audio and video locally with WhisperX and speaker diarization, and it lands on ana-ml2 rather than ana-docker because the work is GPU-shaped: ana-docker offers eight cores already shared with fifty containers and thirty-seven gigabytes of disk, against ninety-six cores, terabytes on /tank and idle capacity on GPU1. The reservation names device 1 explicitly, since GPU0 is fully committed to the gen seat, and the container is confirmed to see that card alone. The image is built from source, which is not a preference. These are Blackwell cards at sm_120; the published CUDA image covers Pascal through Ada only, and the blackwell image the upstream README documents has never been published at all. The path upstream actually ships for sm_120 is Dockerfile.cuda.12.9, carrying CUDA 12.9 and cu128 torch, so that is what gets built. The compose header says so, because the obvious cleanup is to swap in the published image and that would silently drop the deployment to CPU. Two configuration details are load-bearing and documented where someone would go to change them. The application runs as uid 10001 rather than the usual 1000: that Dockerfile moves its user aside for Ubuntu 24.04's own uid-1000 account and chowns /app accordingly, while the entrypoint's remapping covers only the data directories, so at 1000 the process cannot open its database and restarts forever behind a SQLite error that reads as though the machine were out of memory. Secure cookies stay off while the service is reached over plain HTTP, or sessions are dropped by the browser and login appears to loop for no visible reason. Storage is bind-mounted onto /tank because model weights run to several gigabytes and the root pool on that host is nearly full. Also adds the scriberr service alias to internal DNS, following the existing alias convention so consumers name the service rather than the box. |
||
|
vh
|
22ae9cd480 |
ops(ana-gw): disable ACME and retract the all-port VIP claim
The ACME client is unbound from wan1, which removes the HTTP-01 challenge listener that had been holding port 80 open irrespective of allowaccess. An external sweep of fifty-five ports against the WAN address now finds nothing open at all, while the internal interface still serves the GUI and SSH and retains its certificate, which remains valid until late October and simply stops renewing. The previous note's claim that four virtual IPs were unrestricted all-port static NATs is withdrawn. A FortiOS virtual IP can be scoped either by an explicit forwarded port or by a service binding on the object, and the earlier parse inspected only the former. All four use the latter, and the custom services behind them are narrow. None of the fourteen is unrestricted. Ground truth from outside is recorded in place of further config reading, listing what each public address actually exposes. Three configured mappings answer nothing at all and are noted as tidy-up candidates for the migration rather than as exposure. |
||
|
vh
|
f39b66d2e1 |
ops(ana-gw): port 80 on the WAN address is the FortiOS ACME listener
The previous note attributed the open port to an ISP transparent proxy. That was wrong. It terminates on the FortiGate: system acme is bound to wan1, and FortiOS opens port 80 there to answer HTTP-01 challenges regardless of what allowaccess permits, which is why the port stays open with the interface set to ping only. Every non-challenge request returns a fixed 403 whose body reads ACME Access Only. No DNAT is involved; of the fourteen virtual IPs only two land on that address, neither on port 80. The wrong conclusion came from a sniffer filtered on dst host, which matches inbound packets alone and so excluded the replies being looked for. Filtered bidirectionally the box is plainly seen emitting SYN-ACK on port 80. The note records the rule. Two consequences follow. The earlier warning that certificate renewal would fail without http in allowaccess is retracted, since FortiOS opens the challenge port itself. And the listener is not an administrative surface, though its value is now marginal with WAN administration closed, so the note records how to remove it and leaves that decision open. Also captured: four virtual IPs are all-port static NAT and map every port of their external address. |
||
|
vh
|
7bc9754e40 |
ops: adopt AES-128 on both Anaheim tunnels and close public admin
Both tunnels now negotiate AES-128 for ESP, applied make-before-break so neither dropped waiting on its far end: the FortiGate was widened to accept the new cipher alongside the old one first, then each UniFi gateway was flipped. Single-stream throughput moves from 245 to 270 on the NH3 tunnel and from 268 to 304 on the ESH tunnel. Both network objects were diffed against pre-change snapshots and the only field that moved on either is the ESP cipher. The proposal lists are left accepting AES-256 as well. The peers offer only AES-128 so the extra entries are inert, and retaining them means a gateway reverting cannot strand a tunnel. With that up, the WAN administrative surfaces are closed. The interface is back to permitting only ping, and the infra-ops account is again restricted to RFC1918 space. Ports 443 and 22 were confirmed closed from two separate sites and management over the tunnel still works. The close was issued over the tunnel rather than over the WAN, since withdrawing SSH from the interface while connected through it would sever the session mid-command. The box now has no out-of-band path, which the memory records explicitly. Also captured: the two UniFi vault items have different shapes, one a bare key and one a documentation note requiring extraction, which produces an opaque nginx rejection if missed, and the ESH key's first confirmed write. |
||
|
vh
|
6edebe4864 |
ops: forwarding through a downstream WireGuard terminator is free
FortiOS has no WireGuard, so any WireGuard site-to-site has to terminate behind the edge, and ana-wg already fills that role. The earlier 767 Mbit/s figure was taken with traffic terminating on that box, which left the forwarding case unmeasured and overstated what a real deployment would see. Measured properly, transit from NH3 through the tunnel and onward to ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight, so the forwarding hop costs nothing and the full threefold gain over IPsec survives. The container uses around a fifth of four cores at that rate, against the UniFi gateway spending a third of its own four to move a third as much. The note records what still needs deciding rather than presenting this as ready to build: Anaheim hosts have to route to the terminator directly or pay a hairpin through the edge, which was not measured; the terminator introduces a failure mode the edge did not have, with a fallback route as the mitigation; and the NH3 end needs its own terminator, where a Linux host matches the measured figure while the existing UniFi WireGuard server would likely land lower. The test tunnel, its peer and the temporary route were all removed. |
||
|
vh
|
062215e81a |
ops: the tunnel's per-flow limit is the UDM's software AES-CBC
Varying only the cipher settles what inference could not. AES-128 has ten rounds against AES-256's fourteen, so a software-bound path must speed up when the cipher gets cheaper while a hardware-offloaded one will not move. Run as A/B/A on a single stream, AES-256 returns 232, 245 and 243 Mbit/s and AES-128 returns 282 and 275, a gain of about a fifth at identical CPU. The FortiGate's offload engine is therefore not the constraint, and the operator's reading of the UniFi side as software crypto without acceleration is correct. Two earlier claims of mine were wrong and are corrected in place. The observation that the UDM sat at seven percent CPU was a sampling artifact, since UniFi refreshes that statistic on the device report interval and the windows used were four seconds; a sustained run reads thirty-five percent, around one and a half of four cores. The per-session offload hypothesis is refuted rather than merely unproven, because it predicts no response to a cipher change. The remaining figures follow from this. One stream is one core, eight streams engage about three, and aggregate stops responding to cipher choice once several cores are working because the path bounds it. CBC chaining is what makes this expensive, as it serialises blocks and prevents the AES instructions from pipelining, which is also why the same gateway manages far more over WireGuard. The cipher was restored and the network object verified unchanged against its pre-test snapshot. |
||
|
vh
|
8ecffa1fab |
ops: WireGuard over the same path carries one stream at 767 Mbit/s
The operator asked for a WireGuard test between ana-wg and NH3 across the public internet, which is the arrangement that separates the path from the crypto since neither gateway performs encryption in it. A single stream returns 767 Mbit/s against 245 for IPsec over the same wire, and eight streams return 763, so one flow already saturates the path and there is no per-flow penalty whatsoever. That settles several things at once. The limit is not the circuit, the NH3 uplink, the ISP or the physical path, all of which carry 767 on one flow. Ranking the implementations, Linux WireGuard shows no per-flow penalty, UniFi's WireGuard shows roughly 1.4x, and IPsec on this pair shows 2.8x. Latency under a single bulk stream tells the same story, rising to 12.7 ms on WireGuard against 102 ms on IPsec. This overturns the earlier conclusion that changing transport was not worth pursuing, which compared eight-stream figures and so understated the gap for single-stream work by a factor of three. A WireGuard site-to-site terminated on ana-wg now looks worth considering, and the note records the open questions around failover and policy. Attribution between the FortiGate and the UniFi gateway remains unresolved, since both perform IPsec in the slow measurements, but the remedy does not depend on which one is responsible. The test tunnel was removed, ana-wg is back to its original three peers and the generated keys were shredded. |
||
|
vh
|
d42e9d8712 |
ops(ana-gw): establish the per-flow cap is IPsec-specific, not capacity
The operator asked whether the 80F is simply out of capacity. It is not. Routing a single flow between two Anaheim VLANs through the same box, with no tunnel involved, sustains 940 Mbit/s, which saturates the link, and eight flows over that path return the same figure. There is therefore no per-session ceiling in the plain forwarding path and the roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone. Measuring the second tunnel with the same probe supports this. The ESH side, terminating on different gateway hardware at nearly half the round-trip time, returns 268 Mbit/s on one stream against NH3's 245, and 715 against 692 across eight. A window-limited path would have been substantially faster at the shorter round-trip, so the agreement between two dissimilar far ends is further evidence of a rate cap. This also corrects the previous note, which named the FortiGate on reasoning that could not separate the box from the protocol, since every slow path was both. That separation is still not established: both tunnels terminate on UniFi gateways running common firmware, so the cap could belong to either side. The note records the tunnel topology that would decide it and flags the bearing on the pending replacement. |
||
|
vh
|
cf0cb2cbb3 |
ops(ana-gw): pin down what limits a single stream across the tunnel
The earlier note described the constraint as per-flow serialisation without evidence. Pinning SO_SNDBUF and sweeping it shows what it actually is: throughput holds flat between 224 and 247 Mbit/s across a thirteenfold range of in-flight data while round-trip time scales with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions are absent throughout. That is a fixed service rate with a standing queue ahead of it, so socket buffer, window scaling and congestion control are not the lever and should not be pursued. The same measurements surface something with wider consequences than throughput: a single bulk stream lifts tunnel latency from 6.9 ms to 102 ms average, 136 ms peak, with no loss. Interactive traffic sharing the Anaheim link degrades sharply whenever anything moves bulk data, and parallelising transfers makes that worse rather than better. Localisation rests on the FortiGate appearing in both slow paths and in neither fast one, with aggregate throughput over the same security association reaching 692 Mbit/s. Per-session NPU offload fits the shape but was not confirmed on the device; the note records the test that would settle it and flags the bearing this has on the pending FortiGate replacement. |
||
|
vh
|
e41d19f1cb |
ops(ana-gw): close out the Anaheim tunnel cipher question
The AES-GCM cutover was authorised and attempted, NH3 side first. It cannot be completed: UniFi's manual site-to-site IPsec implements no AEAD cipher. Eight GCM spellings were rejected with api.err.InvalidPayload while an otherwise identical body carrying aes256 returned rc:ok, which isolates the enum rather than the request as the cause. The accepted set is aes128/aes192/aes256/3des. Both Anaheim tunnels terminate on UniFi gateways, so this blocks the ESH tunnel on the same grounds. Measuring while testing also retires the premise. NH3's uplink is a 1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather than the ~550 previously recorded from a 4-stream run. Compared with WireGuard over the same gateway and uplink, the deficit narrows from 2.3x at one stream to 15% at eight, so moving this link onto a different transport is not worth doing. The constraint is per-stream, around 245 Mbit/s, with both endpoints idle under load. Parallelising bulk transfers remains the mitigation and is worth 2.8x at no cost; NFS nconnect is the equivalent lever for single-stream mounts such as /mnt/smithy on ana-ml2. FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list, which is inert while the peer offers only CBC and avoids a further renegotiation. The UDM network object was diffed field-by-field against its pre-change snapshot and is unchanged. |
||
|
vh
|
5af362e9d0 |
ops(ana-gw): restore WAN admin access ahead of the FortiGate cutover
Re-open the ana-gw admin GUI on wan1 so the Anaheim edge can be managed remotely if the cutover goes wrong, reversing part of the 2026-08-12 lockdown. Two config changes, nothing else (verified by diffing pre/post `show full-configuration`): - wan1 `allowaccess ping https` — https only; http, ssh, and fgfm stay off, and wan2 is untouched. - `infra-ops` trusthost widened to all routable IPv4; the `admin` account stays locked to 10.0.0.0/8 so the guessable username remains unreachable from the internet. Verified end-to-end from two sites: a real `/logincheck` POST returns AUTH OK over the public path, on a browser-trusted Let's Encrypt cert for ana-fw.phasefinal.com valid through 2026-10-27. Two FortiOS behaviours worth recording, both of which cost time here: a trusthost whose base address is 0.0.0.0 is silently treated as unset (so there is no writable "any" — only decomposed ranges), and trusthost is enforced before the TCP handshake, so a blocked source sees a filtered port rather than a refused login. Follow-ons captured in memory, not actioned: ACME renewal for the admin cert needs port 80 on wan1 (next attempt ~2026-09-27), and a ~5 SYN/s source in 179.51.184.0/21 now draws SYN-ACKs at no measurable CPU cost. |
||
|
vh
|
b6340519bc |
memory: snapshot — Anaheim tunnel at 25% of circuit; selene retired; hrafn CI fixed
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired. |
||
|
vh
|
0ad332bb4a |
feat(booth): per-row link removal + render the link board as real UI
The standing link board is the one MULTI-WRITER booth -- every agent session appends operator-facing URLs to it. "Delete the folder" was the only granularity available, so removing one dead link meant hand-editing markdown. It is 32 rows and only grows. booth links row number, entry id, raw row booth unlink 3 by row number booth unlink 8b40e0a5 by entry id (what the UI's x posts) POST /b/<name>/unlink form field `entry` = content id ROWS ARE ADDRESSED BY CONTENT ID, NEVER BY POSITION. The board is append-only and multi-writer: another session can post between listing it and clicking x, and an index would then delete a neighbour. An id either matches the row you saw or matches nothing. A row number typed at the CLI is resolved to its id BEFORE anything is deleted. Appends and prunes now take the same flock on .links.lock, so a post cannot be lost inside a prune's read-modify-write. UI: a booth carrying links.md renders as rows -- description, URL, provenance, copy button, per-row x -- instead of a markdown blob. links.md is filtered out of the gallery so it does not appear twice; the header counts LINKS not files; the empty-state and the one-click "Wipe now" both stand down for a board (same rule as the kept lane: nothing durable is one click from gone). booth/links.py extracted, STDLIB ONLY. The CLI needs this logic and must not require the service venv -- importing app.py drags in FastAPI, so deleting a line from a text file would have needed a web framework installed. THREE BUGS FOUND BY TESTING, all in the shell wrapper while the module was correct throughout -- module-only tests would have caught none of them: - `[ "$n" -eq 0 ] && echo ...` as the LAST statement made `booth links` exit 1 whenever the board had rows. `unlink`'s index lookup calls it inside $( ) under `set -e`, so a successful listing killed the caller and the removal silently did nothing while reporting success. - ids are 8 hex chars and roughly one in forty is ALL DIGITS; those were read as row numbers, resolved to nothing, and removed nothing. Now disambiguated by the id's actual shape, not by "is it numeric". - filtering links.md out of the gallery left `items` empty, so a full board rendered "This booth is empty" and an empty <div class="gallery"> under 32 visible rows. 87 tests (was 76): parser tolerance of hand-written prose, content-id stability across concurrent appends, removal precision, UI branch behaviour for board/normal/empty booths, and subprocess CLI tests pinning the two shell bugs. Deployed to nh3-dev and verified against the live 32-row board read-only; board file byte-identical afterwards. |
||
|
vh
|
4be880f36c |
feat(booth): kept boards can be deleted from the UI; document the TTL-reset trap
Kept boards had no delete path in the UI at all. The kept lane deliberately omits the wipe control -- a one-click wipe next to the durable stuff is a footgun -- but "deliberate" had been implemented as "impossible": the only routes out were ssh or a hand-written API call. Now it is two deliberate acts. A `release` control on kept cards drops the sentinel, the board moves to the ephemeral lane, and the existing x wipes it from there. Release is reversible -- POST /b/<name>/keep pins it again. POST /b/<name>/unkeep release the pin POST /b/<name>/keep pin it (round-trip, so release is not a one-way door) FOUND WHILE TESTING, and it invalidates the previously-documented workaround: removing the sentinel BUMPS the booth directory's mtime, and booth age is the newest mtime in the tree -- so a released board's clock RESETS from 10,000s to 0s and it survives another full TTL. The old comment said "remove the sentinel first (it rejoins the sweep)", which is true but means the board lives another 24h, not that it gets reaped. Unkeep-and-wait is a delay, not a delete. test_releasing_a_board_RESETS_its_ttl_clock pins that behaviour deliberately so nobody re-derives the workaround. Release is what unlocks the x; the x is what deletes. CLI: `booth rm` already worked on kept boards but said nothing about it. It now announces "(was KEPT -- durable board)" so wiping something durable can never look identical to wiping run output. Not a block -- a CLI user naming a booth is being explicit. 5 new tests (67 pass). Verified live on nh3-dev: release renders on all four kept boards, the ephemeral lane keeps its x, and the links board is untouched with its sentinel intact. |
||
|
vh
|
b8a535507a |
feat(judge-bench): keep the judge harness; warn about the 7-way alias collision
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced. |
||
|
vh
|
ca3c984f93 |
feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design
Benchmarked selene against gen on selene's own job: 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a neutral JSON prompt and Selene's native Atla template. 288 calls, all free local. neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) gen won on both templates and selene's BEST sat below gen's WORST. Selene was given its own fine-tuned template as a fairness check; it gained one point, not the three it needed. Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters most. gen returned tie correctly on the JSON template. Selene also compressed the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only win was ~3x latency, unexercised at ~60 calls/day with zero queueing. TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately: - chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says consumers bind the capability, not a concrete model. Sampler profile copied from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served config matches the benchmarked condition. - selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A served-name is a contract about what the model IS; answering it with a different model hides a material change behind a stable string. Operator ruling: "never repoint a named model at a different model's endpoint -- that is intentionally misleading." Verified: the gateway now returns HTTP 400 "Invalid model name" for it. Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role moved onto an existing seat rather than allocating anything new. Canonical litellm config synced from the host; ana-ml2 README and recommended-model-settings updated. compose.yaml kept for reference, not deployed. |