Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to
autoneg True / setting_preference auto from the payload staged before the
original change. ~50 s of renegotiation, nothing rebooted.
The force failed progressively and I called it too early. Errors on the
ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with
flapping the operator noticed before my soak had accumulated enough to raise
it. I read a two-minute flat window as a plateau and said so; a marginal link
declares itself over hours, not minutes.
Do not re-force this port. The cable is being replaced at the weekend and a
correctly-coded DAC negotiates 10G on its own.
Operator is keeping the override rather than reverting: he has a replacement
cable and will run it at the weekend. Recorded as a held position so a later
session does not revert it as an oversight or re-run the diagnosis.
Also corrects a claim this runbook made an hour earlier. I wrote that the
zero-error 1G link ruled out a marginal cable. It did not — it only proved the
cable was clean at 1G. The two-hour soak shows errors climbing monotonically at
10G (200 to 221 rx, ~1 per 23 MB) plus one self-recovering link drop, so the
autoneg fallback had a real basis and the fix is the cable, not the config.
The OEM SFP-H10GB-CU3M between ESH-Media SFP+1 and UDM SFP+2 linked cleanly
at 1000 with zero errors on both ends, which rules out a damaged cable — a
marginal one links at 10G and throws CRCs rather than dropping a rate. The
switch reported sfp_compliance "Unknown": it reads the EEPROM but cannot
parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate.
A TP-Link TL-SM5220 DAC on the adjacent UDM port runs at 10000 with the same
speed_caps and autoneg on, which exonerated the port, the firmware and
autoneg before anything was changed.
Forced autoneg off / 10000 / full duplex on the UDM end only. ESH-Media's
end followed to 10000 on its own, which is the proof the cable was always
capable and only the negotiation was wrong. Both ends now 10000.
The UDM end was chosen because it is the recoverable one: ESH-Media reaches
the controller through this link, so a failed force there would strand the
switch. The path to the UDM was traceroute-verified not to cross the link,
and the revert payload was written before the forward one was sent.
Cost ~60 s of link renegotiation. ESH-Media, the E7-ESH-Media AP and the
USW Flex XG ESH-Office-Vuong all returned at uplink_speed 10000 with uptimes
unbroken; the Zigbee coordinator and HA's path to it were re-verified.
⚠ The port is now pinned: a non-10G module will not link there. Documented,
along with the ~200 link-up errors that then went flat and the fact that
steadily climbing errors would mean the cable genuinely cannot do 10G.
I described ea:f6:0a:ca:f5:b4 as "locally administered, no OUI" and treated
it as the class of MAC that may regenerate at boot. ha-dev corrected it and
the correction verifies: the device advertises e8:f6:0a:ca:f5:b4 over mDNS,
which differs in exactly the locally-administered bit, and E8:F6:0A is
registered to Espressif Inc. in the IEEE registry while EA:F6:0A resolves to
nothing. That is the standard ESP32 pattern — one factory base MAC in eFuse,
sibling interface MACs derived deterministically — so the reservation is
keyed correctly and cannot drift on its own.
The PoE-cycle test still stands and is now corroborating evidence rather than
the only evidence. The residual risk narrows to a firmware change to the
derivation scheme.
Also from ha-dev: tcp/7638 is open alongside 6638; mDNS crosses the VLAN
boundary so HA rediscovers without help; and HA's pending smlight config flow
is keyed on the mDNS service name, which did not change, so a stale flow may
still hold the dead 10.0.10.58 and should be dismissed rather than confirmed.
ha-dev declined the dns: resolver fix on their stack — configuring by IP
costs them nothing and the entry would couple HA name resolution to AdGuard
uptime for no present benefit.
The coordinator arrived on esh-userland (VLAN 10) at a DHCP-assigned
10.0.10.58. It now sits on esh-iot (VLAN 90) at 10.0.90.10, reserved on
the UDM and named slzb-mr1u.esh.internal.
Address on the network side, not the device — the pfi-gx10 ruling: a
reservation moves with the hardware, a device-side static goes stale.
10.0.90.10 is below the 10.0.90.40-250 pool so it cannot be handed out.
The MAC is locally administered (ea:f6:0a:ca:f5:b4, no OUI), which is
exactly the case where a reservation can silently stop matching. The
PoE power-cycle that makes the device re-DHCP is also the stability
test, so it cost nothing: it rebooted and came back on exactly
10.0.90.10, which only happens if the MAC held.
ESH-Media carried 8 port_overrides and that PUT is a whole-array
replace, so the array was diffed to prove exactly one field on one port
changed before sending, and read back after.
Inter-VLAN reachability needed no firewall work and was proven from
inside the HA container before the move, against existing IoT devices.
Testing from the Docker host would have proven the wrong thing — HA
runs on a macvlan address, 10.0.50.46, not the host's 10.0.50.45.
Documented but NOT fixed: the HA container cannot resolve any
.internal name (its resolver is Docker's 127.0.0.11, upstream excludes
the fleet AdGuard). Pre-existing, verified against names that predate
this change. HA should be configured with the IP; changing the DNS of
a live home-automation stack is ha-dev's call.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.
Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:
io pressure full avg10 96.5% -> 0.01%
in-flight writes 45, completing none -> 0 0
D-state processes 11 -> 0
32 MiB dsync write never completed -> 0.24s (133 MB/s)
The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.
Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.
It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.
Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.
Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".
Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.
Each guard is something a hand-run got wrong first:
- an account may not own its own home. A `sudo mkdir` before sysadminctl
leaves /Users/<account> root-owned; the account then authenticates, gets a
shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
the wrong account — it did, at a working install, and only permissions
stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
INDEPENDENTLY of agent-default-model, so a correct default still showed
DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
gen-reasoning, 403 on gen), not trusted from the mint.
The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.
Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
My smoke test used a refused port on a live host and presented it as the
check for the 2 s budget. Measured: it returns in 79 ms, because a refused
port answers instantly with an RST. It verifies the glyph and nothing about
the deadline. An address that black-holes — an unrouted LAN host — takes
2.065 s and is the one that exercises it. Both are now in the page with what
each actually proves, because letting the fast case imply the slow one is how
a status line that hangs the prompt ships with a green test beside it.
Found by forseti in their own copy of the same probe and confirmed here
rather than adopted.
The smoke test also moves from expected-values-in-trailing-comments to
printing `got [x] wanted [y]` on one line. The comment form is the shape that
produced the false pass that caught two agents inside an hour, and the
expected-value column is what caught it — so the page should use the thing it
recommends.
I wrote that Claude Code runs the status line in a bare shell. What I
actually measured was that a HAND-RUN from an interactive session inherits
that session's exported ALTHING_HANDLE; Claude Code's own invocation is a
different execution and was not observed. forseti caught it before the
sentence had been read by anyone else.
The evidence points the other way: this box's Claude Code process carries
ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ, inherited from
the dev-launch shell, and a spawned child would normally inherit both —
observed independently on two seats. Whether Claude Code scrubs before
spawning was not observed either.
Nothing depends on the answer, which is why the page now says "do not depend
on the environment being present" instead of asserting its absence, and
records it as an open question with both sides. The smoke test still strips
the variables, because that is the harder case and the one whose false pass
caught two agents inside an hour.
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.
Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.
Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.
The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
Eight of seventeen live seats were still on fifo after the 3.3.1 deploy,
because a SessionStart hook cannot reach a session that was already running.
Operator's call: leave them. They are reachable, mail is held at the post
office regardless, and each self-corrects at its next session start; the
residual is that a reaped fifo waiter goes silent until restart, which costs
timeliness and never mail.
Recorded with the no-unsolicited-fleet-notification directive next to it,
since the tempting action here is precisely the prohibited one — audit,
report the count, let the operator decide.
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.
The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.
deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.
Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.
That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).
The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.
Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
The cross-reference loop filtered live waiters on `kill -0 "$pid"`. A lock
file left by a reaped listener names a pid the kernel is free to reissue,
so the loop would report an unrelated process as a live waiter — and a
phantom waiter is how a false "your seat is unreachable" notice reaches a
seat that is fine, with nothing in the output to falsify it.
`session_listener.sh --stop` already refuses this standard: it walks
/proc/<pid>/cmdline for the `--_route=<handle>` segment before signalling,
on the grounds that a live pid proves existence and not identity. The audit
now asserts the same thing and prints STRANGER on a mismatch.
Verified against the live box: six waiters, all identity-confirmed, all
reporting push.
A version number cannot tell you what a stale plugin cost. 0.0.1 and
0.1.1 differ by two hooks and a script, so the runbook now carries a
check that compares hook lists across cached versions and looks for
pane-route.sh directly.
Also records why this hid for five days, which is the more transferable
half. A missing deploy surface does not present as an error -- it presents
as "the migration needs manual work", and there was a ready explanation
for that, because four of five seats were non-Claude and genuinely did
need hand-holding. The seat that falsified the story was our own: a Claude
Code seat that should have self-declared and did not, and it looked
exactly like the other four. Nobody asked why the automatic path had not
fired on the one seat it was built for.
So: when a migration needs manual intervention, verify the automatic path
was actually deployed before concluding it does not apply to your case.
Deploying althing touches four independent surfaces on nh3-dev. Three
were known. The fourth -- the plugin -- had no step in any runbook and
drifted for five days before anyone noticed.
The plugin chain is repo plugin/ to the marketplace directory to Claude
Code's cache, and neither hop was automated. The marketplace directory
was a frozen copy from Aug 28 carrying only the UserPromptSubmit hook,
with no SessionStart, no SessionEnd and no pane-route.sh at all. So the
claim that CC seats re-declare their pane route automatically at session
start was never true on this box, which is why every seat had to be
hand-declared with a pid measured by hand.
The script backs up the marketplace directory before syncing, re-stamps
its marketplace.json from the repo's plugin.json, and uses `claude plugin
update` for the cache rather than hand-editing installed_plugins.json --
that is Claude Code's own bookkeeping and a subtle mistake there breaks
the plugin in a way that looks like an upstream bug.
The runbook also carries the two things most likely to waste someone's
afternoon: `uv tool install .` without --force is a silent no-op that
exits 0 having done nothing, and a live waiter reporting mode:pull is a
seat that will never be poked, with the audit loop for finding them.
Synapse v1.120.0 -> v1.159.0 and Element-web v1.11.80 -> v1.12.27 (2026-09-01).
The existing build steps date from the AIPA era and are now marked as provenance
rather than as instructions.
Records what only existed in a session transcript:
- Schema migrations are one-way; rollback is restore-from-dump. Pre-upgrade
pg_dump procedure, with a pg_restore --list verification step.
- Why the appservice user namespace is now exclusive: false. exclusive governs
who ELSE may act, not what the appservice may do, so on a closed single-admin
server it locked out all other account creation to prevent squatting that
cannot occur. Includes the two things not to do: narrow the regex (orphans 13
accounts) or rename the id (Synapse keys ownership on it).
- The shared-secret registration HMAC takes no trailing null after notadmin.
- Room alias convention #<agent>-<purpose>, operator-ratified, with its cost
accepted deliberately and its rationale stated as room-identity-carries-tier
rather than push-payload-carries-room-name.
- Push reality: the pusher is event_id_only, so the notification is assembled
on-device by Element X's service extension. Records the resulting
server-invisible failure mode when the phone cannot reach the homeserver.
- QR sign-in requires Matrix Authentication Service and why it is deferred.
Ops ownership recorded: worldtree-dev writes the bridge, infra-ops operates
this instance.
Found while answering a question from the operator, relayed via brokkr-smithy-dev,
about whether a recorded Qwen3.8 degeneracy at ~1,700 tokens relates to a length
sensitivity just measured on the tuned Gemma-4. The record is §3.7 and the number is
~2,000 -- but reading it to answer that question surfaced that the section is stale.
§3.7 presented "disable prefix caching, keep MTP" as THE MITIGATION, resolved
2026-08-17, and stated the gen seat runs that config. It does not and has not since
that same day: APC-off passed a synthetic 7-turn probe and the operator still saw
severe degeneration in real use, so it was reverted. The multi-day hunt resolved to
the AEON W4A4 quant being defective, with MTP / prefix-caching / gateway merely
amplifying it (§3.8 records the corrected causal story; §3.7 was never updated to
match).
Verified against the live container rather than against the compose file alone:
vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3.
stacks/gen-seat/compose.yaml carries the full corrected history inline and is the
current authority.
§3.7's superseded text is kept and fenced rather than deleted -- it is the history of
a mitigation that looked right and was not. Added a dated row to §7 per the standing
rule that a wrong playbook claim gets a superseded-claims entry, not just a fix.
The lesson inside the lesson is worth more than the correction: §3.7's own standing
rule is "gate MTP on a multi-turn coherence probe, not just single-shot acceptance."
The APC-off mitigation was gated on exactly that probe, passed it, and still failed in
real use -- the multi-turn probe was itself too small to gate on. A passing probe is
not sufficient evidence at any size that has not been calibrated against real use.
The entry reported an 'output-stability regression' as a novel run-2 finding.
Both halves were false and the corrections are more instructive than the
original conclusion, so they stay in-line rather than being edited over.
Not new: run 1's own gate record already carried the same effect with a caveat
attached and unresolved. Two runs across two different base models makes it a
property of the RECIPE, not of the base swap -- which also means a third run
that changes the base again will not fix it.
Not degeneracy, and not a separate finding: all 46 flags were too_short rp turns
of 3-14 words, and the two collapse guards fired ZERO times on any run. It is
the left tail of a length distribution that had been measured and reported in
the same message. Truncation is the same mechanism mirrored on the story side.
Both are thresholds calibrated on the base's output shape applied to a model
with a different one -- 4.6.1, which both parties had written down and neither
applied.
The surviving lesson is sharper: a short-answer gate cannot see length behaviour
AT ALL, and because it could not, the effect went two full runs before anyone
named it. The cost of a gate-set blind spot is measured in runs.
Adds 4.6.3.1 on trip points inside the serving stack's jitter -- same seed, same
weights, rate moves 9.6% -> 12.6%, sd 1.77pp. Not 'the gate is
non-deterministic' but 'the trip point sits inside the jitter', because the fix
follows from the precise statement. Includes the split-design rule for measuring
such a rate, and the rule that a measured rate must carry its corpus in its
name.
The output-side test ran on the live seat. There is no shoulder at 123: the
120-139 bin holds three of ninety-six and is a TROUGH, and 17.7% of generations
cross a cap PIPPA can never cross. The clip-as-boundary reading is dead, killed
by the test that could have confirmed it.
Corrects this document's own earlier read, which compared the tuned MEAN (88.5)
to PIPPA's MEDIAN (67) and concluded 'comfortably inside the upper body'. Median
to median it is 62 against 67. Mixing statistics across a comparison produced a
more reassuring answer than the data supports.
What the data shows instead is bimodality -- a mode at 20-39, a trough, a second
mode astride PIPPA's centre, a tail to 505, against a base with no such shape.
The tune changed rp length's SHAPE rather than its centre: roots whose length
distributions do not overlap learned as distinct modes rather than blended into
an average. And the skew is rp-ONLY, which localises it to the family the
clipped root lives in and is the strongest support the turn-share mechanism gets
from the output side.
Consequence for pair generation: chosen/rejected sampled from a bimodal
generator inherit the mixture, not a mean, and naive sampling over-draws the
short mode.
Also records that the degeneracy rate is NOT yet a usable baseline -- same arm,
same seed, VOID flipped no->YES across a re-run because the 10% budget sits at
the noise boundary. A guard whose trip point is at the noise floor produces
disagreement between honest observers rather than silence. Replicates running.
The run-2 gate found tuned rp turns 36% shorter than base. brokkr hypothesised
the mix was teaching PIPPA's 2023 Character.AI product clip; the corpus side is
now measured and confirmed. PIPPA's max is 123 words EXACTLY, 100% at or under
it, and 0.00% in the 124-130 band -- a wall, not a preference. Every other root
crosses its own p99 smoothly.
The mechanism is sharper than 'PIPPA is in the mix'. PIPPA is 70.3% of bot TURNS
but only 37.5% of bot WORDS, precisely because its turns are clipped -- and
length is learned per turn, not per token. By loss tokens it looks like a third
of the dialogue signal; by end-of-turn demonstrations it is seven in ten from a
source that cannot exceed 123 words. Generalises: a length-clipped root is
over-represented in the length signal by exactly the ratio its clipping creates.
Counter-evidence recorded too: the tune landed near PIPPA's MEDIAN (67), not its
CAP, which is central tendency rather than learning the boundary. Weaker claim
than the hypothesis, and not demonstrated either way.
Filed here rather than only in the gate record because preference pairs
generated FROM this tune inherit its length distribution in both chosen and
rejected -- DPO would train an artifact in as an explicit objective. Settle the
length question before generating pairs.
Run 2's reasoning battery reported zero truncations and zero degenerates on both
arms across four passes. The same tune, measured on long-form generation in the
same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real
output-stability regression, structurally invisible to that gate because its
answers are short.
Not a bug in the battery -- a coverage property. An instrument measures the
regime it samples, and output length is a regime. Generalises to context length,
conversation depth, and any axis where the gate's operating point is narrower
than production's.
The actionable form: enumerate the regimes your gate set spans, name the ones it
does not, and decide deliberately rather than discovering the gap downstream.
Corollary on sequencing -- put a long-form generation in the gate and put it
early, because a length-dependent regression is exactly the one you want found
before four clean short-task passes make everyone comfortable.
From the run-2 gate. A memorisation probe reporting 0.00% across all 72 items is
the correct output for a model that has not seen the corpus, and is also the
exact output of a probe that is not firing. Nothing in the number distinguishes
them. brokkr-smithy-dev drove the overlap function with known-answer inputs
(identical 100%, half-verbatim 65.38%, unrelated 0%, empty 0%) before trusting
the null, which is what converts a suspicious zero into evidence.
This is 4.5's inert gate wearing a different face: there a check that could not
return 'fail', here a measurement that cannot return non-zero. A clean null is
the most reassuring output an instrument produces and the least
self-evidencing.
Same section records the identical-on-both-arms variant: the diversity battery's
rp family froze zero markers, so its attractor hit rate read 0.0 on base AND
tuned. That reads as a clean result and means the instrument cannot discriminate
on that family. Report as a bounded limitation, never as a delta of zero -- a
check returning the same value for every input is not measuring.
Checklist gains the line.
4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2
launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on
the same checkout during its seven hours and _git_commit() was called at save
time. Commit AHEAD of the code that ran, naming changes it never executed --
including the provenance fields this section prompted. Same defect as run 1's
BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at
launch, carry it, and record the dirty flag beside the commit rather than
instead of it. Generalises to every run-scoped identity: anything read at save
time describes the world at save time.
4.6.1 -- the inverse of the inert gate, and it costs trust rather than
correctness. A coherence gate false-rejected 'The capital of Portugal is
Lisbon' as degenerate against a global 15-word floor. The floor was calibrated
against the wrong reference, not set too strict. Lowering it globally would
blunt the check where short output genuinely is degeneration; the fix is a floor
per prompt. Write the positive test alongside the negative one.
Run 2 is an SFT on the official instruct base, so it will refuse at near-stock
rates by design; targeted DPO is where refusals get pruned on chosen axes. That
was the trade accepted when the stock base was picked over a third-party
abliteration.
Surveys what is on disk against what the stage needs. Ready: the merged tune,
the SFT adapter, GPU0 once the eval seat comes down, the whole non-loss half of
the SFT harness, two unvetted Gutenberg preference sets, and the LitBench-RM
judge.
Missing, in order of pain: preference data for the refusal axes (nothing on
disk targets it -- the Gutenberg sets are prose-quality), the axis list itself,
and a DPO trainer (trl is not installed).
The gating item is not technical: WHICH refusal axes are in scope and which are
explicitly kept. Data generation, pair counts, the held-out split and the
success probe are all functions of that list, so nobody should generate a pair
before it is written down. Flags that the domain-compliance probe should
measure run 2 BEFORE pruning, since the pre-number is the only baseline that
will ever exist.
Also records the operational trap: do the trl install AFTER a run finishes,
never during one -- a resolution that upgrades transformers under a live
process can break its save path.
§4.3's generalisation was stated and then not applied to the manifest that
prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and
the one pairing that would have caught the §4.1 cache failure -- the mask's sha
against the loss-token delta -- existed by accident, because someone had asked
for an encode report for unrelated reasons.
Adds the audit table, and the rider that matters more than the table: put the
observed check where it can actually FAIL. chat_template_sha256's pair is the
sha of the string the tokenizer carries, but asserting that in the parent one
line after assigning the file to the tokenizer compares a value to itself. It
belongs in the encode worker -- a different process, across a pickle boundary,
where an unset config key silently leaves every worker rendering through the
checkpoint's own template.
brokkr-smithy-dev pointed §4.5's own test at §4.3's remedy: recording
`attn_implementation_resolved` is a check that cannot fail on the axis the
failure lives on.
A silent Dynamo fallback to uncompiled flex leaves
`config._attn_implementation == "flex_attention"` untouched while the run
computes at ~20x the cost and, per torch's own docs, does not work correctly
through the backward pass. The field records the request's RESOLUTION, not its
SURVIVAL. On the failure mode that matters it reports success either way.
So the section now requires the step-time distribution beside it -- n, min,
p50, p99, max -- which is the check that can actually fail. Compiled sits at
p50 ~20 s; a fallback at ~400 s. One perf_counter() in on_step_end buys it.
Distribution rather than a mean, because a mean hides exactly the bimodality a
PARTIAL fallback produces.
Generalised past this instance: any provenance field recording a CONFIGURED
value is a claim about intent. If the failure you fear is the configuration
silently not taking effect, you need a second field recording an OBSERVED
consequence, and the pairing is the check. A settings dump alone is decorative.
Two implementation details are called out because both were wrong in the first
draft -- percentiles nearest-rank so every reported value is a real
observation, and exclude the FIRST step rather than the slowest, since step 1
carries compilation but is not reliably the maximum on a variable-width run.
New §4.7.1: rotate the log on relaunch. Run 2's first attempt died on the
warmup_ratio TypeError and the relaunch appended, so the traceback sat at line
15 of a file whose live run began at line 39 -- and a `tail -n +1 -F` monitor
replayed the dead traceback as a fresh event. One file describes one run.
Checklist gains both lines.
The playbook covered why a run is SLOW. It did not cover the more expensive
failure: a run that COMPLETES, reports plausible numbers, and is wrong about
itself. Seven of those turned up on the Gemma-4 ERP/RP tune between 08-24 and
08-26 and not one raised an error.
New §4, seven landmines plus a pre-launch checklist:
4.1 a cache key must cover the MEANING of the cached thing. The encode
cache missed the impersonation mask; run 2 would have reused run 1's
unmasked encodings and written impersonation_mask_sha256 into its own
manifest while doing it. No error, no count change, normal loss curve.
4.2 validating a VALUE is not validating the PARAMETER. warmup_ratio was
in range and deleted from transformers 5. Build kwargs as data and
diff the NAMES against the installed signature -- you cannot check the
argument list of a call you have already made.
4.3 record what the run RESOLVED to, never what it requested. Run 1
recorded no attention backend, so an MFU panel profiled the serving
seat under sdpa and recommended adopting flex_attention for a run that
was already using it.
4.4 never train from a dirty tree; harness_commit will name a commit that
does not describe the run. Annotate afterwards, never edit the shipped
artifact -- and state what is NOT wrong, or the note casts doubt on
every field it omits.
4.5 a watchdog whose pgrep pattern appears in its own argv can only ever
return "alive". The inert-gate shape in a liveness check.
4.6 an instrument nobody runs is not an instrument. Mutation-check any
test guarding a property that fails silently.
4.7 fix a stale measurement at the source. "~4.3 HOURS to rebuild the
encode cache" (really 145.5 s) was copied into a new launcher by the
same person who had just measured the real number.
4.8 the pre-launch honesty checklist, ten minutes.
Also:
- Header and framing widened. The file is now a training playbook with a
throughput half and an integrity half; the filename stays for inbound links.
- Sections 4-7 renumbered to 5-8. External refs are all to §1.1 and §3.4 and
are unaffected.
- Four rows added to the superseded-claims table, including the kernel table /
68% quadratic / 8.6% MFU set, which describe the serving seat rather than
the training run.
- gemma4-erp-tune-sizing.md §6 carries a correction banner with the explicit
falls/survives split, because that is the doc someone actually reads before
a run.
§3.13, plus the probe that produced it. Two lessons, both about measuring the
wrong thing confidently.
First: a tune applied AFTER an abliteration can walk it back, and a
reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's
preregistered gate measured none of it — a tune that gains 41 items of
contradiction detection and quietly restores refusals passes every check. The
compliance axis has to be added explicitly.
Second, and this is the trap: measure the axis the abliteration was actually
FOR. Ours was run so the model engages explicit fiction. The probe reached for
mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and
carried a recorded baseline. Different refusal surface entirely, and a model
moves on them independently. 29/100 general-harm refusals on a tune whose prose
the operator was praising at the time is not obviously a defect and may be
desirable: general-harm refusals returning while domain compliance holds is
close to the ideal shape for an internal creative seat. The measurement was
real; its relevance was assumed.
Also recorded, because both were nearly missed:
- Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the
load-bearing number is 71. Stock refused 100/100; near that would mean the
abliteration was undone. 71 complying means partially walked back on one
axis — a different finding, and only one of the two threatens the seat.
- A baseline from a different harness is not a baseline. The recorded 3/100
came from the abliteration tool's scorer, which reads first-token probability
distributions; a probe that generates and regexes is a different instrument.
Run your own against both arms on the same seat or report the number alone.
- A refusal regex undercounts, so classify hard/deflect/comply — and the free
discriminator: if both arms return zero deflections the model is binary; if
only one does, the regex is fine. An artifact does not care which arm it runs
against.
§3.12. brokkr-smithy-dev caught and retracted his own recommendation mid-thread;
recording it before it reads back later as advice.
A common remedy for an overfit tune is a partial merge back toward the base to
recover general capability. The published recipes that recommend it merge into
the STOCK instruct checkpoint. On an abliterated base, following that literally
re-introduces the exact refusal directions the abliteration was run to remove —
and it is silent, because the merged model looks healthier on general benchmarks
while the property the seat exists for quietly returns.
Rule: any merge-back targets the SAME base the LoRA was trained against, never
the upstream stock weights however similar the name.
The wider lesson is about recipe-card provenance. Community cards are
per-checkpoint artifacts and do not transfer across dense-vs-MoE,
stock-vs-abliterated, or size variants. The worked example: a recommendation
carried from a card for a DENSE STOCK 31B onto a MoE ABLITERATED 26B-A4B on the
strength of a shared family name. The overfitting warning on that card happened
to come from the right architecture; the pipeline, reward stacks and merge-back
came from the wrong one. Same family, three axes apart.
So: before quoting a recipe card at a decision, state which checkpoint it was
written for and which axes differ. "Same family" is not an answer.
§3.11. Three consecutive "what about X as a base?" questions in one session,
each answerable in minutes, none of which had been asked before a 7-hour
training window was committed. Writing the check down so it runs first.
1. does it fit for TRAINING - BF16 weights against the real measured peak,
not the weight figure (Gemma-4 is 48.1 GiB of weights and peaks at
79.7 GiB at mb2/seq-16k). Model-line names lie: "Mistral Small 4" is 119 B,
238 GB in BF16, more than both cards combined. QLoRA is not an escape
hatch for MoE - bitsandbytes walks nn.Linear and fused 3-D experts are not
that.
2. if MoE - does the serving engine implement get_expert_mapping. Zero means
LoRA cannot be served at all. gemma4*.py -> 0; deepseek_v2, mixtral,
glm4_moe, ernie45_moe -> present.
3. does the model class support LoRA - and GREP THE CLASS, NOT THE FILE.
Point 3 has teeth and I nearly got it wrong twice in one turn. mistral.py greps
as SupportsLoRA=0 and is fully LoRA-capable via LlamaForCausalLM.
mistral_large_3.py greps as 0 for both and inherits get_expert_mapping from
DeepseekV3ForCausalLM. Capability is inherited; a file-level grep misses it and
only MRO resolution answers it. Same class of error as asserting a substring
instead of an effective value.
Worked results recorded for the three candidates evaluated:
Gemma-4 26B-A4B fits, no expert mapping -> trainable, MERGE-ONLY
Mistral Small 4 119B 238 GB, has mapping -> servable, NOT trainable here
Ministral 3 14B ~28 GB, dense, inherited -> passes all three
Adds a fourth glance at architecture shape, since it predicts how much of this
playbook applies at all: uniform head_dim <= 128 with no sliding window keeps
both flash and cuDNN reachable and makes §3.1/§3.3 moot, while mixed head dims
plus a sliding window is exactly what forces dense O(n^2) attention onto
Ampere-generation kernels for 65% of the step.
§3.10. The quantization playbook already says prove your targets before
spending GPU time; this is the same rule one step later, and easier to skip.
A ~7h LoRA run was built assuming the adapter could be hot-swapped onto a
quantized base at serve time. The sizing doc flagged serving as unsettled and
said the requirement was needed "while he is early, not after the run" — the
concern was identified correctly and then the check was deferred. Tested
afterwards, vLLM refuses outright: gemma4's model class implements zero
occurrences of get_expert_mapping, which process_packed_modules_mapping
requires for any MoE model. One grep, available months earlier.
Two generalisations recorded:
- Feature support is per-architecture, not per-family. LoRA works for the DENSE
sibling of this same model family and not the MoE one, so "model X is
supported" says nothing about X's variants.
- A capability gap in the serving engine cannot be worked around from the
training side. The adapter here never touched experts and was refused anyway,
because the refusal keys on the model being MoE, not on what the adapter
targets.
Includes the mechanical check: grep the engine's model class for the capability,
then start the engine with the feature flag alone — no adapter required, since
--enable-lora forces the machinery to initialise and that is where it fails.
The recovery is cheap here (merge, ~35 min per tune). The cost of finding out
late is that it forecloses an architecture choice after the training window has
already been spent.
The §5 open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops
as it did on vLLM 0.24.0 (#47639), with merged weights as the fallback if it
did. Retested on vllm/vllm-openai:latest against the NVFP4A16 base plus the
live run's checkpoint adapter.
It does not no-op. It refuses to start:
AttributeError: To support LoRA for MoE model,
'get_expert_mapping' must be implemented
And the reason is bigger than the quant. The check is in
vllm/lora/utils.py::process_packed_modules_mapping and branches on whether the
model is MoE — quantization is not in the condition. gemma4.py, gemma4_mm.py,
gemma4_mtp.py and gemma4_unified.py contain zero occurrences of
get_expert_mapping, while deepseek_v2, glm4_moe and ernie45_moe do implement
it. So vLLM cannot serve a LoRA on Gemma-4 at all, BF16 or quantized. Merging
is not a workaround for a quantization limitation; it is the only path for this
architecture.
This holds even though the adapter never touches experts —
validate_adapter_parameters forbids per-expert params, so all 205 targets are
attention and dense MLP. The refusal is about the model being MoE, not about
what the adapter targets.
Worth recording that the current behaviour is an improvement: a loud refusal
beats the 0.24.0 silent no-op, which would ship a base model wearing the tune's
name and pass every check that does not compare against base.
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.
The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.
Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.
Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
stale 365-line one, because training rendered through upstream and the
mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
save_pretrained clamps every prompt forever)
Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
Run-01 was killed at step 19 by operator instruction to root-cause before
spending a ~13.9h window. Two independent methods now agree on where the step
time went, and neither was the hypothesis the consult panel converged on.
Scaling fit (3 points, 2 params, residuals <3ms over an 8x range):
A = 6.87e-4 s/token, B = 8.85e-8 s/token^2
quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384
No fixed term was needed, which refutes launch-bound outright.
Profiler kernel table (device rows only):
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
The attention kernels are sm80 — Ampere-generation CUTLASS running on an
sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the
mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS.
Correctness cleared separately: the sliding mask asserts at max 1024
allowed/row, so the 25 windowed layers were genuinely windowed. The same probe
found that right-padding is what pins the 5 global layers to an explicit 4D
mask and off the is_causal fast path — measured at 9.4% slower for 24% less
loss work at fixed width.
The largest available win is not the attention kernel. The corpus is 29.9%
padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall
clock, no new dependency, unchanged peak memory. Bucket size turned out not to
be a diversity knob — roots per accumulation window are flat across a 256x
range, so the global micro-batch shuffle does that work alone and the bucket
should be tight.
Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic
home (sibling to the quantization playbook), the four probes under
scripts/training-probes/ with raw output kept for re-derivation, and a §6 to
the sizing doc carrying the Gemma-4-specific numbers and round-2 restart
parameters.
Measured negatives recorded so they are not re-chased: grouped_mm (0.9%
slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over
the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever),
FA4 on sm_120.
Round-1 state preserved: 609MB encode cache, order manifest, truncation
report, resume script. No checkpoints — it died at step 19 and the first was
due at 100, so the lora_B inert-adapter gate never ran and moves to the
restart.
Ran the loss path on the real checkpoint on GPU0 with synthetic tokens.
The arithmetic held for parameter counts and was badly wrong for
activation memory.
naive CE bsz1 seq 8192 81.93 GiB
naive CE bsz1 seq16384 OOM
chunked CE bsz1 seq16384 65.66 GiB
chunked CE bsz2 seq16384 79.71 GiB <- the run config
chunked CE bsz4 seq16384 OOM
The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the
~5 GiB estimated: the estimate modelled gradient checkpointing as
storing layer inputs plus a modest recompute peak, and the real MoE
recompute peak with top-8-of-128 routing and its scatter/gather buffers
is far heavier. Dense-model intuition does not size an MoE run.
Two predictions landed exactly — 205 target modules and 74,342,400
trainable params at r64 — which is why the rest of the model of the
thing is still worth trusting.
The headline is that chunked CE at seq 16384 costs 16 GiB less than
naive CE at seq 8192, so chunking is what makes brokkr's 16384
recommendation reachable rather than an optimisation on top of it.
max_seq_len moves 8192 -> 16384 on his truncation finding: the cap
drops 6.2% of samples but 22.4% of tokens, concentrated entirely in
dialogue, which is 60% of the mix.
Also records the four harness changes this required (eitri-smithy
62b556b), including the inert-adapter trap: without
enable_input_require_grads() alongside gradient checkpointing on a
frozen base, no gradient reaches the adapters, every one stays at its
initialisation, and the run completes successfully having learned
nothing.
Operator chose a third placement over the two the sizing offered: rather
than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to
GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no
co-tenant and gen never goes dark beyond its own restart.
Revised run parameters, since a whole card changes them:
- micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving
888 optimizer steps instead of 444. At one epoch the step count is
worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert
per step against ~512 at micro-batch 1 — a far healthier GEMM on
704-wide experts.
- Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock
but pushes activations to ~24 GiB per sequence, which forces
micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow.
- Scriberr stays on GPU1. The previous revision suggested moving it to
GPU0, which was correct only while training was going to live on GPU1.
Records the ordering constraint in both directions, the elway identity
requirement, and that sec's aliases should be allowed to fail at the
gateway rather than be substituted with another model.
The proposed shape was QLoRA r64. It cannot be run as specified. The
checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter
tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj
[128,2816,704] — no .weight suffix, so they are parameters, not modules).
bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params /
42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit
saves ~3.1 GiB of 48.07 and does not error while doing it.
Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per
additional 8192-token sequence.
Two sizing items were absent from the brief and both are load-bearing:
- vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping
30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at
~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a
95.6 GiB card — it starts, then OOMs on the first long sample. Fused or
chunked linear CE is mandatory and must be smoke-proven before a window
is booked, since Liger may not carry a Gemma-4 MoE patch.
- v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the
full-attention layers). A v_proj target silently produces no adapter
there, and k_proj adapts K and V simultaneously. 45.96M trainable at
r64 across q/k/v/o.
Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and
gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on
GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr.
Recommend standing down sec (2 aliases, last request ~5h ago) rather than
gen (7 aliases, 765 busy-engine log lines in 24h).
Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours
at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at
effective batch 16, which makes the wall-clock-checkpointing amendment
concrete rather than hypothetical.
Package as a uv venv on /tank: root is 91% full (36 GB) with
/var/lib/docker on it.
Replaces it with why the remaining segments have no eligible hosts:
two are appliance-only and three have no IPv6 enabled yet, pending the
firewall-policy pass that SLAAC on a client segment would require.
esh-pve-nas and esh-vm-db join esh-docker-vm on 4411:b105, at
:50:55 and :50:60 respectively, each applied by the same prefix-deriving
if-up.d hook so the last two groups read straight off the IPv4 address.
Two obstacles are recorded because both will recur. The Proxmox node had
link-local only despite every relevant sysctl appearing correct, because
its bridge carries per-interface forwarding and the kernel ignores router
advertisements on a forwarding interface unless accept_ra is explicitly
two rather than one. The fix takes the advertised prefix while declining
the default route, so the hypervisor gains an address without any change
to how it routes; this was verified after applying, with the v6 default
route count still at zero.
The database VM refuses key authentication for the privileged accounts
and its unprivileged login cannot escalate without a password, so the
hook went in through the QEMU guest agent from the hypervisor, which
executes as root inside the guest. The document notes the base64
indirection needed to get a multi-line script through intact.
The scheme has existed since August as a single line of persistent
memory, which a snapshot then deleted. It is a naming convention
rather than temporal state, so it now lives in docs/pfi as a proper
document, and the memory entry is reduced to a pointer at it. The
document carries the full table, the address structure, the reasoning
about which slots can and cannot hold a name, and the recipe for
applying one to a host.
It also corrects the conclusion the original note ended on. That note
held that these names could never appear on the wire, which is true
of everything UniFi is able to assign but not of what a host can
assign to itself, and the distinction is the whole difference between
a joke and an address.
AdGuard on esh-docker-vm now holds the esh-server name, at
2607:73c0:402:1d02:4411:b105:50:45, where the segment identity and the
IPv4 address are both legible. It is applied by an if-up.d hook that
derives the prefix at runtime rather than hardcoding it, backgrounds
itself with a retry so it cannot stall interface bring-up, and adds
nothing to the existing interface configuration.
This is load-bearing rather than decorative. The gateway advertises an
IPv6 resolver to clients, macOS prefers it over the IPv4 one, and it
previously pointed at an address derived from that host's MAC.
Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.
Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.
TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:
- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
consumers bind the capability, not a concrete model. Sampler profile copied
from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
config matches the benchmarked condition.
- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
served-name is a contract about what the model IS; answering it with a
different model hides a material change behind a stable string. Operator
ruling: "never repoint a named model at a different model's endpoint --
that is intentionally misleading." Verified: the gateway now returns
HTTP 400 "Invalid model name" for it.
Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.
Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
Measured 2026-08-22 on one target with one instrument: raising MTP
num_speculative_tokens from 3 to 7 improved accepted length from 2.753 to
3.041 per forward pass while throughput fell from 114.9 to 74.0 tok/s.
Reporting acceptance alone would have recommended a 36% regression.
The cause is architectural rather than model-specific. A single-module MTP
head has no depth of its own, so vLLM runs it autoregressively and k draft
tokens cost k sequential forward passes. Past a shallow depth the drafting
cost exceeds what the extra accepted tokens save.
Records the comparison rule that follows: match k when comparing two
speculative methods, or the measurement is of depth rather than method. A
parallel-drafting drafter at k=7 against an autoregressive MTP at k=3 is not
a method comparison. In the case that produced this, the depth control
showed most of the apparent acceptance advantage was depth, while the
throughput advantage was real and came from parallel drafting -- our MTP was
better at position 0 and still lost overall.
Only the measured, model-agnostic result is recorded here. The
DFlash2-specific findings, the hypotheses that remain unproven, and the
wrong turns taken along the way live in
persistent-memory.d/2026-08-22-dflash2-spec-decode.md with explicit
epistemic labels, deliberately kept out of the playbook.
load_calib tokenizes with tok(..., truncation=True, max_length=seqlen). For a
fast tokenizer that mutates the Rust backend's truncation state in place, and
the subsequent tok.save_pretrained() persisted it, so every mixed-NVFP4 build
shipped a tokenizer.json carrying
"truncation": {"direction": "Right", "max_length": 2048, ...}
against a source whose value is null. Every prompt was clamped at the
calibration length, permanently.
It hid because older transformers does not enforce the text-vs-ids count
check. On a newer one the seat dies at startup with a message that names
images and never mentions tokenizers:
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384].
The cap also silently limited image resolution well before it killed
anything -- at 2048 the largest servable image is about 1448x1448, since
(edge/patch)^2 / merge^2 image tokens have to fit under it.
Fix saves a pristine tokenizer re-read from the source rather than the
mutated calibration object, and then asserts truncation is null so the
defect fails the build instead of shipping again.
Playbook gains section 3.14 with the symptom, the cause, the audit one-liner
and a table of which builds were affected, plus a fourth mandatory post-step.
The transferable lesson is called out: this is the third case of an artifact
carrying config authored against an older transformers that a newer one
begins enforcing, so an image bump is a config-compatibility event rather
than just a version change.
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
preetpatel/Qwen3.8-27B-Uncensored-NVFP4 is disqualified on two independent hard
failures, both read directly off the artifacts via HTTP Range requests against the
safetensors header (about a megabyte, not a 20 GB download):
- ZERO mtp tensors. The author's recipe.yaml asks to ignore re:.*mtp.*, but the
written config.json has no mtp ignore entry while re:.*visual.* expanded to 110
explicit ones. That asymmetry is llm-compressor pruning a pattern that matched
nothing, i.e. the MTP head was never loaded. Costs roughly half our decode.
- NVFP4 W4A4, 4-bit activations. Precisely the AEON failure mode: the fidelity
gradient is W4A4 < W4+FP8 < W4+bf16, W4A4 drove ~15-20% stochastic degeneration,
and it collapses past ~30k context. The gen seat serves 262K.
orcarouter/Qwen3.8-27B-Uncensored checks out as a quant source: stock-Qwen base
rather than a reasoning-compression finetune, Arditi-style single-direction
abliteration, 15 mtp and 333 visual tensors verified present, chat template
byte-identical to the heresy build we are serving, and the gate is already accepted
on our token.
Also records the author's FP8 release as a noted-but-not-recommended third option:
far more traction, but 30.9 GB against NVFP4's 22 GB, and on a zero-sum GPU0 that
+9 GB comes out of the KV pool and breaks 262K context.
And states the imatrix constraint plainly. Our recipe has always requested
imatrix_mse and always silently fallen back to uniform MSE; playbook 3.13 warns
against assuming an imatrix would help before verifying llm-compressor can consume
external importance data at all. The W4A16 portions are data-free by construction
and cannot use it regardless.
vh