All four outbound-NAT rules on the FV gateway now match source
10.251.50.0/24 instead of fv-ml1's single address, so a second host at FV
works on arrival rather than reproducing a failure whose symptoms point at
routing rather than NAT.
Anaheim got a /24 rule of its own through the API. The 2026-09-13 ANA rule was
written with write_config and is invisible to source_nat/search_rule, so
leaving it as the only ANA coverage would have kept one destination on a
different code path from the other three. The legacy /32 rule is now redundant
but harmless -- it NATs identically and first-match wins -- and is noted in the
runbook for deletion from the UI, since it is the one rule the API cannot see.
Descriptions rewritten to name the real scope. Three of them said "fv-ml1 to X"
while covering the whole subnet, and a description that understates a rule's
reach is the same trap as the Anaheim-only scope that caused this.
Verified after: fv-ml1 reaches NH3, nh3-dev, ESH, Anaheim, Irvine, the mesh and
the internet; nh3-dev, esh-docker-vm and ana-docker all reach FV and each
other; the FV BMC remains reachable inbound. Pre-change config backup taken.
Also records that 10.251.250.0/24 (BMC/management) is deliberately NOT covered
-- inbound reachability is what out-of-band recovery needs, but a management
host originating traffic to another site would hit this same wall.
FV could not reach any site but Anaheim. The cause was a single outbound-NAT
rule on the FV gateway, added 2026-09-13 and scoped to Anaheim only --
docs/runbooks/fv-to-ana-nat.md says so in as many words: "Other remote sites
remain outside this fix's scope." Three mirrors added, same interface and
source, only the destination differing: 10.100.0.0/16, 10.0.0.0/16 and
10.6.110.0/24.
After: fv-ml1 reaches NH3, ESH, Anaheim, Irvine, the mesh and the internet.
Regression sweep clean across nh3-dev, nh3-docker and esh-docker-vm.
The runbook now records what the failure looks like, because it presents as a
routing or Tailscale fault and is neither. fv-ml1 reached mesh addresses
perfectly and LAN addresses not at all; the FV firewall log showed the outbound
passing with src=10.251.50.54 and no reply returning; temporary counting rules
proved nh3-scale received 5 packets and sent 4 replies; both peers' AllowedIPs
were correct. The discriminator that settles it is that every other site pair
works -- nh3-docker to esh/ana/FV and esh-docker-vm to FV all succeed -- so a
general subnet-to-subnet limitation is ruled out and only outbound SNAT is
left.
Also reverts the remote-site MASQUERADE rules added to nh3-scale earlier on the
asymmetric-return theory. They fired but were not the fix, so they are removed
rather than left to accumulate as NAT that achieves nothing.
Applied via source_nat/add_rule + apply with a pre-change config backup taken
first. Source scope is still fv-ml1's /32, so a second FV host will hit this
again -- flagged in the runbook.
The FV edge firewall was not backed up anywhere. Its config now lands in
/var/lib/restic/stage/fv-gateway-config.xml via ana-docker's pre-backup hook,
so the existing 01:00 restic snapshot captures it. ana-docker is one of the
three egress addresses the firewall's WAN allowlist permits, which is why the
pull lives there rather than with the FV hardware — a site that has lost power
cannot back itself up, and FV lost power two days ago.
Non-fatal by design: an unreachable firewall must not abort the nightly
database dumps. But a bad pull must not be promoted either. The summary loop
only rejects EMPTY staged files, and this endpoint answers an auth failure
with a perfectly non-empty HTML error page — which would have been backed up
as a firewall config that is the right size and restores nothing. The block
checks the body really contains <opnsense> and writes nothing otherwise.
Three tests cover it, including the HTML-error-page case. The first draft of
those tests was worthless: _fv returned a Path out of a TemporaryDirectory
context, so the tree was deleted before the assertions ran and every
exists()-is-False check passed regardless of what the script did. Only the
positive test failed, which is the sole reason the broken negatives were
caught. They now snapshot inside the tempdir's lifetime, and the docstring
says why.
Also records two OPNsense API lessons in docs/pfi/opnsense-api-reference.md:
endpoints are actions and must never be probed for existence by POSTing at
them — that is how /api/core/system/reboot took the FV site dark for 3.5
minutes while looking for an apply call this same file already documented —
and the apply step is service/reconfigure, which auth/user notably lacks, so
an API-only key edit persists in config.xml and does nothing until the OS user
sync runs at boot.
Credentials in /etc/restic/fv-gateway.env (root:600), template committed,
values vaulted as fv-gateway/opnsense-api-{key,secret}. Pre-change config
snapshot vaulted as fv-gateway/config-backup-20260914.
Operator ruling: remove the groups and see what breaks. Nothing did.
ana-docker llmuser sudo+docker -> none; irv-ml1 llmuser sudo -> none (it was
never in docker there). 45 containers on ana-docker and 18 on irv-ml1 all
still running with zero unhealthy, and lora-training-worker stayed active.
Extended to irv-ml1 because it is the same account with the same defect and
gpasswd -a reverses it in one command; ana-docker was only the host the audit
happened to run against first.
The durable lesson is why it was safe, and it is a measurement trap rather
than a permissions one. reported 19 processes on ana-docker
and 3 on irv-ml1, which reads as a busy service account. Nearly all of them
were CONTAINER processes whose in-image UID is 1001 and therefore collides
with llmuser on the host — /proc/<pid>/cgroup shows docker-*.scope. A
container's runtime UID is unrelated to host group membership, so the groups
were buying those workloads nothing. The single real host workload sets
User=/Group= explicitly through systemd, which does not consult the sudo group
either.
Recorded in the conventions doc so the next audit checks the cgroup before
concluding a host account is busy — otherwise a UID collision blocks a
cleanup that carries no risk.
Operator ratified four conventions on 2026-09-14. docs/pfi/fleet-conventions.md
is the pin; playbooks/audit-host-conventions.yaml is its instrument.
Pinned, verified free on all eight surveyed hosts (dynamically-allocated
system accounts cluster in 989-999 and descend, so 800-899 is safe):
800-849 svc-* service accounts
850 infra-ops uid+gid
851 docker gid
852-899 reserved for fleet-wide groups
1000 the human account (vh)
Deliberately a pin for NEW hosts, not a migration mandate. The UID drift
(infra-ops is 1001/1002/1003/2001) is tolerable because there is no central
identity anywhere and a UID only has to agree where files cross hosts. They
do on /mnt/smithy — but that export is owned by Synology UIDs that resolve on
neither host and is 0777 throughout, so cross-host sharing works today
BECAUSE permissions are wide open. Aligning UIDs does not fix something
broken; it earns the right to drop that 777. Recorded as such rather than as
an urgent defect.
The audit playbook reports and never enforces, so a standard cannot quietly
become a flag day. Verified against nh3-dev, ana-docker, corviduo-dev and
nh3-extdev; it immediately surfaced two things the survey had missed —
llmuser holds sudo AND docker on ana-docker, and seven stacks on corviduo-dev
run from outside /opt/docker/compose (three under /home/vh, four under /opt,
including the three CI/CD-driven Worldtree deployments that must not be
moved).
Also supersedes the CLAUDE.md posture that made corviduo-dev the one host
excluded from fleet normalisation: the operator ruled all ops on it belong to
infra-ops. Its application layer stays CI/CD-owned.
Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology
(cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative
seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index
entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md.
Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its
memory detail) so memory does not lag the work.
Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4
age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all
other entries are <14 days old. Reported per the archival discipline.
Adds docs/pfi/llm-seat-catalog.md, the durable hand-curated record of what each
seat IS -- lineage, provenance, model-card facts, quantization, speculative
decoding, licenses, and measured warm tok/s + deep-prefill depth results with
their harness and date. It complements the auto-generated
fv-ml1-gpu-seat-inventory.md (live placement/KV/concurrency): the two split by
volatility, and the catalog defers to the inventory for any live number.
Adds scripts/seat-bench.py so the catalog's speed/depth numbers are reproducible
(warm decode tok/s, n=3, conc=1, median; deep prefill at ~0.97x max-model-len
with an allocator-log OOM scan). Serial by design -- concurrent deep prefills
would confound both OOM and tok/s.
Captures the 2026-09-14 measurements: all six generative seats prefill to ~255K
(coder ~16K) with zero OOM/CUBLAS/illegal-memory; warm decode 62.7-337.3 tok/s;
per-seat VRAM. seat-inventory.py now cross-links the catalog in its footer.
Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched),
operator-directed. Every target seat now serves native 262,144 context with
concurrency in the requested 1.2-2.5x band, verified from live boot logs:
cyberprev (sec) 262144 @ 1.37x depth-probed CLEAN to 259,722 tokens
flash-next (gen) 262144 @ 1.31x (untouched, already in band)
gen-small (NEW) 262144 @ 2.56x MTP k=3 measured 69.6% accept / 3.09 len
char-rp 262144 @ 1.22x (was 1.02x; util 0.40->0.52)
char-rp-fast 262144 @ 2.04x (util cap 0.30->0.24, pinned KV unchanged)
- gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic
(MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4.
There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so
this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent
and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases.
- coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled,
waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB)
sits just under the util knob's resolution, so hitting <=3x reliably needs a
--kv-cache-memory byte pin (compose change) rather than the util fraction.
- cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to
259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch,
capped at 163840 for depth crashes), this checkpoint holds native depth.
- Gateway (operator calls): summarizer + classifier -> gen-small; new
classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large
stays on flash. All verified end-to-end.
- GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's
reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than
moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a
single-seat move messier than the in-GPU rebalance.
Seat inventory regenerated from the live containers.
Live-read regeneration after promoting cyberprev into sec/sec-reasoning and
retiring mog-sec. GPU0 now carries cyberprev alone; sec/sec-reasoning aliases
resolve to :8025.
The first cyberprev quant served pure gibberish despite passing every structural
check. Root cause: the hotdogs/Qwen3.8-27B-abliterated-cyber-preview checkpoint
ships broken -- 850 of 1199 tensors (the whole transformer body) carry a TRIPLED
`language_model.` prefix, confirmed present in the upstream HF index (an unsloth
export bug, not our download). transformers silently loaded the unmapped tensors
as random-init, the pipeline quantized noise, and structural verification passed
because it inspects the output config, never that input names mapped. A plain
greedy coherence smoke caught it; nothing structural could.
Fix: rewrite tensor names against a known-good reference of the same architecture
(qwen38-27b-uncensored-bf16), verified the repaired name-set == reference name-set
exactly (1199 tensors, 0 collisions), and re-quantized from the corrected source
(cyberprev-bf16-fixed). v2 is coherence-verified (spec off AND live with dflash):
Paris/Berlin/Rome, correct arithmetic precedence, primes, clean cyber-domain chat
with a proper thinking split. dflash k=7 measured at 2.77 mean acceptance length.
Seat serves cyberprev-27b / cyberprev-27b-thinking on :8025/GPU0 with dflash k=7,
displacing the retired sentinel-r3. Gateway: removed the retired sentinel-r3 /
sentinel-r3-reasoning aliases and added cyberprev-27b / cyberprev-reasoning -- NOT
repointed, since serving cyberprev weights under sentinel's name is silent
substitution. Verified end-to-end through the gateway; sentinel-r3 now 400s.
Playbook gains a superseded-claims row: structural verification does not imply a
good quant; a coherence smoke is the load-bearing gate, and third-party sources
must have their tensor names diffed against a reference before quantizing.
Broken v1 quant dir removed.
Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.
The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.
orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.
A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.
Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.
Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.
Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000,
GPU caps 275W/card). All-night fv-ml1 seat reorganization:
- flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1),
inverting vLLM's 4xH100 recipe; KV 14->10 GiB.
- gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases
repointed); 27B dense gen seat retired, 38 GB freed on GPU0.
- char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a
leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier.
- Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7
cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm).
gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded
(170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE;
the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell
image). Runtime decision pending -- this is the resume point.
Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift
alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad.
Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.
The seat documentation must stay current, and a hand-written document cannot.
The LiteLLM config described char-rp as a 31B model on a host and GPU it had not
been on since 2026-08-24 -- three weeks of silent drift in a file that read as
authoritative, and the reason a seat spent that period serving a model nobody
intended. Anything typed here drifts the same way; anything read off the running
containers cannot.
scripts/seat-inventory.py derives the whole document from the host:
- placement and VRAM from nvidia-smi compute-apps, mapped to containers through
/proc/<pid>/cgroup -- nvidia-smi reports the vLLM engine child while docker
reports the container pid, so matching them directly silently yields nothing
- weights and KV tokens parsed from each engine's own startup log, not derived
arithmetically, with concurrency computed as KV tokens over context
- architecture, layer and expert counts, and the exact quantization group scheme
(W4A4 vs W4A16 distinguished) from each model's config.json
- speculative-decoding method and k from the container argv, which is how the
three incompatible methods on this box became visible
- lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model
directory, not inside it, which is why an earlier pass wrongly reported two
fully-documented seats as having no provenance
- gateway aliases resolved from the LiteLLM config on ana-docker
--check compares the committed document against the live box and exits non-zero
when they diverge, ignoring only the generation timestamp. Suitable for CI or a
scheduled drift alarm; read-only throughout, safe against production.
Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which
asserts the flag exists in the derived argv and aborts rather than running a
campaign that silently ignored it.
Read from docker inspect, nvidia-smi and each model's own config.json rather
than from compose files or the gateway config, so the document records what is
deployed rather than what was intended.
Findings worth naming:
- char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on
ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer
128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it,
but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment
block carries sampler defaults keyed to the model it names.
- Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base
(:8099). Nothing is listening on either; callers get a connection failure
rather than a clear retirement error.
- Three different speculative-decoding methods coexist -- dflash k=7 with a
separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next.
The method string is architecture-specific; copying a --speculative-config
between seats without changing it will not work.
- Three seats run the floating :latest tag, so what they would pull on recreate
is not what is running and was never recorded.
- Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw
during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn.
Also commits the flash-next-seat MTP k=3 change made today: compose gains the
speculative-config with its measured rationale, and .env.example documents the
KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
I claimed in 100670e that DCGM's config enforcement was plausibly gated to datacenter
SKUs and told the operator not to plan around it. That was a guess presented as a caveat
and it is wrong. Verified against NVIDIA's own documentation at the operator's request.
Supported platforms explicitly cover 'All NVIDIA Maxwell and newer non-datacenter (e.g.
NVIDIA GeForce or NVIDIA Quadro) GPUs', and the feature-overview table marks
Configuration Management as supported for Tesla, Titan, Quadro and GeForce alike --
where Configuration Management explicitly includes 'Power Limit: Set the maximum allowed
power consumption'. What is actually gated on non-datacenter cards is diagnostics: Level
1 only, against All Levels on Tesla. Configuration was never the restricted part.
One soft edge retained rather than papered over: the table says 'Quadro', the former name
for the professional line, and RTX 6000 Ada / RTX PRO 6000 are its successors, so placing
them in that column is inference rather than quotation. One command on the box settles it.
What does not change is the distribution question. DCGM_CONFIG_POWER_BUDGET_GROUP is
available to us, but the docs still never state how a group budget is divided, and the
NVML argument is untouched -- there is no bank-level register, so it resolves to per-GPU
writes either way and the likely finding is static even division, which is exactly
4 x 250 W. The experiment is therefore promoted from curiosity back to a real test.
DCGM is NVIDIA's own Data Center GPU Manager -- first-party, Apache-2.0, packaged as
datacenter-gpu-manager -- and it layers above NVML rather than beside it: nvidia-smi is a
thin CLI over NVML's per-GPU primitives, and DCGM is a daemon plus dcgmi adding health,
diagnostics, config enforcement, policy and group abstractions on top. Which is why its
group notion still resolves to N per-GPU writes underneath.
The caveat that matters, and it undercuts the experiment suggested in the previous
commit: DCGM is datacenter-oriented and parts of it are gated to datacenter SKUs of the
Tesla/A100/H100 class. Our cards are professional/workstation parts -- RTX PRO 6000
Blackwell Max-Q and RTX 6000 Ada -- and several DCGM capabilities are unsupported or
degraded outside that line, plausibly including config enforcement, which is precisely
the power path. So DCGM_CONFIG_POWER_BUDGET_GROUP may return 'unsupported on this
device'. Downgraded from 'worth testing' to five minutes of curiosity after the real
work, and explicitly not a planning assumption.
None of which touches the plan: nvidia-smi -pl 250 is plain NVML and works on these
cards. DCGM would only have bought the group-budget experiment and nicer telemetry, and
is probably not installed anyway since beszel-agent-nvidia shells out to nvidia-smi.
DCGM_CONFIG_POWER_BUDGET_GROUP ('the power budget for the entire group') exists
alongside DCGM_CONFIG_POWER_CAP_INDIVIDUAL, so the concept is first-class. The docs do
not state how a group budget is distributed, and the deduction is that it cannot be
anything exotic: the only enforcement primitive underneath is NVML's per-GPU
nvmlDeviceSetPowerManagementLimit and there is no bank-level register, so any group
budget resolves to N per-GPU writes. Static even division is one write each; 'each card
free until they are all loaded' requires continuous re-writing, which is a control loop
rather than a hardware feature. Worth a ten-minute test when the box returns, in case
NVIDIA already runs that loop.
Records the design constraint that matters more than the logic: power readings lag and
-pl application takes tens of milliseconds, so a reactive daemon overshoots during a load
ramp -- and the ramp is the dangerous moment, being the same all-cards-at-once shape as
this box's ten restart:unless-stopped containers starting together. So any such loop must
be safe-by-default and opportunistic upward: boot at budget/N, only ever raise after
observing idle neighbours. Inverted, it works for weeks and then fails on precisely the
event it existed to prevent.
And the reason to defer it: 4x250 W is already 1000 W, so the static cap is the
conservative floor of the dynamic scheme rather than an alternative. The daemon's entire
contribution is the one-card-busy case, worth perhaps 5% throughput, which is rare for a
serving fleet that puts one seat per card and common only for a training window.
Operator set fv-ml1 at 250 W per card (83% of TGP, ~5% throughput) and ana-ml3 at 200 W
(67%, ~10-15%). Records the term that decides whether 250 W actually clears a 15 A feed,
because it is easy to drop: a power limit bounds BOARD power, and the wall sees that
divided by PSU efficiency.
Four cards at 250 W is 1000 W of board; add 180-300 W of host components and divide by
~0.90 and the plug sees ~1310-1445 W, against a 15 A circuit's 1440 W NEC continuous
derating -- an inference box serving all day being a continuous load. So 250 W lands ON
the limit rather than under it, where 200 W would give ~1090-1220 W with real margin.
The deciding term is the host draw, which is still an estimate, so the procedure is: set
250 W, verify at the plug under four-card load, fall back to 200 W if it reads near
1440 W. A cap is a claim; the ammeter is the verification.
Two consequences recorded alongside. Caps bound sustained draw and not transients -- the
enforcement window is short but not instantaneous -- and while a breaker's
thermal-magnetic curve forgives brief overload, a UPS's overload protection does not. So
250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA
unit, which it exceeds even capped. And ana-ml3's 200 W across only two cards is
deliberately conservative at 400 W total, relaxable if Anaheim's measured headroom beats
its trip history.
Operator clarification, and it separates two boxes I had been conflating. fv-ml1 is
4x Blackwell RTX PRO 6000 Max-Q at 300 W each (Max-Q being the reduced-TGP SKU; the
Workstation Edition is the 600 W part), 391 GB VRAM, deployed and currently dark.
ana-ml3 is 2x Ada Generation RTX 6000 at 300 W, 96 GB VRAM, not yet deployed. The 200 W
cap directive is ana-ml3's.
With the TGP known, the outage stops being a vague 'undersized' and acquires a
mechanism: two Max-Q cards at 300 W is ~600 W of card, plus a host carrying 566 GB of
RAM, drives, fans and PSU conversion loss at perhaps 200-350 W, against an Eaton 1500
VA's real ~900-1200 W. That lands at or just over the rating, which is precisely what
explains a full day of service on one card and failure minutes into the second. The host
term is the only one being guessed; idle-at-the-plug measures it directly.
It also surfaces something that is not a UPS question at all. Four cards at 300 W plus
~300 W of host is ~1500 W against a 15 A circuit's 1440 W continuous derating, so four
cards uncapped is marginal on the breaker with no UPS in the path. Capping therefore
belongs at fv-ml1 as well as ana-ml3, or fv-ml1 needs a 20 A feed -- and worth noting
today's incident only ever had two of the four cards working.
ana-ml3's placement constraints sharpen too: sm_89 has native FP8 but no NVFP4, so the
in-house NVFP4 quants stay at FV, and at 96 GB total it cannot host the Flash-Next seat
at all -- that needs 74 GiB resident on a single card, and the offload moves the n-gram
table rather than the experts.
Corrects the SKU: RTX 6000 Ada, 300 W, not the ~600 W initially recalled. That makes
200 W a cap to 67% of TGP -- the favourable part of the concave perf/watt curve, roughly
10-15% of throughput -- rather than the severe 33% cap a 600 W part would have implied,
and it very likely sits above the card's enforceable floor, so the check becomes a
formality rather than a gate.
The protective value is worth stating: four cards at 300 W uncapped is ~1200 W, which is
roughly the neighbourhood that overwhelmed a 1500 VA unit at FV with only TWO Blackwell
cards drawing. Capping to 800 W makes a repeat of today a non-event.
Two consequences that follow from Ada independent of power, and both are placement
constraints rather than details. sm_89 has native FP8 but NOT NVFP4, which is
Blackwell-only -- so the in-house NVFP4 quants that most of this fleet runs will not be
accelerated on that colo's cards, and its seats want FP8 W8A8 builds or the NVFP4
checkpoints stay at FV. And it unparks the triton-backend item, which is a hard no on
Ampere because fp8e4nv is unsupported on sm_86 and was explicitly deferred to Ada;
sm_89 has what it needs.
VRAM is 4x48 = 192 GB against fv-ml1's 391 GB, so big-model placement stays at FV. The
Flash-Next seat needs 74 GiB resident on one card and would not fit a 48 GB Ada card
even with the n-gram table offloaded -- the offload moves the table, not the experts.
Operator directive, and the right generalisation of the FV outage: decide the power
envelope first and size the cards into it, rather than installing cards and discovering
the constraint by tripping it. Four cards at 200 W is 800 W, which fits a real circuit
with a real UPS and headroom.
Records three things to settle before it is a plan. First, 200 W may sit below the
card's enforceable floor: nvidia-smi -pl is bounded by Min Power Limit, often around
half of TGP on a high-TGP part, and a sub-floor request is refused -- quietly, depending
on how it is scripted. Run nvidia-smi -q -d POWER before any build planning depends on
the number.
Second, the 600 W figure wants confirming against the actual SKU. The Ada parts do not
land there -- RTX 6000 Ada is 300 W, L40/L40S 300/350 W, 4090 450 W -- while 600 W is
Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or the figure
may be a two-card total. Read it off the device rather than a spec sheet.
Third, the workload asymmetry is in this fleet's favour: decode is
memory-bandwidth-bound and tolerates a cap far better than training does, with a concave
perf/watt curve where 60-70% of TGP costs roughly 10-15% of throughput. A cap to a third
of TGP is deeper into the steep region; measure it on the first card rather than
predicting, and expect prefill-heavy and training work to pay more than a serving seat.
And persist the cap. A hand-set limit holds until the next reboot and then silently
stops holding, which is the worst shape available given that the thing rebooting the box
is likely to be the power event the cap existed to prevent.
Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits
into two halves with different answers.
A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W
on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a
load-bearing supply and goes back to being what it is for. What it does NOT cover is the
colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just
tripped, the outcome is a firewall running on battery with nothing upstream to talk to
and the drive happens anyway. Added as a question for the facility, because it decides
whether split power delivers remote diagnosis or merely feels like it does.
Records the case where none of it matters: removing an undersized UPS does not remove
the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A
or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which
is what makes that single ammeter reading the load-bearing measurement of the visit.
Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card
TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring
cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room
the dial has, and persist any cap -- one that evaporates on reboot will hold right up
until the next power event and then silently stop holding.
Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.
Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.
The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.
Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34
minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left
uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding,
and nobody should buy hardware on it until the discriminator below has been read.
On the evidence that change is the wrong shape to have caused this, and it is recorded
as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a
single source /32 and a single destination /16 cannot stop the gateway, the BMC or the
public WAN address from answering inbound; no routes, filter rules, WAN settings or
subnet advertisements were touched; pfctl -sr came back byte-identical; and it was
verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up.
Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50,
a different subnet. They correctly declined to claim BMC health, but the observation is
void rather than negative and should not be reasoned from.
The discriminator costs nothing and is already on disk: power.log is written locally to
/tank every 10 s by a shell loop on the box and does not depend on the network.
Entries past 06:56Z mean the machine never lost power, which makes this a routing fault
and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime
and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory
outright.
So the first action on site is now to READ, not to fix. The two hypotheses lead to
completely different remediations and only one of them needs a new UPS.
Durable capture so tomorrow's session does not have to reconstruct either half.
Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single
RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA --
74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144
context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end.
The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes
the paused worker-based #53899 and designs out its entire bug family;
text_config.ple_embedding_dtype is the load-or-fail discriminator for any community
build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore
gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin
nearly OOM with no visible failure; MTP is off pending measurement here rather than
written off, because the recipe's number is cross-harness and tested k=3 only while the
head is one layer run autoregressively; and a container once reported (healthy) with no
published port at all, because the healthcheck runs inside the boundary it was trusted
to validate.
Then the outage. Records it as will-not-self-recover, so no session wastes effort
polling a dead site, and carries the three things that change the visit: bypass the UPS
rather than using its surge-only bank (both banks share one 12 A inlet -- the surge
bank bypasses the inverter, not the current rating), recover power.log before anything
else because it is the only load measurement that exists anywhere, and bring seats up
one at a time because ten restart:unless-stopped containers loading at once is the
largest transient the box can make into whatever just failed.
Also records what is still half-done: the stale homepage labels on the 10 containers
that died before they could be recreated, which the staged bring-up fixes as a side
effect, and the eight drifted stacks plus three untracked host-only stacks that were
deliberately left for a deliberate reconciliation.
Operator's read is that the UPS the box was plugged into overloaded and died, and it
fits better than the breaker-trip theory: a UPS's output rating sits far below the
circuit's, so it is the first protective device to give -- which explains why the site
let go at TWO cards loaded rather than four, and why the ~25 W firewall died with it.
Records the operationally important consequence: a tripped UPS resets, an overloaded
one can kill its output stage permanently. If it is dead, nothing on site can be reset
back to life, so the visit needs the means to BYPASS the UPS or it is wasted.
Elevates recovery of /tank/.../power.log to the first action on site. It sampled all
four cards every 10 s up to the cut, lives on /tank rather than in a container, and is
the only measurement of what the load actually drew -- without it a replacement UPS
gets sized by guesswork. Also states that no load figure exists yet, only idle.
Corrects an earlier claim of mine in this session: there is NO local fallback for the
19 dark aliases. Probed -- every free local model is on fv-ml1, and irv-ml1 runs no
chat seat at all, only TTS/ComfyUI/arbo/clipper work on two partly-occupied Ampere
cards. The only non-fv chat backends are paid. Any paid coverage must go under a new
opt-in alias name rather than a silent repoint of summarizer/gen/classifier.
Written while the site is down so recovery does not have to be reconstructed later.
Records what was measured rather than what is suspected: every FV address including
the BMC is unreachable while all three other sites answer, the campaign's last log
line was off_A rep 2 at 06:56:04Z, and the site was dark by 06:58:40Z.
Names three candidate causes with the evidence that would distinguish them, because
the instrument that could have settled it -- the per-card power log -- died with the
box. The two-card-load hypothesis fits the timing and the two prior Anaheim breaker
trips on this same chassis, but it is circumstantial and is recorded as such.
Carries the recovery hazard that matters: every seat on the box is
restart:unless-stopped, so resetting power alone brings ten vLLM containers up
loading at once -- the largest transient the box can produce, into a circuit that may
have just tripped. Staged sequence given, gen first and flash-next last.
Also records the OOB gap the outage exposes: OPNsense-as-subnet-router protects
against box-down/gateway-up, and not at all against the site-wide loss that actually
happened, because the BMC's only path out is through that same gateway.
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.
Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.
The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.
LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.
Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.
DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.
Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
fv-ml1-cutover.md covers what to change; this covers the on-site hour, where the
binding constraints are limited time and not locking ourselves out of a site an
hour away.
The load-bearing correction is that fv-ml1 running tailscale is NOT a second
independent path. Its default route is the OPNsense LAN address, so its node
needs egress through OPNsense to reach the control plane; OPNsense down means
fv-ml1 is unreachable by construction. An earlier version of this plan claimed
otherwise and proposed proving it by rebooting the firewall, which would have
failed on site with the operator standing at the rack. He caught it.
What the fv-ml1 node does buy is narrower and still worth one command: it covers
control-plane lockout -- a bad admin rule, a WebGUI on the wrong interface, a
lost password -- where the box still routes but cannot be managed. It does not
cover data-plane failure. Those are different failure classes and conflating
them produced the wrong test. The gate is now a deny-admin-from-mesh test run
while the console is in arm's reach, which exercises the bypass that actually
exists.
Deliberate deviation from the cutover runbook: flat /24 with the BMC at
10.251.50.50 rather than a mgmt VLAN at 10.251.250.50. The VLAN needs OPNsense
config plus switch tagging during the scarcest hour, for something addable
remotely in ten minutes.
Also records the DNS chicken-and-egg (the mesh needs public resolvers because
the fleet AdGuard is unreachable until the mesh is up) and the interface
auto-assignment trap measured on the lab VM the same night.
Plex never hardware-transcoded on esh-pve-nas LXC 105 despite correct passthrough,
cgroups, group membership, authenticated HuC firmware, a lifetime Plex Pass,
HardwareAcceleratedCodecs=1, and the Arc already selected as HardwareDevicePath.
Root cause sat below all of that: intel-media-va-driver 22.3.1 (Apr 2023, stock
jammy) predates Arc/DG2 support and exports only __vaDriverInit_1_14, against the
libva 2.22 that Plex bundles and loads via RPATH rather than the system one. A
half-finished prior attempt at the same fix was also present -- libva and libva-drm
hand-installed at 2.22 with libva-x11 left at 2.14, breaking every X11 VA-API
consumer on va_fool_postp.
Fixed with Intel's client-GPU repo: iHD 24.3.4 (__vaDriverInit_1_22, an exact ABI
match) plus a consistent libva 2.22.0.2-87 set, which also brings the orphaned
manual install back under dpkg. The repo track is rolling, so the six packages are
pinned in /etc/apt/preferences.d/intel-gpu-pin and apt-mark held; verified by a
simulated upgrade that moves 152 packages and touches none of them.
Two findings worth more than the fix:
pct snapshot refuses on a guest with a bind mount and still exits 0, so a script
guarding a change with it proceeds without the rollback point it believes it has.
The ZFS dataset snapshot is the working path, verified by reading it back.
A synthetic Plex Transcoder invocation is not a valid test of Plex's transcode
path. Plex bundles its own libc among 61 libraries; the harness produced three
distinct failure modes that were artifacts of not reproducing that runtime, and it
failed identically before and after a fix that worked. With no positive control its
negatives carried no information. Only a forced transcode settles it, and PASS is
recognisable by Plex naming the device. The original empty decoder/encoder line was
an absence of evidence rather than evidence of failure -- TranscodeSession was 0.
Jellyfin LXC 107 has the same stale stack and the same Arc available; left alone
per the operator, and it ships its own ffmpeg so this may not transfer verbatim.
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and
create an alias for char-rp-fast."
G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under
its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone
from the gateway and now returns an explicit 400 rather than a substitution; 0 of
17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays
erp-seat because asset-engine derives seat liveness from it.
The first quant of that A4B served NaN and passed its healthcheck doing it. It was
built with the dense v2-31B recipe, whose ignore list has no router regex, so all
30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts
run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool,
correct served name, and every completion returned finish_reason=length with the
full token count and content: null. The model was emitting a full budget of tokens
that decoded to the empty string. Raw /v1/completions was empty too, ruling out the
chat template and the reasoning parser. The signal that named it was logprobs:
vLLM refused to serialize the response, "Out of range float values are not JSON
compliant: nan".
The lesson is about the control rather than the router. That tree had already been
structurally diffed and passed -- against a verified-good DENSE quant of the same
Gemma-4 family. A dense model has no routers, so the single thing that was wrong
was the single thing the control could not distinguish. Diffing instead against
Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222
ignore entries against 252, the 30 missing being layers.N.router.proj. A positive
control is only worth what it can distinguish, and "same family" is not "same
architecture class".
Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and
refuses a router in the quantize set before any GPU time. The live seat then passed
prose with no channel-prefix leak, a solid-colour image read correctly, an auto
tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from
Pfish-6 unchanged. The broken tree is parked on ana-ml2 as
...-NVFP4A16.BROKEN-routers-quantized-20260910.
Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus
multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's
six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder
profiling. The substitute was reversibility and ordering: named .env backup, prove
the seat on its real port while no alias points at it, move the alias last. That is
why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes
across two attempts; no consumer saw a broken alias.
Playbook gains the router-quant failure signature and the control-class rule in
3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6.
Quality is NOT established: no RP eval, no long-context check, no A/B against
Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G.
The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError`
on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a
`per_layer_config` key zerofata's canonical one lacks -- and the standing fix was
to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong.
`pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The
config was serialized by 5.16.1, which materializes `per_layer_config` from
`global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not
the gemma4 resolver. Under the image's own transformers the same config loads
fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the
toolchain moved mid-diagnosis. Two things separated "malformed upload" from
"moved toolchain": reproducing the real failing call (a bare AutoConfig load does
not reproduce it; the trigger is reached through AutoTokenizer) and keeping
zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control.
The fix drops `per_layer_config` rather than forcing global access. It is exactly
redundant -- keys are precisely the ten full_attention layer indices, sole value
(512, 4), verbatim the global fields -- and forcing instead would make
`config.head_dim` answer 256 to the callers building the 512-wide layers.
patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever
stops holding.
Verified on the tensor table rather than the exit code: the output is identical
family-for-family and count-for-count to the August canonical quant, with 356
BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load
leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test
has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a
live seat displaced.
Also fixes the A4B output, which had a truncation cap baked into its tokenizer
(max_length 8192) from being quantized with the calibration corpus.
Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16
to say drop the dataset outright for any A16 scheme.
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.
pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final, merged-run02, merged-run03,
merged-run03-s{025,050,075}, merged-test,
nvfp4a16-test 8 dirs -> 354 GB
gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.
The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.
Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.
The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
The operator asked brokkr-smithy to look for a smaller Qwen3.6/3.8; brokkr reported
none exists and that both 27Bs are the same hybrid kernel path one size up. Verified
independently against the HF API rather than adopted, and it holds -- with two facts
brokkr's note did not carry.
1. No official Qwen3.6 or Qwen3.8 below 27B. The listing is Qwen3.6-{27B, 35B-A3B}
and Qwen3.8-{27B, 2.4T-A95B, Flash-Next}.
2. Neither family publishes a -Base checkpoint AT ALL. Every Base newer than Qwen3
is Qwen3.5. Since the regime requires a non-instruct carrier, that rules the
3.6/3.8 lines out on its own, independent of size.
3. Qwen3.6-27B and Qwen3.8-27B both report model_type qwen3_5, 64 layers as 16 full
+ 48 linear-attention, vision tower present -- confirmed as stated.
So the dense Qwen3 ruling is not a compromise against a better available option; it
is the only dense Base option. Also recorded as insurance, not a proposal:
Qwen3-8B-Base and Qwen3-14B-Base exist, so the sweep can extend past 4B without
changing family if the carrier curve has not flattened there.
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
The base-viability pre-flight had three checks (fits / MoE expert mapping / LoRA
support) and would have passed Qwen3.5-0.8B-Base clean while it trained 2.6x
slower than a dense model 2.3x its size. Check 4 closes that: read `layer_types`
for a linear_attention majority AND probe for mamba_ssm / causal_conv1d / fla /
kernels. It is the intersection that is slow -- a hybrid shape with the kernel
present is fine, a dense shape does not care.
Carries the measured table (gx10 GB10, n=10/arm, spreads 0.6-2.6%), plus the two
things a hybrid Base checkpoint brings that a dense one does not: a vision tower
and MTP head that target_modules="all-linear" would train on text, and the module
rename that AutoModelForCausalLM introduces relative to the vLLM serving class;
and unsafe cross-document packing, since SSM state ignores the attention mask.
Section heading corrected from "three greps" to "four checks". The example was
made runnable and verified on the box rather than shipped untested.
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is
retired (its gate failure was the detector bug fixed in cc42d76, but the run was
independently poor).
- served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6
- ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact
- SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails,
'kernel does not support current device'); Marlin is correct for a weight-only
scheme, and vLLM's 'no native FP4' warning is about the scheme not the card
- the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s,
3.2x the old ceiling, with single-stream latency unchanged
- head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363
rows, counting by the artifact gave 77, because the alias had carried three
different models that day. Wrong in the direction that looks careful.
From a near-miss brokkr-smithy-dev caught and declined to send: a post-fix
verification read raced a gateway restart and reported the alias still present.
A disagreement between two observers is not a finding until the boring
explanation is ruled out, and during an incident the pressure runs the other way.
- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
(infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
The empty-snapshot promote and brokkr-smithy-dev's -s-on-a-touched-sentinel are
the same defect from opposite sides. Records the direction split (false
reassurance / false refusal / false alarm) because the mitigations differ and a
post-mortem that lumps them loses the actionable half.
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
vh