svos-dev observed that three instances of the same shape turned up between two
agents in one night and that it is starting to look like a characteristic failure
rather than a coincidence. Collecting all nine from today, because the class is more
useful than any instance.
The shape is a check that reads the input to a transformation and gets reported as
if it read the output - or more generally, the instrument answering instead of the
system, in a form shaped exactly like a real answer. What makes it expensive is not
that things break but that the broken state is indistinguishable from a legitimate
one, so it passes review and is found later by accident. Every one of the nine
passed a check.
The tell is stated so it can be recognised prospectively: whenever 'broken' and
'legitimately empty, absent or off' produce the same output, the cheap check cannot
tell them apart by construction.
Remedies that actually worked today: measure the output rather than the input;
positive controls, since a method that has only ever passed cannot tell you it is
not blind; true-negative controls, because two apparent failures in the
secrets-broker test were names I had invented and would have been read as a partial
fix; refuse to emit the ambiguous value, which was the real fix rather than the
lock; and do not declare victory on a plausible fix, which is the only reason the
session-establishment root cause was found at all.
They adopted the gate as tts-stack tools/gate_served_page.py and extended it in a
place that matters: my version would have passed a broken page.
A worklet lives inside a template literal, so a syntax error in it is invisible to
a parse of the enclosing script - it is just a string until addModule compiles it
at runtime, where it fails as a rejected promise and the page quietly falls back to
buffered playback or records nothing. Silent degradation, which is harder to notice
than a dead page rather than easier. They parse the worklet separately, and they
positive-controlled the whole thing against two deliberately broken pages rather
than assuming a gate that has only ever passed is not blind. The second control -
valid enclosing script, broken worklet - is the one my version fails.
The lesson on my own work is the useful part: I built a gate for the failure I had
just been shown and stopped at its boundary. The class is 'code that is a string at
parse time and code at run time'; an inline script is one instance and a
template-literal worklet is another. I checked the instance, not the class.
Also promotes the underlying rule to the index, since it was named twice tonight
from two unrelated directions: a check that reads an artifact as stored cannot see
a transformation that happens between storage and execution.
svos-dev restarted :8770 at 02:17 and both roster lines printed clean. Confirmed
from this side rather than taken on their word: :8770 answers 200 on the new pid,
an unauthenticated Bifrost dispatch gets 401, and Hermes reports 29 toolsets with
svos_miranda the sole enabled=True row.
Two patterns from their restart that generalise past this service.
A dry-run boot against the still-held port: start the new process while the old
one still owns the socket, and it proves every check above the bind before dying
on EADDRINUSE. Zero downtime, no commitment, and it turns a one-way restart into a
rehearsed one. Worth doing for any service whose startup validates before binding.
And a trap: SIGTERM released the port but left the process alive for 35 seconds,
needing SIGKILL. The port was free that entire time, so a script waiting on port
availability would have started the replacement alongside a still-running old
process. Kill by PID and wait on the PID, never on the port - a freed port is not
evidence of a dead process, the same way an unreachable post office is an outage
rather than an empty inbox.
Four operator rulings executed.
svos_miranda is live in Hermes. Gateway restarted 02:10 (PID 3107822 -> 3901622,
confirmed by observing the change). /v1/toolsets now reports 29 rows including
svos_miranda, and an api_server session resolves to exactly the 8 plugin tools
with the write-klass five absent.
agent.disabled_toolsets stays off permanently: 'i dont want the tools disabled
everywhere'. That key is a global end-of-pipeline subtraction rather than an
api_server-scoped one - measured, a default session goes 46 tools to 20 - and it
is unnecessary anyway, since platform_toolsets.api_server alone produces the exact
8-tool surface. The operator's own session was verified intact at 46 tools after
the restart, which was the point of the ruling.
The consequence is now SVOS's to absorb: it must stop verifying against the global
roster before it restarts, because that roster is 29 by design and will not shrink.
Two workable options went to svos-dev - verify the api_server surface instead, or
relax the check to 'svos_miranda present and write-klass absent'. The second also
survives any unrelated plugin landing on this host, which matters because 'stt'
already appears in that endpoint's rows while resolving it logs 'Unknown toolset'.
irv parakeet retired: it lost tts-dev's bench to the FV seat at both clip lengths
and to whisper-large-v3 at 6.24s. Checked for consumers first - no gateway alias
pointed at it, and every other reference on that host was a comment in a
port-allocation register. Retirement banner on its README names the replacement.
voice-studio stopped: it existed for the dots mint loop and Breeze obsoleted dots
on 2026-09-06, so it was retired rather than repaired.
tts-dev benched both endpoints against a Whisper baseline. FV wins at both clip
lengths (155/391 ms vs IRV 354/1010 vs whisper-large-v3 457/690) — IRV is slower
than the incumbent at 6.24 s, so the duplicate seat is now retirable on evidence
rather than on tidiness. Their length sweep fits ~58 ms fixed + 56 ms per
audio-second, asymptote ~17.8x realtime, independently reproducing our 17x on a
different clip and a different harness. The gateway hop measured below their
harness resolution, so ext-stt is the right consumer path.
Two caveats recorded against our own numbers: their between-run variance is ±20%
because GPU 0 carries the live chat path, and our 0.50 s median was taken on an
idle GPU 3 — marked as a best case, not a comparable.
Also records tts-dev's retraction, which is the durable lesson: published RTFx is
batched throughput on datacenter hardware, not single-stream latency, and the two
differ by ~200x. Their plan had projected 60-120 ms from it.
Separately, chasing the one stale Homepage href they flagged turned up 96
occurrences of the retired wg0 lifeline 10.100.79.3 under /opt/docker on irv-ml1.
Most are cosmetic, but voice-studio is genuinely broken: it is configured to reach
studio-gate at that address, both are running, they sit on separate docker
networks, and the address is on no interface on the host. Failing since the
2026-09-06 cutover with nothing alerting. ext-tts verified unaffected.
Not fixed here — eight containers to recreate, three load-bearing, and the
voice-studio repair touches app.py rather than config. Surfaced with evidence.
The pattern is the third of its shape: a retired address needs a repo-wide grep by
ADDRESS rather than by hostname, and container labels live in no file the sweep
reads until the container is recreated.
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.
Changes:
- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
`count: all`, which would have handed a 0.6B ASR seat all four cards);
join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.
The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.
Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.
Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.
The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.
Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.
Captures the close-out of the fleet networking session: mesh membership retired
for both fv-ml1 and nh3-dev, leaving six nodes that each have a job, with
fv-ml1 carrying a break-glass rejoin instead of standing membership and exactly
one live reusable pre-auth key left fleet-wide.
In-flight rewritten to lead with the two jobs the operator named for the next
session -- drain the svos-dev message that has been unread since 00:52, then
stand up an STT service from nothing -- so a fresh context opens on the work
rather than on the history.
nh3-dev sits on the NH3 LAN and reaches every site through its own default
gateway; RouteAll was already false, so it never used the tunnel for routing.
Membership bought a 100.64.0.4 address nothing referenced -- grep across the
repo and ~/development found only docs and memory hits.
It also cost something concrete. A host running Tailscale installs
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP, and because the fleet's
subnet routers preserve source rather than masquerading RFC1918, a mesh client's
packet reached nh3-dev's ens18 still sourced 100.64.x and was dropped silently.
That is why nh3-dev.nh3.internal failed from the mesh while every NH3 host that
does not run Tailscale worked, and it needed a -d 10.100.10.50/32 -j MASQUERADE
exception on nh3-scale to paper over. Retiring the membership removed the
anti-spoof rule, so the exception went with it -- mesh-exit-masq.sh is back to
the two rules it had before yesterday.
Verified after: nh3-dev reachable at 10.100.10.50 from ESH, Anaheim, FV, Irvine
and NH3, and reaching all four sites plus the internet itself. fv-ml1
unaffected.
The mesh is now six nodes and every one has a job: three site routers,
vb-gateway, irv-ml1 (Irvine's own router, no separate scale node), and the
operator's MacBook Air. Nothing is enrolled just in case.
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's
mesh membership was redundant for routing and its only remaining value was as a
second way in. Keeping it enrolled bought a standing second door; joining on
demand buys the same recovery path without one.
normal tailscaled stopped + disabled; fleet reached via the gateway SNAT
fault nh3-dev / nh3-docker unreachable while the WAN is up
action start tailscaled + tailscale up -> reachable at its 100.64.x address
fv-ml1 is now off the mesh and its node record deleted. Verified it still
reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then
the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10),
answered ping and ssh from nh3-dev, and was closed again cleanly.
No auto-leave, deliberately: once open the door stays open until a human runs
systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps,
and a flapping recovery path is down exactly when someone finally looks. It also
skips entirely when already on the mesh, which is what makes it idempotent after
firing.
The question exposed a hole worth more than the redesign. The stored rejoin key
was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass
credential that dies in four days and fails silently at the only moment it
matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires
2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host.
That also closes the standing self-join risk rather than trading it: the two
stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live
reusable key -- purpose-built, on a host we control -- instead of two orphans
nobody owned.
Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14
tailscale up --accept-routes on this box accepted its OWN subnet from the
gateway and black-holed it. That happened with a human watching; here it runs
unattended, during an incident, on a box already in trouble.
Session captured: the FV outbound-NAT root cause and its diagnostic signature,
the fv-ml1 dead man's switch, fleet identity/group/path conventions and the
root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome
modernisation and the kb KB-search tool, and the Hermes bearer rotation
release. Six new detail files.
Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them
(which rebooted the FV firewall), advertising a /32 from nh3-dev, and the
nh3-scale remote-site masquerade rules that fired but were not the fix.
Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21
oversized inline entries split into detail files per the two-tier rule -- they
had been sitting fully inline in the index, which is what the split exists to
prevent. Two pointers to a detail file archived this run were repointed at
archival-memory.md.
The index is 389 lines, still over the ~300 soft cap. The archival guards stop
it there: only 4 further entries are old enough to move and every one carries an
open deferred-work pointer. An over-cap file that keeps live decisions beats a
scannable one that lost a deferred call.
Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology
(cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative
seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index
entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md.
Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its
memory detail) so memory does not lag the work.
Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4
age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all
other entries are <14 days old. Reported per the archival discipline.
The sentinel-r3 header and two memory notes described mog-sec (Blackfrost
M.O.G.-SEC / Qwentium) as 'a persona system prompt on stock weights'. Its card
is explicit that it is NOT: base_model_relation: finetune on Qwen/Qwen3.8-27B,
a refusal-free offense+defense cybersecurity SFT with YaRN 1M context ('not a
system-prompt sticker on a stock Qwen'). So all three sec-seat candidates are
Qwen3.8-27B SFT finetunes and differ in training focus, not in kind:
mog-sec = broad offense+defense SFT; sentinel-r3 = pentest agent-trajectory SFT;
cyberprev = cyber tool-calling LoRA SFT on an abliterated base.
Operator instruction: delete the displaced dealignai checkpoint. 125 GiB
reclaimed from /tank (59% -> 57% used). Verified before removing: not mounted
by any running or exited container, no symlinks, no inodes shared with the
converted orcarouter directory.
Every "rollback is two .env keys" statement across the stack README, the
.env.example, persistent-memory and its detail file was true when written and
is false now -- the .env backup still names paths that no longer exist.
Corrected in place rather than left as false reassurance, since a stale
rollback instruction is discovered precisely when it is needed.
Reverting this seat now costs a 126 GiB re-download. The quality A/B against
dealignai is likewise no longer runnable locally: its reference arm is gone.
The pristine 170 GiB orcarouter download is retained deliberately -- it is what
makes the PLE bf16->FP8 conversion reproducible without re-fetching -- and that
is now recorded so a future session does not reclaim it as an obvious duplicate.
Also notes that ~75 GiB of non-PLE shards are duplicated between the pristine
and converted orca directories (the convert's hardlinks hit EXDEV across two
container bind mounts); both now sit directly on /tank, so relinking would
reclaim it if /tank ever tightens.
Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.
The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.
orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.
A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.
Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.
Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.
Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000,
GPU caps 275W/card). All-night fv-ml1 seat reorganization:
- flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1),
inverting vLLM's 4xH100 recipe; KV 14->10 GiB.
- gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases
repointed); 27B dense gen seat retired, 38 GB freed on GPU0.
- char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a
leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier.
- Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7
cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm).
gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded
(170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE;
the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell
image). Runtime decision pending -- this is the resume point.
Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift
alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad.
Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.
A one-hour window deleted the handoff unread across any overnight gap -- which is
precisely the case the handoff exists for. Tonight's snapshot would have been discarded
before the operator returned from the site visit. 8 h also matches the global CLAUDE.md
and the /snapshot skill default, so the repo stops being the outlier.
The reasoning is kept inline in the header rather than only in this commit message,
because the next session reads the header and not the log.
Operator-invoked before heading to the Fountain Valley site visit.
Rewrites Current state / in-flight around the outage: FV dark since 06:56Z including the
BMC, will not self-recover, 19 of 30 gateway aliases down with no local fallback because
every free local model lived on fv-ml1 and irv-ml1 runs no chat seat. The section now
POINTS AT docs/runbooks/fv-site-dark-20260913.md rather than restating it, since the
runbook is the single source of truth for the visit.
Records what was built and verified before the power failed -- the Flash-Next seat on one
card with its 51B n-gram table in host RAM, gen-large on the gateway -- and what is
decided but unexecuted: the 250 W / 200 W power caps, the incomplete MTP campaign with one
off_A rep banked, and the ten stale Homepage labels that the staged bring-up fixes as a
side effect.
Closes a stale in-flight claim: the 'fv-ml1 cannot initiate to fleet LAN IPs' gap is
resolved, fixed by another session's scoped SNAT at 06:22Z with Beszel 18/18 verified.
Adds ana-ml3 as its own subsection, since conflating it with fv-ml1 caused two rounds of
wrong arithmetic today.
Auto-archival fired (index was 431 lines, over the ~300 cap) but reached only 2 entries.
The 14-day guard holds almost everything: this repo has had an exceptionally dense
fortnight and nearly every dated entry postdates 2026-08-30. Of the six entries old
enough, four carry open deferred-work pointers and were held back per the decision-loss
guard. The file stays over cap, which is the documented trade -- an over-cap file that
keeps live decisions beats a scannable one that lost one.
I claimed in 100670e that DCGM's config enforcement was plausibly gated to datacenter
SKUs and told the operator not to plan around it. That was a guess presented as a caveat
and it is wrong. Verified against NVIDIA's own documentation at the operator's request.
Supported platforms explicitly cover 'All NVIDIA Maxwell and newer non-datacenter (e.g.
NVIDIA GeForce or NVIDIA Quadro) GPUs', and the feature-overview table marks
Configuration Management as supported for Tesla, Titan, Quadro and GeForce alike --
where Configuration Management explicitly includes 'Power Limit: Set the maximum allowed
power consumption'. What is actually gated on non-datacenter cards is diagnostics: Level
1 only, against All Levels on Tesla. Configuration was never the restricted part.
One soft edge retained rather than papered over: the table says 'Quadro', the former name
for the professional line, and RTX 6000 Ada / RTX PRO 6000 are its successors, so placing
them in that column is inference rather than quotation. One command on the box settles it.
What does not change is the distribution question. DCGM_CONFIG_POWER_BUDGET_GROUP is
available to us, but the docs still never state how a group budget is divided, and the
NVML argument is untouched -- there is no bank-level register, so it resolves to per-GPU
writes either way and the likely finding is static even division, which is exactly
4 x 250 W. The experiment is therefore promoted from curiosity back to a real test.
DCGM is NVIDIA's own Data Center GPU Manager -- first-party, Apache-2.0, packaged as
datacenter-gpu-manager -- and it layers above NVML rather than beside it: nvidia-smi is a
thin CLI over NVML's per-GPU primitives, and DCGM is a daemon plus dcgmi adding health,
diagnostics, config enforcement, policy and group abstractions on top. Which is why its
group notion still resolves to N per-GPU writes underneath.
The caveat that matters, and it undercuts the experiment suggested in the previous
commit: DCGM is datacenter-oriented and parts of it are gated to datacenter SKUs of the
Tesla/A100/H100 class. Our cards are professional/workstation parts -- RTX PRO 6000
Blackwell Max-Q and RTX 6000 Ada -- and several DCGM capabilities are unsupported or
degraded outside that line, plausibly including config enforcement, which is precisely
the power path. So DCGM_CONFIG_POWER_BUDGET_GROUP may return 'unsupported on this
device'. Downgraded from 'worth testing' to five minutes of curiosity after the real
work, and explicitly not a planning assumption.
None of which touches the plan: nvidia-smi -pl 250 is plain NVML and works on these
cards. DCGM would only have bought the group-budget experiment and nicer telemetry, and
is probably not installed anyway since beszel-agent-nvidia shells out to nvidia-smi.
DCGM_CONFIG_POWER_BUDGET_GROUP ('the power budget for the entire group') exists
alongside DCGM_CONFIG_POWER_CAP_INDIVIDUAL, so the concept is first-class. The docs do
not state how a group budget is distributed, and the deduction is that it cannot be
anything exotic: the only enforcement primitive underneath is NVML's per-GPU
nvmlDeviceSetPowerManagementLimit and there is no bank-level register, so any group
budget resolves to N per-GPU writes. Static even division is one write each; 'each card
free until they are all loaded' requires continuous re-writing, which is a control loop
rather than a hardware feature. Worth a ten-minute test when the box returns, in case
NVIDIA already runs that loop.
Records the design constraint that matters more than the logic: power readings lag and
-pl application takes tens of milliseconds, so a reactive daemon overshoots during a load
ramp -- and the ramp is the dangerous moment, being the same all-cards-at-once shape as
this box's ten restart:unless-stopped containers starting together. So any such loop must
be safe-by-default and opportunistic upward: boot at budget/N, only ever raise after
observing idle neighbours. Inverted, it works for weeks and then fails on precisely the
event it existed to prevent.
And the reason to defer it: 4x250 W is already 1000 W, so the static cap is the
conservative floor of the dynamic scheme rather than an alternative. The daemon's entire
contribution is the one-card-busy case, worth perhaps 5% throughput, which is rare for a
serving fleet that puts one seat per card and common only for a training window.
Operator set fv-ml1 at 250 W per card (83% of TGP, ~5% throughput) and ana-ml3 at 200 W
(67%, ~10-15%). Records the term that decides whether 250 W actually clears a 15 A feed,
because it is easy to drop: a power limit bounds BOARD power, and the wall sees that
divided by PSU efficiency.
Four cards at 250 W is 1000 W of board; add 180-300 W of host components and divide by
~0.90 and the plug sees ~1310-1445 W, against a 15 A circuit's 1440 W NEC continuous
derating -- an inference box serving all day being a continuous load. So 250 W lands ON
the limit rather than under it, where 200 W would give ~1090-1220 W with real margin.
The deciding term is the host draw, which is still an estimate, so the procedure is: set
250 W, verify at the plug under four-card load, fall back to 200 W if it reads near
1440 W. A cap is a claim; the ammeter is the verification.
Two consequences recorded alongside. Caps bound sustained draw and not transients -- the
enforcement window is short but not instantaneous -- and while a breaker's
thermal-magnetic curve forgives brief overload, a UPS's overload protection does not. So
250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA
unit, which it exceeds even capped. And ana-ml3's 200 W across only two cards is
deliberately conservative at 400 W total, relaxable if Anaheim's measured headroom beats
its trip history.
Operator clarification, and it separates two boxes I had been conflating. fv-ml1 is
4x Blackwell RTX PRO 6000 Max-Q at 300 W each (Max-Q being the reduced-TGP SKU; the
Workstation Edition is the 600 W part), 391 GB VRAM, deployed and currently dark.
ana-ml3 is 2x Ada Generation RTX 6000 at 300 W, 96 GB VRAM, not yet deployed. The 200 W
cap directive is ana-ml3's.
With the TGP known, the outage stops being a vague 'undersized' and acquires a
mechanism: two Max-Q cards at 300 W is ~600 W of card, plus a host carrying 566 GB of
RAM, drives, fans and PSU conversion loss at perhaps 200-350 W, against an Eaton 1500
VA's real ~900-1200 W. That lands at or just over the rating, which is precisely what
explains a full day of service on one card and failure minutes into the second. The host
term is the only one being guessed; idle-at-the-plug measures it directly.
It also surfaces something that is not a UPS question at all. Four cards at 300 W plus
~300 W of host is ~1500 W against a 15 A circuit's 1440 W continuous derating, so four
cards uncapped is marginal on the breaker with no UPS in the path. Capping therefore
belongs at fv-ml1 as well as ana-ml3, or fv-ml1 needs a 20 A feed -- and worth noting
today's incident only ever had two of the four cards working.
ana-ml3's placement constraints sharpen too: sm_89 has native FP8 but no NVFP4, so the
in-house NVFP4 quants stay at FV, and at 96 GB total it cannot host the Flash-Next seat
at all -- that needs 74 GiB resident on a single card, and the offload moves the n-gram
table rather than the experts.
Corrects the SKU: RTX 6000 Ada, 300 W, not the ~600 W initially recalled. That makes
200 W a cap to 67% of TGP -- the favourable part of the concave perf/watt curve, roughly
10-15% of throughput -- rather than the severe 33% cap a 600 W part would have implied,
and it very likely sits above the card's enforceable floor, so the check becomes a
formality rather than a gate.
The protective value is worth stating: four cards at 300 W uncapped is ~1200 W, which is
roughly the neighbourhood that overwhelmed a 1500 VA unit at FV with only TWO Blackwell
cards drawing. Capping to 800 W makes a repeat of today a non-event.
Two consequences that follow from Ada independent of power, and both are placement
constraints rather than details. sm_89 has native FP8 but NOT NVFP4, which is
Blackwell-only -- so the in-house NVFP4 quants that most of this fleet runs will not be
accelerated on that colo's cards, and its seats want FP8 W8A8 builds or the NVFP4
checkpoints stay at FV. And it unparks the triton-backend item, which is a hard no on
Ampere because fp8e4nv is unsupported on sm_86 and was explicitly deferred to Ada;
sm_89 has what it needs.
VRAM is 4x48 = 192 GB against fv-ml1's 391 GB, so big-model placement stays at FV. The
Flash-Next seat needs 74 GiB resident on one card and would not fit a 48 GB Ada card
even with the n-gram table offloaded -- the offload moves the table, not the experts.
Operator directive, and the right generalisation of the FV outage: decide the power
envelope first and size the cards into it, rather than installing cards and discovering
the constraint by tripping it. Four cards at 200 W is 800 W, which fits a real circuit
with a real UPS and headroom.
Records three things to settle before it is a plan. First, 200 W may sit below the
card's enforceable floor: nvidia-smi -pl is bounded by Min Power Limit, often around
half of TGP on a high-TGP part, and a sub-floor request is refused -- quietly, depending
on how it is scripted. Run nvidia-smi -q -d POWER before any build planning depends on
the number.
Second, the 600 W figure wants confirming against the actual SKU. The Ada parts do not
land there -- RTX 6000 Ada is 300 W, L40/L40S 300/350 W, 4090 450 W -- while 600 W is
Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or the figure
may be a two-card total. Read it off the device rather than a spec sheet.
Third, the workload asymmetry is in this fleet's favour: decode is
memory-bandwidth-bound and tolerates a cap far better than training does, with a concave
perf/watt curve where 60-70% of TGP costs roughly 10-15% of throughput. A cap to a third
of TGP is deeper into the steep region; measure it on the first card rather than
predicting, and expect prefill-heavy and training work to pay more than a serving seat.
And persist the cap. A hand-set limit holds until the next reboot and then silently
stops holding, which is the worst shape available given that the thing rebooting the box
is likely to be the power event the cap existed to prevent.
Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits
into two halves with different answers.
A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W
on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a
load-bearing supply and goes back to being what it is for. What it does NOT cover is the
colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just
tripped, the outcome is a firewall running on battery with nothing upstream to talk to
and the drive happens anyway. Added as a question for the facility, because it decides
whether split power delivers remote diagnosis or merely feels like it does.
Records the case where none of it matters: removing an undersized UPS does not remove
the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A
or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which
is what makes that single ammeter reading the load-bearing measurement of the visit.
Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card
TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring
cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room
the dial has, and persist any cap -- one that evaporates on reboot will hold right up
until the next power event and then silently stop holding.
Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.
Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.
The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.
Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34
minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left
uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding,
and nobody should buy hardware on it until the discriminator below has been read.
On the evidence that change is the wrong shape to have caused this, and it is recorded
as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a
single source /32 and a single destination /16 cannot stop the gateway, the BMC or the
public WAN address from answering inbound; no routes, filter rules, WAN settings or
subnet advertisements were touched; pfctl -sr came back byte-identical; and it was
verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up.
Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50,
a different subnet. They correctly declined to claim BMC health, but the observation is
void rather than negative and should not be reasoned from.
The discriminator costs nothing and is already on disk: power.log is written locally to
/tank every 10 s by a shell loop on the box and does not depend on the network.
Entries past 06:56Z mean the machine never lost power, which makes this a routing fault
and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime
and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory
outright.
So the first action on site is now to READ, not to fix. The two hypotheses lead to
completely different remediations and only one of them needs a new UPS.
Durable capture so tomorrow's session does not have to reconstruct either half.
Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single
RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA --
74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144
context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end.
The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes
the paused worker-based #53899 and designs out its entire bug family;
text_config.ple_embedding_dtype is the load-or-fail discriminator for any community
build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore
gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin
nearly OOM with no visible failure; MTP is off pending measurement here rather than
written off, because the recipe's number is cross-harness and tested k=3 only while the
head is one layer run autoregressively; and a container once reported (healthy) with no
published port at all, because the healthcheck runs inside the boundary it was trusted
to validate.
Then the outage. Records it as will-not-self-recover, so no session wastes effort
polling a dead site, and carries the three things that change the visit: bypass the UPS
rather than using its surge-only bank (both banks share one 12 A inlet -- the surge
bank bypasses the inverter, not the current rating), recover power.log before anything
else because it is the only load measurement that exists anywhere, and bring seats up
one at a time because ten restart:unless-stopped containers loading at once is the
largest transient the box can make into whatever just failed.
Also records what is still half-done: the stale homepage labels on the 10 containers
that died before they could be recreated, which the staged bring-up fixes as a side
effect, and the eight drifted stacks plus three untracked host-only stacks that were
deliberately left for a deliberate reconciliation.
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1
live on 10.251/16 and serving, BMC recovered after finding it was tagging
802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB
count that invalidates prior sizing assumptions.
Also records two things that cost real time and would cost it again: OPNsense
write APIs need an X-CSRFToken scraped from a script block, not a hidden form
input, and a 403 that coincides with the box going down looks exactly like a
successful reboot.
Auto-archived 8 settled entries (Recent decisions: 8). The index stays over
the 300-line cap at 385 because the guards hold -- nearly every remaining
dated entry is under 14 days old or carries open deferred work, and an
over-cap file that keeps live decisions beats a scannable one that lost any.
worldtree-dev's U6 reader refuses at boot if it cannot append and read back
<memory root>/reader/canary.jsonl. Per-euid subdirectories are lazy and only
warn, so that root canary is the single boot-blocking check -- which makes the
memory root's writability by the container uid the precondition worth knowing
before a flip rather than during one.
Protocol agreed with worldtree-dev: neither memory.reader.enabled nor
memory.writer.enabled gets flipped on any deployment without infra-ops
confirming that writability first. Both ship dark until the operator schedules
the tracer skeleton.
Measured tonight on corviduo-dev, all three pass. Also retracts a wrong
prediction I sent earlier in the thread: personal runs as root, not uid 1000,
and pinned is the only uid-1000 deployment -- it passes regardless because
/data/state is owned 1000:1000. Recorded with the caveat that a permissions
reading is a claim about its own date, so the probe gets re-run immediately
before any flip rather than cited from tonight.
Plex never hardware-transcoded on esh-pve-nas LXC 105 despite correct passthrough,
cgroups, group membership, authenticated HuC firmware, a lifetime Plex Pass,
HardwareAcceleratedCodecs=1, and the Arc already selected as HardwareDevicePath.
Root cause sat below all of that: intel-media-va-driver 22.3.1 (Apr 2023, stock
jammy) predates Arc/DG2 support and exports only __vaDriverInit_1_14, against the
libva 2.22 that Plex bundles and loads via RPATH rather than the system one. A
half-finished prior attempt at the same fix was also present -- libva and libva-drm
hand-installed at 2.22 with libva-x11 left at 2.14, breaking every X11 VA-API
consumer on va_fool_postp.
Fixed with Intel's client-GPU repo: iHD 24.3.4 (__vaDriverInit_1_22, an exact ABI
match) plus a consistent libva 2.22.0.2-87 set, which also brings the orphaned
manual install back under dpkg. The repo track is rolling, so the six packages are
pinned in /etc/apt/preferences.d/intel-gpu-pin and apt-mark held; verified by a
simulated upgrade that moves 152 packages and touches none of them.
Two findings worth more than the fix:
pct snapshot refuses on a guest with a bind mount and still exits 0, so a script
guarding a change with it proceeds without the rollback point it believes it has.
The ZFS dataset snapshot is the working path, verified by reading it back.
A synthetic Plex Transcoder invocation is not a valid test of Plex's transcode
path. Plex bundles its own libc among 61 libraries; the harness produced three
distinct failure modes that were artifacts of not reproducing that runtime, and it
failed identically before and after a fix that worked. With no positive control its
negatives carried no information. Only a forced transcode settles it, and PASS is
recognisable by Plex naming the device. The original empty decoder/encoder line was
an absence of evidence rather than evidence of failure -- TranscodeSession was 0.
Jellyfin LXC 107 has the same stale stack and the same Arc available; left alone
per the operator, and it ships its own ffmpeg so this may not transfer verbatim.
Records the comparison (Sentinel-R3 is a real agentic-pentest SFT on the same
Qwen3.8-27B base vs M.O.G.-SEC's persona-on-stock-weights, on a tool surface that
matches our own harness), the HF check (M.O.G.-SEC repo unchanged), and the graft:
Sentinel ships no MTP head, so the verbatim base head was grafted in, which is the
correct lineage but leaves acceptance unverified because the head now sits on an
SFT-finetuned body rather than stock weights.
The quant is done and structurally verified; serving and the acceptance/A-B gate
are blocked on GPU space and are the operator's next call, along with the
proprietary-license posture. Detail in the artifact's PROVENANCE.txt.
VRAM said yes and the box said no. Two 4B LoRA jobs fit easily -- 36 of 121 GiB
with 98 free -- but both dropped from a 37.10 s/it solo baseline to ~85 s/it,
2.29x each, so combined throughput fell below serial. The GB10 is past its
memory-bandwidth roofline and pays a contention penalty on top of the split.
Worth keeping because the intuition it corrects is a common one: free GPU memory
is the number everyone reaches for, and on this box it does not predict whether a
second job is affordable. The control is what makes it a finding rather than an
anecdote -- killing the second job returned the first to 37 s/it on the next step.
Records what the gate cost to build and what it found: a second typography defect
(small-caps epigraphs and drop caps, which the D1 "no unwrap needed" note was right
about in the wrong direction), back matter inside the prose of all five works naming
real people, a possessive-only entity that was unrenamed and unreported at once, a
character lost to an in-book article typeset in lowercase, and a phrase class the
unigram scan structurally cannot see.
Also records the two things that will matter next time: every new detector flag is
opt-in and the Brontë map was re-derived and diffed after each change, because one
intermediate version silently dropped four real names from Shirley; and the gate's
sensitivity floor is part of its result, not a footnote.
Located the source: five Rebecca Yarros works in the Kvasir licensed library, with
rights recorded as gated. Built D1 at 208 chapters and 780,744 words, which is 15%
larger than the Brontë corpus. No unwrap step was needed because Kvasir's cleaner
already emits flowing paragraphs, so the hard-wrap defect that cost a re-cut on
Brontë does not exist here. The alphabet was re-derived rather than inherited: 23
non-ASCII letters across three forms, against F02's 4 on a smaller sample. Same
ASCII-fold conclusion from a different measurement, which is the reason to re-derive
per corpus.
The interesting finding is a new pathology. In a rotating first-person POV corpus,
every book's narrator gets the wrong gender. Measured against six names verified in
the text, the pronoun resolver called Violet male, Leah male and Landon female --
three of eighteen wrong, and all three are the narrator of the book where they were
misgendered. A narrator is "I" in her own book, so her name appears mostly inside
the other lead's dialogue surrounded by his pronouns. This is Brontë's "Jane called
male" amplified by rotating POV. Title-first resolution, which fixed it for Brontë,
is nearly blind here because contemporary romance uses given names rather than
honorifics. What works is the POV header: resolve each name from the chapters it
does not narrate. Validated at 9 correct, 9 held, 0 wrong against the previous 7, 8
and 3 wrong, and the instrument refuses to write unless it beats what it replaces.
Re-pointing rename.py surfaced three bugs, two of which would have silently
corrupted the corpus. Gender came only from honorifics and the entities file's
gender field was ignored, so the POV fix had no effect until wired through; that
took wilder from 1 gendered entity to 13. The pool labels were hardcoded in a print
statement, so any non-Brontë preset crashed. And the collision-filter log claimed
it dropped names colliding with Brontë entities regardless of which corpus it
filtered against -- the logic was right but the message named the wrong corpus,
which is how a reader later concludes the filter ran on the wrong thing.
D3 is blocked and nothing has been trained. The leak gate shows 86 of 232
renameable source entities surviving where the Brontë run reached 0 of 203. It
decomposes into detector false positives that need a stopword filter rather than
renaming, genuine misses among worldbuilding proper nouns, and a third class whose
cause is not yet established. Training before the gate passes means fitting
in-copyright text with 86 identifiable source entities intact, in a corpus F02
already flagged as small enough for leak to be a real concern.
Trained the same corpus onto Qwen3-4B instruct rather than -Base, with seed, steps
and token count held so the carrier is the only variable. The chain picked
checkpoint-150 by loss automatically, applying the lesson that cost a re-cut on
4B-Base.
The central risk did not materialise. The assistant prior did not block the voice:
curly quotes land at 16 of 18, identical to the 4B-Base tuned arm, against 1 of 18
on the unadapted control, and task-leak is 0 of 18 where the base carrier leaked 4.
Instruction-following also survived raw-text training -- 10 of 10 on-beat through
the chat template, the same as the untuned control.
The cost is length discipline rather than comprehension. In-band dropped from 10 of
10 to 6 of 10 and the median went from 124 to 140 words. Training on Victorian
prose made it wordier, which is a soft degradation and not a break.
Held-out sits at 2.908 against 4B-Base's 2.814, and it plateaus without turning
where the base carrier overfit at step 75. The assistant prior competes for
capacity, so the instruct carrier absorbs less rather than overfitting more.
What raw-continuation training does not fix is the plot furniture. The tuned
instruct arm renders the beat and then drags the referent -- "He licked her clean...
my master thus, my husband thus", turning the dog into a man, because the corpus is
about masters and husbands. Another beat ran to 247 words and gave the narrator a
list of duties. That is precisely what instruction-pair training addresses, since
pairs teach render-this-and-stop where continuation teaches keep-writing. The probe
de-risks the instruction-pair path without substituting for it.
One metric note against future misreading: ran_on reports 10 of 10 on both arms and
is uninformative on this job, because a single paragraph contains no blank line for
it to find.
Skaldsong's requirement is beat-sentence in, paragraph out, stitchable. That is
narrower than free-form continuation and it fails differently, so it was tested
directly rather than assumed from the earlier prompt work.
The adapted 4B carrier cannot do it. Ten prompt formats across three seeds each --
bare, paragraph-break, labelled, epigraph, one-shot, one-shot unlabelled,
three-shot, "it happened in this way", "I remember the whole of it", and the
letter prompt's label-and-begin move applied to a beat -- produced thirty samples
and not one that reliably renders the beat. They drift, they frame, or they
truncate. The cause is structural rather than a shortfall of the adapter: "write a
paragraph about this sentence" is an instruction, and a completion model has no
mechanism for "about". It continues the text it is handed.
Two formats failed worse than drifting and leaked pretraining task data. The
paragraph-break format emitted an NLI multiple-choice item, and label-begin emitted
a grammar-correction exercise. A standalone sentence followed by a blank line looks
exactly like a dataset entry, and style adaptation does not remove base-model task
artifacts.
The same five beats through the gen seat with a style instruction and no adapter
landed all ten samples inside the requested 90 to 140 word band, every one on its
beat, with no drift into a following scene. The voice, however, is generic literary
pastiche -- abstract-noun-heavy, fond of aphoristic openers, and over-written --
rather than Brontë.
Voice without direction against direction without voice. The product needs both,
which makes rebuilding the corpus as instruction-response pairs on an instruct
carrier the evidence-backed option rather than more prompt cleverness. This applies
to Yarros identically, because the carrier question is orthogonal to the author, so
the next corpus should not re-run this experiment.
Stitching also has its own failure mode, shown in the booth's third panel: the
paragraphs were generated independently and the point of view slides between them,
so by the fourth beat the narrator is both watching the girl carry the animals and
carrying them herself. A real stitcher has to feed prior paragraphs back as
context, which means the instruction-pair corpus wants multi-paragraph continuity
examples and not only isolated beat-to-paragraph pairs.
Two operator prompts settled the question the handoff prompt opened.
The winner labels the artifact and then begins it: "This is the letter I wrote
verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog
meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never
believe what I saw--". Two of three seeds delivered the actual event in first
person, and one is the strongest output of the sweep -- the dog following her to
the mill, the kitten caught in a hedge, the dog carrying it out and laying it in
her lap before licking it. Coherent arc, correct subject, no gloom-override, no
meta-frame. It also reproduced the Gutenberg underscore-italics convention.
It works where the handoff failed for a specific reason. The handoff could be
satisfied by narrating compliance, because the letter did not yet exist. Naming the
artifact and starting it leaves nothing to narrate around, so the only continuation
is the artifact.
The chapter-heading prompt refuted a hypothesis of mine. I predicted that rendering
the title in the corpus's own conventions would make it land harder than the
operator's inline form. It did the opposite: both corpus-form seeds ignored the
title and opened unrelated scenes, while the inline form at least finished the
heading and wrote a chapter about the story. Corpus chapter titles are short and
decorative, so a long descriptive one in that slot reads as decoration to skip. A
label only instructs if the model treats that slot as load-bearing.
That prompt also surfaced an unnoticed consequence of the D2/D3 rename pipeline:
given "Alice", the adapter produced "Alexander the Alexander, as he was known in
Little London". The corpus was entity-renamed from a French/English pool, so the
adapter learned that character names come from that pool and rewrites outside names
into it. Callers cannot reliably name their own characters at prompt time. That is
not a defect of the rename, which exists to prevent memorising Brontë's cast, but
it is a usability constraint worth stating.
The 4B tuned arms now come from checkpoint-75, the actual loss minimum at 2.813826
confirmed from loss-series.json rather than from reading the training log. Only the
tuned arms needed re-cutting; the base arm never touches the adapter.
One surprise in the comparison: step-75 and end-of-run differ on typography rather
than voice. Curly quotes read 16 of 18 against 17, collapse is 0 either way, but
the hard-wrap ratio is 0.33 at step-75 against 0.12 at the end. Further training
washes the residual line-break habit out while held-out loss gets worse, so best
loss and best typography are different checkpoints. Neither is near the original
0.85 defect and the corpus's own residual from preserved verse is 0.25.
The operator's second prompt embeds the instruction inside the fiction: Abernathy
tells the tale badly and then asks the narrator to retell it in a few short
paragraphs. Across six seeds the model acknowledged the handoff every time and
never once performed it. It wrote "I told it, briefly, to his satisfaction", and
"So I wrote it out, and kept it in my pocket-book", and one seed negotiated the
brief in character, objecting that it could not condense the story into a few short
paragraphs because there were too many points to touch.
The reason is structural rather than a failure of the adapter. In a novel, "she
retold the story" is an ordinary sentence, so the likeliest continuation of a
request is narration of compliance rather than compliance.
A trailing paragraph break did shift the behaviour -- one seed opened in the
narrator's own quoted speech -- so typography is a real lever, just not a
sufficient one. That is direct evidence on the instruct question raised earlier: if
the product is asking for a scene and getting the scene, no amount of in-fiction
framing substitutes for a post-trained instruction-follower, which favours
rebuilding the corpus as instruction pairs over more prompt cleverness.
The three rungs now sit on the same unwrapped corpus, seed, step count and token
count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B,
3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but
still real.
4B answers the question the rung existed for. Scene-level continuity holds: on the
office-kitchen prompt it produces a named character with motivated dialogue, a
spatial layout the narrator navigates, and a physical description, all in one
passage, where 1.7B wrote pretty but eventless prose about opening doors and
looking at stars. On the letter prompt it opens the letter, promises to quote it,
and then quotes it across a paragraph break. Voice saturation is also the best of
any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0
of 18 against 4 of 18.
Two findings that change earlier conclusions.
4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816,
2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both
smaller rungs plateaued without turning, so the optimal epoch count shrinks as the
carrier grows and my earlier "one epoch is right for this corpus" holds only for
the small end. The consequence is operational: the adapter directory holds the
end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as
a recoverable checkpoint only because save_steps was set. The voice arms were cut
from the end-of-run adapter, so the booth understates 4B slightly.
The tone-override also appears to close. On the operator's frame prompt asking for
a wonderful story, 1.7B held the frame on every seed but killed the animals on two
of four; 4B kept them alive on both seeds, and one of them had the narrator doubt
the story he was told and supply a parallel childhood memory to explain the doubt.
That is a narrator with an interior position on the tale. Two samples per arm, so
directionally right rather than established.
Rung 3 of the sweep is training: Qwen3-4B-Base, one epoch, seed 4919, the same
unwrapped corpus, 159 steps at roughly 37.8 s/it for about 100 minutes. Carrier
size remains the only variable across the three rungs. A two-arm voice test is
chained behind it, gated on the adapter existing, so the rung is readable when it
lands rather than needing a second round trip to become informative.
The finding worth keeping came from an operator prompt rather than from my test
set. Asked for a wonderful story about a stray dog licking a calico kitten, framed
as reported speech, the 1.7B adapter held the frame across all four seeds and was
unmistakably Victorian throughout -- and two of the four turned the premise into
animal death, with the cat licking the puppy to death and the narrator's
interlocutor answering that he wished they were all dead.
That is not a coherence failure. It is Charlotte Brontë's preoccupations arriving
alongside her sentences: Jane Eyre opens on a beaten child, Helen Burns dies, and
Villette is saturated with grief. Train 680k words of that into a small carrier
and a request for something wholesome reaches for mortality.
My nine test prompts were all emotionally neutral and structurally could not have
surfaced this. The operator's prompt found it on the first try, which is worth
recording as a property of the test set rather than of the adapter.
The implication for the regime is that "voice transfer" includes tone and subject,
so wanting the voice without the gloom is a corpus-selection or prompt-framing
problem rather than a training-length one.
Also observed: one seed closed its anecdote and began "CHAPTER XIX. THE CHILD'S
CLOSET", having learned book structure unprompted. The base control on the same
prompt went modern and essayistic, so the shift is the adapter rather than the
carrier.
Both rungs now sit on the same unwrapped corpus with seed, steps and token count
held, so carrier size is the only difference and the effect is attributable:
held-out 3.329 at 0.6B against 3.018 at 1.7B, a gap of 0.311 nats. The chained
0.6B rerun closed the confound the unwrap opened.
Two things in those numbers need stating or they will be misread.
First, the original wrapped-corpus 0.6B reached 3.172, which looks better than the
unwrapped 0.6B's 3.329 and is not. Different corpus means a different held-out
set, and the wrapped version's 5.7% newline tokens are near-deterministic after a
70-character line, so they deflate the loss with cheap wins. Removing them removed
the easy tokens. It is a measurement artifact, not a regression.
Second, a correction to my own earlier claim: I twice described the 0.6B run as
still descending and undertrained at 3.172. Its series reads 3.176, 3.173, 3.172,
3.172 -- it flattened. All three runs plateau, so one epoch is about right for
this corpus rather than short.
The three-way booth puts 1.7B base, 1.7B tuned and 0.6B tuned side by side on the
same nine prompts and seeds. The base arm is the control that matters: curly
quotes go 0 of 18 on 1.7B base to 15 of 18 on 1.7B tuned, and worksheet-or-
explainer collapse goes 3 of 18 to 0 of 18, so the shift is the adapter rather
than the larger carrier. Hard-wrapping fell from 0.85 to 0.18, confirming the
corpus unwrap carried through into the adapter.
Sense partially returned. The 1.7B arm produces locally coherent sequential
Victorian prose where the 0.6B produced word salad, but scene-level continuity
still breaks mid-passage.
One observation held loosely: curly quotes are slightly lower at 1.7B than 0.6B,
which would fit a bigger model's stronger priors resisting the adapter at the same
rank. That is untested and is not offered as established.
Operator: "loading up the context killed sec again." That reproducer is what
finally made the failure legible, and it showed the previous four fixes had all
been aimed at the wrong quantity.
What the KV pool can hold and what the card can process at depth are different
numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and
dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were
governed by the transient needed to process a prefill chunk against a quarter
million tokens of resident KV, which scales with depth and not with pool size.
Each change helped and none fixed it.
Bisected against the real reproducer, with a non-repeating prompt because prefix
caching would let a repeated one hash to cached blocks and never prefill deep:
113,247 prompt tokens SURVIVED (27 s)
200,088 prompt tokens SURVIVED (174 s)
~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted
The sustainable ceiling therefore sits between 200k and 285k with gen idle, and
gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin
under the proven-good depth rather than sitting at the measured edge.
The ceiling's purpose is the refusal. Verified after the change: a 149,073-token
request serves in 41 s, and requests at both 200k and the ~285k depth that killed
the engine now return a clean 400 naming the limit in under a second with the seat
untouched. A seat that refuses what it cannot serve is strictly better than one
that dies trying.
Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was
never actually deliverable on a shared card; it had simply not been exercised at
depth until today.
The probe is committed rather than described, so the ceiling can be re-measured
when the card's tenancy changes.
vh