196416f3da67e5235199cc6b5ffbe0eb751c6d7d
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
196416f3da |
docs(cyberprev-seat): correct model lineage — it is a cyber SFT finetune, not an abliteration
Earlier commits and the compose header called cyberprev "an abliteration of Qwen3.8-27B (refusal-direction weight edit, no finetune)". That was wrong. Per the hotdogs model card (base_model_relation: finetune), cyberprev is an offensive-security agentic tool-calling LoRA, merged at scale 1.0, trained on hotdogs/cyber-sft-agent-qwen38 (8,400 rows across 22 pentest tools: nmap, sqlmap, metasploit, hydra, crackmapexec, ...). The abliteration is INHERITED from its base (hotdogs/Qwen3.8-27B-abliterated); the new capability in this repo is the cyber SFT itself. So the sec-seat lineage is: mog-sec = persona on stock weights; sentinel-r3 = pentest-trajectory SFT on stock weights; cyberprev = cyber tool-calling SFT on an abliterated base. All three are SFT-or-prompt variants; the earlier framing of cyberprev as "not a finetune" was backwards. Corrects the compose header lineage block, the homepage dashboard description, and the gateway alias comment. No functional change; the running seat and its weights are unchanged (label refreshes on next container recreate). |
||
|
vh
|
b94d173887 |
fix(cyberprev-seat): serve coherent v2 quant from name-repaired source; wire gateway
The first cyberprev quant served pure gibberish despite passing every structural check. Root cause: the hotdogs/Qwen3.8-27B-abliterated-cyber-preview checkpoint ships broken -- 850 of 1199 tensors (the whole transformer body) carry a TRIPLED `language_model.` prefix, confirmed present in the upstream HF index (an unsloth export bug, not our download). transformers silently loaded the unmapped tensors as random-init, the pipeline quantized noise, and structural verification passed because it inspects the output config, never that input names mapped. A plain greedy coherence smoke caught it; nothing structural could. Fix: rewrite tensor names against a known-good reference of the same architecture (qwen38-27b-uncensored-bf16), verified the repaired name-set == reference name-set exactly (1199 tensors, 0 collisions), and re-quantized from the corrected source (cyberprev-bf16-fixed). v2 is coherence-verified (spec off AND live with dflash): Paris/Berlin/Rome, correct arithmetic precedence, primes, clean cyber-domain chat with a proper thinking split. dflash k=7 measured at 2.77 mean acceptance length. Seat serves cyberprev-27b / cyberprev-27b-thinking on :8025/GPU0 with dflash k=7, displacing the retired sentinel-r3. Gateway: removed the retired sentinel-r3 / sentinel-r3-reasoning aliases and added cyberprev-27b / cyberprev-reasoning -- NOT repointed, since serving cyberprev weights under sentinel's name is silent substitution. Verified end-to-end through the gateway; sentinel-r3 now 400s. Playbook gains a superseded-claims row: structural verification does not imply a good quant; a coherence smoke is the load-bearing gate, and third-party sources must have their tensor names diffed against a reference before quantizing. Broken v1 quant dir removed. |
||
|
vh
|
1418edb03e |
feat(cyberprev-seat): mixed-NVFP4 quant of the abliterated cyber-preview, displacing sentinel-r3
Third sec-seat candidate: hotdogs/Qwen3.8-27B-abliterated-cyber-preview, an abliteration (refusal-direction weight edit) of Qwen3.8-27B aimed at the cyber-offense refusal surface -- distinct from mog-sec (persona on stock weights) and sentinel-r3 (SFT finetune). Operator instruction: quant it, take sentinel down, serve it with mtp or dflash. Quantized to the house mixed recipe via services/gen-seat-mixed-quant/ (NVFP4 W4A4 on MLP layers 0-55 + FP8 W8A8 on attn/linear_attn/lm_head/MLP 56-63, FP8 KV). The prior attempt (2026-09-11/14) died with "Cannot determine num_attention_heads" because it ran from a bare .venv whose newer compressed-tensors reads that field at top level; quant_mixed_nvfp4.py already promotes text_config attention fields for exactly this reason, and the run through the canonical vLLM-image + llmcompressor 0.13.0 / compressed-tensors 0.18.0 path (versions recorded from the container) completed clean. post_quant.py did its job: grafted the 15 MTP tensors verbatim (BF16), and re-injected re:^mtp.* into the ignore list after llm-compressor pruned it for matching no module at quant time -- without which vLLM loads the grafted head uninitialised and speculative decoding runs at 0% acceptance. It also caught a missing preprocessor_config.json (absent from the abliterated source AND its hotdogs upstream); restored from Qwen/Qwen3.8-27B, verified byte-identical to the working sentinel-r3 build, so the vision tower preprocesses. Verified from the tensors, not the config: NVFP4 covers MLP 0-55, FP8 covers 56-63, no overlap, 168 weight_packed tensors (56x3), 15 BF16 MTP, 333 BF16 vision. 51.0 GiB bf16 -> 21.0 GiB. Served under its own name (cyberprev-27b / -thinking), NOT sentinel-r3 -- serving different weights under a retired name is silent substitution. Takes over :8025 and GPU 0 (co-resident with mog-sec). dflash k=7 is the default, carried from the sentinel measurement, but is re-measured against MTP on this ABLITERATED body before cutover, since abliteration is exactly what can desync an MTP head. Drafter (qwen38-27b-dflash2-drafter) shared with mog-sec. Context capped at 163840 not native 262K, inherited from mog-sec's hard-won depth lesson; raise only after a non-repeating deep-prefill probe on this checkpoint. |
||
|
vh
|
4954cf6756 |
test(flash-next-seat): depth-probe orca to 258K clean; record a blind instrument
Bisected context depth on the orcarouter checkpoint with non-repeating prompts (unique random hex per probe, so prefix caching cannot short-circuit the prefill). Six depths from 31,978 to 258,517 tokens, all served. The load-bearing evidence is the engine allocator log: zero OOM, CUBLAS, or illegal-memory entries across the run. That is the same detector that caught the dealignai near-miss at 155K on the previous checkpoint, where it did fire. The probe also ran under real concurrent operator load, making it a stricter test than a solo run rather than a weaker one. Positive control passed: a mis-sized first attempt produced a ~265K-token prompt and got a clean 400 naming the limit instead of killing the engine, so the probe could detect the failure mode it was looking for. Calibration for re-runs: random hex words tokenize at 7.9 tokens/word here. vLLM #54919 (long prefill starving decode for 3-7 minutes) did not reproduce: 258K prefilled in 28.9 s, roughly 8,900 tok/s, scaling near-linearly from 32K. Records that the probe's memory-headroom half was BLIND and must not be reused. It reported an identical 95,460 MiB used / 2,427 MiB free on every row across an 8x range of depths, which is the tell. Two causes: --kv-cache-memory pins the pool and the engine logs "skipped memory profiling", so GPU usage is flat with respect to depth; and the actual risk is a transient activation spike during prefill, which before/after nvidia-smi bracketing structurally cannot observe. Peak-activation headroom therefore remains unmeasured; the pass/fail result rests on the allocator log alone. Also qualifies the earlier 167.5 tok/s decode figure as a possibly-contended lower bound, and records the operator's independent 140 tok/s average measured in real use while this probe was loading the same card. |
||
|
vh
|
d6cc22f6a0 |
chore(flash-next-seat): hardlink duplicated orca shards — 76 GiB reclaimed
The PLE conversion copied 28 non-PLE shards instead of linking them, because os.link hit EXDEV across two separate container bind mounts of the same filesystem. Both directories sit directly on /tank, so the duplication was recoverable. All 28 files verified byte-identical by SHA-256 before any change -- the point of a hardlink is that two paths become one file, so "same size" is not sufficient evidence. Relinked with ln to a temp name followed by an atomic rename over the target, never rm-then-ln, which would leave a window where the file does not exist for a process that reopens it. Generated as 56 literal-path commands rather than a loop, per the no-destructive-ops-on-a-variable-path rule. 294G apparent across the two directories -> 218G actual. Done with the seat live and serving; it did not blink. Records the coupling this creates: the pristine and converted directories now SHARE INODES, so editing a shared file in place in either one changes both. config.json and model.safetensors.index.json are deliberately excluded from the sharing, since the conversion changed them. Also consolidates two duplicate disk bullets left by earlier edits. |
||
|
vh
|
1b5d6ba23a |
docs(flash-next-seat): dealignai weights deleted — record that no local rollback exists
Operator instruction: delete the displaced dealignai checkpoint. 125 GiB reclaimed from /tank (59% -> 57% used). Verified before removing: not mounted by any running or exited container, no symlinks, no inodes shared with the converted orcarouter directory. Every "rollback is two .env keys" statement across the stack README, the .env.example, persistent-memory and its detail file was true when written and is false now -- the .env backup still names paths that no longer exist. Corrected in place rather than left as false reassurance, since a stale rollback instruction is discovered precisely when it is needed. Reverting this seat now costs a 126 GiB re-download. The quality A/B against dealignai is likewise no longer runnable locally: its reference arm is gone. The pristine 170 GiB orcarouter download is retained deliberately -- it is what makes the PLE bf16->FP8 conversion reproducible without re-fetching -- and that is now recorded so a future session does not reclaim it as an obvious duplicate. Also notes that ~75 GiB of non-PLE shards are duplicated between the pristine and converted orca directories (the convert's hardlinks hit EXDEV across two container bind mounts); both now sit directly on /tank, so relinking would reclaim it if /tank ever tightens. |
||
|
vh
|
4390be947d |
feat(flash-next-seat): serve orcarouter weight-only NVFP4 on gen-large
Swaps gen-large from the dealignai ModelOpt W4A4 build to orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both axes (W8 float attn, W4 float experts, input_activations: null) and so avoids the 4-bit-activation long-context degradation mode. The checkpoint was previously recorded as unloadable on any mainline vLLM, requiring a from-source PLE-loader patch. That conclusion was wrong on cost. Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as branch 1, before any quant-config type check, and its NotImplementedError for CompressedTensorsConfig is scoped to the PLE path only -- experts and dense load through the ordinary compressed-tensors paths. Verified by instantiating the real config and calling the selector both ways before doing any work. orcarouter ships a bf16 PLE, so the fix was to make the declaration true: convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit in one shard file with nothing else in it. Global amax 0.0894, per-shard outlier ratio 1.66x, scale chosen exactly representable in bf16 so no scale-rounding error stacks on quantization; amax maps to 446.17/448, no clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation -- the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors) and vision tower carried through untouched. A second, independent blocker followed: orcarouter labels its 12 QSA layers qwen_sparse_attention, which vLLM rejects; it accepts full_attention and selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this and the dealignai checkpoint before renaming -- without that check the rename silently selects plain attention and serves a subtly wrong model that still passes a healthcheck. Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx, MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure came from a different harness, so this is not claimed as a win over it; what it does establish is that weight-only experts did not cost decode speed. Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at 262K. Rollback is two .env keys; dealignai remains on disk. Also corrects the README's MTP-is-off section, stale since k=3 was deployed, and adds a superseded-claims row to the quantization playbook. |
||
|
vh
|
ed3f1ede3b |
memory: snapshot — FV recovered, fv-ml1 seat reorg, gen-large blocked on orca runtime
FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000, GPU caps 275W/card). All-night fv-ml1 seat reorganization: - flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1), inverting vLLM's 4xH100 recipe; KV 14->10 GiB. - gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases repointed); 27B dense gen seat retired, 38 GB freed on GPU0. - char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier. - Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7 cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm). gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded (170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE; the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell image). Runtime decision pending -- this is the resume point. Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad. Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched. |
||
|
vh
|
c2a70c13db |
feat(fv-ml1): daily drift alarm for the seat inventory
Wires scripts/seat-inventory.py --check to a user systemd timer on nh3-dev (09:15 daily, Persistent=true so a missed run fires on next boot) that posts to althing when the committed document stops matching the live box. Alarms rather than auto-committing. A drift means something changed on the HOST, which deserves a human look -- silently regenerating the doc would erase the evidence of when the change happened and why, which is how the char-rp substitution went unnoticed for three weeks. The alarm includes the changed table rows, not just the fact of divergence, so it does not send the reader hunting. ⚠ The post goes --to infra-ops, which is the fleet ops handle the reading session also runs as. That is the documented exception -- a memo from cron to a future session, the same pattern as the Beszel alerts -- so the message says so in its first line, to stop a future session triaging its own alarm as peer mail and trying to reply to it. SuccessExitStatus=0 1 because a detected drift is a deliberate non-zero exit, not a unit failure. |
||
|
vh
|
a91b841d86 |
feat(fv-ml1): generate the seat inventory from the live box instead of maintaining it by hand
The seat documentation must stay current, and a hand-written document cannot. The LiteLLM config described char-rp as a 31B model on a host and GPU it had not been on since 2026-08-24 -- three weeks of silent drift in a file that read as authoritative, and the reason a seat spent that period serving a model nobody intended. Anything typed here drifts the same way; anything read off the running containers cannot. scripts/seat-inventory.py derives the whole document from the host: - placement and VRAM from nvidia-smi compute-apps, mapped to containers through /proc/<pid>/cgroup -- nvidia-smi reports the vLLM engine child while docker reports the container pid, so matching them directly silently yields nothing - weights and KV tokens parsed from each engine's own startup log, not derived arithmetically, with concurrency computed as KV tokens over context - architecture, layer and expert counts, and the exact quantization group scheme (W4A4 vs W4A16 distinguished) from each model's config.json - speculative-decoding method and k from the container argv, which is how the three incompatible methods on this box became visible - lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model directory, not inside it, which is why an earlier pass wrongly reported two fully-documented seats as having no provenance - gateway aliases resolved from the LiteLLM config on ana-docker --check compares the committed document against the live box and exits non-zero when they diverge, ignoring only the generation timestamp. Suitable for CI or a scheduled drift alarm; read-only throughout, safe against production. Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which asserts the flag exists in the derived argv and aborts rather than running a campaign that silently ignored it. |
||
|
vh
|
2d83a895c1 |
docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers
Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production). |
||
|
vh
|
f9a8f176ca |
fix(mtp-bench): give each arm its own container name instead of tuning teardown waits
Container teardown latency here is unpredictable, not merely slow. Measured 2026-09-13 on the same container in the same session: once ~55s, once 0s. Removing a container holding ~92 GB of GPU memory plus the offloaded PLE mapping leaves an Exited record owning its name for that whole window, so any fixed wait or retry budget is a guess that will eventually be wrong -- a 12x5s retry lost arm k1 by roughly two attempts. Arms now use $NAME_BASE-$arm, set in boot() after stop_bench has torn down the previous arm. Names are never reused, so the collision is impossible by construction and teardown latency stops being load-bearing. cleanup() reaps every fn-mtp-bench-* container at exit. stop_bench keeps the GPU-memory wait -- the next container genuinely needs the device, and unlike the container record nvidia-smi clears promptly and reports truthfully. It no longer waits on the container listing at all. Three earlier hypotheses were wrong and are recorded so they are not retried: the name is not held by a phantom (the container is present and Exited), the removal does not fail (rm -f succeeds; it is asynchronous), and GPU memory release does not track name release (memory frees first, by a wide margin). |
||
|
vh
|
aa5ee7e0a6 |
fix(mtp-bench): docker ps -a is the wrong probe for name release; wait on GPU memory and retry the conflict
The previous fix polled `docker ps -a` until the bench container's name disappeared. That probe is useless here and the fix was a no-op: measured 2026-09-13, the container stops being listed while the daemon still holds the name reservation, so the poll returns false early and `docker run` hits a Conflict naming a container ID that `docker inspect` already reports as 'no such object'. Arm k1 was lost twice this way. Two changes, each aimed at something actually observed: - stop_bench waits on GPU memory falling below 1000 MiB rather than on the container listing. That is the resource the next container genuinely needs, and nvidia-smi reports it truthfully. - boot() retries `docker run` while stderr matches 'already in use', up to 12 attempts at 5s. The daemon's name lag is transient, and forfeiting an arm over it is silent data loss -- run_arm turns a failed boot into a skipped arm that still lets the campaign report success. Both probes verified against real artifacts before deploy: grep -c 'already in use' on the captured k1.runerr returns 1, and the nvidia-smi query returns a bare integer that compares correctly under [ -gt ]. The earlier fix passed bash -n and was still inert, so syntax is not the check that matters here. |
||
|
vh
|
67a7dc539e |
fix(mtp-bench): stop_bench raced docker name release, silently dropping arms
`docker rm -f` returns once removal is INITIATED, not once complete. The
bench container holds ~92 GB of GPU memory plus the offloaded PLE mapping, so
the name stays reserved for several seconds while the device is released. The
next `docker run --name` then dies with a name Conflict.
That failure was near-invisible: run_arm treats a failed boot as
`arm SKIPPED`, the campaign continues, and it still prints CAMPAIGN DONE.
Observed 2026-09-13 -- arm k1 lost the race after off_A's loaded container,
while k2 won it only because k1 had never started a container to tear down.
Every arm that follows one which actually ran is exposed, so k3 and off_B were
both on track to vanish from a run that would have reported success.
stop_bench now polls until the name is released (120s ceiling, warns and
continues). Added a completeness gate: the campaign asserts every expected
res-<arm>-rep<n>.json exists and reports CAMPAIGN INCOMPLETE naming each
missing result rather than DONE, so a gutted run cannot look like a clean one.
Verified the docker ps format string empirically -- an earlier draft nested
quotes so the template rendered as '{{.Names}}' with literal quotes, which
grep -qx could never match, making the wait a no-op that passed bash -n.
|
||
|
vh
|
0a9cf58e19 |
fix(mtp-bench): peak-power summary mis-keyed GPU 0 by timestamp
power.log lines begin '<HH:MM:SS> 0, <W> W, ...', so splitting the first pipe-field on ', ' yields a[1] = '20:18:00 0' rather than '0'. GPU 0 was therefore keyed by sample timestamp, emitting one bogus row per sample and no recoverable peak, while GPUs 1-3 aggregated correctly. Take the GPU index as the last whitespace-separated token of a[1]. Verified against a two-sample fixture: the old parser emits a spurious row per timestamp, the patched one reports 0/1/2/3 once each at the correct peaks. The end-of-campaign summary is the GPU-side input to the fv-ml1 circuit budget, so a silently wrong GPU 0 row is a measurement fault, not cosmetic. The in-flight campaign still runs the old on-host copy (editing a running bash script corrupts execution by byte offset); its summary will be recomputed from the raw power.log. |
||
|
vh
|
06a68f3f6e |
memory: raise the handoff staleness window from 1 h to 8 h (operator)
A one-hour window deleted the handoff unread across any overnight gap -- which is precisely the case the handoff exists for. Tonight's snapshot would have been discarded before the operator returned from the site visit. 8 h also matches the global CLAUDE.md and the /snapshot skill default, so the repo stops being the outlier. The reasoning is kept inline in the header rather than only in this commit message, because the next session reads the header and not the log. |
||
|
vh
|
816b8d4893 |
memory: snapshot — FV dark, Flash-Next seat built, power policy settled
Operator-invoked before heading to the Fountain Valley site visit. Rewrites Current state / in-flight around the outage: FV dark since 06:56Z including the BMC, will not self-recover, 19 of 30 gateway aliases down with no local fallback because every free local model lived on fv-ml1 and irv-ml1 runs no chat seat. The section now POINTS AT docs/runbooks/fv-site-dark-20260913.md rather than restating it, since the runbook is the single source of truth for the visit. Records what was built and verified before the power failed -- the Flash-Next seat on one card with its 51B n-gram table in host RAM, gen-large on the gateway -- and what is decided but unexecuted: the 250 W / 200 W power caps, the incomplete MTP campaign with one off_A rep banked, and the ten stale Homepage labels that the staged bring-up fixes as a side effect. Closes a stale in-flight claim: the 'fv-ml1 cannot initiate to fleet LAN IPs' gap is resolved, fixed by another session's scoped SNAT at 06:22Z with Beszel 18/18 verified. Adds ana-ml3 as its own subsection, since conflating it with fv-ml1 caused two rounds of wrong arithmetic today. Auto-archival fired (index was 431 lines, over the ~300 cap) but reached only 2 entries. The 14-day guard holds almost everything: this repo has had an exceptionally dense fortnight and nearly every dated entry postdates 2026-08-30. Of the six entries old enough, four carry open deferred-work pointers and were held back per the decision-loss guard. The file stays over cap, which is the documented trade -- an over-cap file that keeps live decisions beats a scannable one that lost one. |
||
|
vh
|
5a5f5c267e |
power: RETRACT the DCGM caveat — config management and power limits ARE supported on our cards
I claimed in
|
||
|
vh
|
100670eed1 |
power: what DCGM is, and why not to plan around it on workstation-SKU cards
DCGM is NVIDIA's own Data Center GPU Manager -- first-party, Apache-2.0, packaged as datacenter-gpu-manager -- and it layers above NVML rather than beside it: nvidia-smi is a thin CLI over NVML's per-GPU primitives, and DCGM is a daemon plus dcgmi adding health, diagnostics, config enforcement, policy and group abstractions on top. Which is why its group notion still resolves to N per-GPU writes underneath. The caveat that matters, and it undercuts the experiment suggested in the previous commit: DCGM is datacenter-oriented and parts of it are gated to datacenter SKUs of the Tesla/A100/H100 class. Our cards are professional/workstation parts -- RTX PRO 6000 Blackwell Max-Q and RTX 6000 Ada -- and several DCGM capabilities are unsupported or degraded outside that line, plausibly including config enforcement, which is precisely the power path. So DCGM_CONFIG_POWER_BUDGET_GROUP may return 'unsupported on this device'. Downgraded from 'worth testing' to five minutes of curiosity after the real work, and explicitly not a planning assumption. None of which touches the plan: nvidia-smi -pl 250 is plain NVML and works on these cards. DCGM would only have bought the group-budget experiment and nicer telemetry, and is probably not installed anyway since beszel-agent-nvidia shells out to nvidia-smi. |
||
|
vh
|
94fb7b7208 |
power: answer the bank-budget question — DCGM has the concept, the dynamic part is a control loop, and 4x250 already is 1000 W
DCGM_CONFIG_POWER_BUDGET_GROUP ('the power budget for the entire group') exists
alongside DCGM_CONFIG_POWER_CAP_INDIVIDUAL, so the concept is first-class. The docs do
not state how a group budget is distributed, and the deduction is that it cannot be
anything exotic: the only enforcement primitive underneath is NVML's per-GPU
nvmlDeviceSetPowerManagementLimit and there is no bank-level register, so any group
budget resolves to N per-GPU writes. Static even division is one write each; 'each card
free until they are all loaded' requires continuous re-writing, which is a control loop
rather than a hardware feature. Worth a ten-minute test when the box returns, in case
NVIDIA already runs that loop.
Records the design constraint that matters more than the logic: power readings lag and
-pl application takes tens of milliseconds, so a reactive daemon overshoots during a load
ramp -- and the ramp is the dangerous moment, being the same all-cards-at-once shape as
this box's ten restart:unless-stopped containers starting together. So any such loop must
be safe-by-default and opportunistic upward: boot at budget/N, only ever raise after
observing idle neighbours. Inverted, it works for weeks and then fails on precisely the
event it existed to prevent.
And the reason to defer it: 4x250 W is already 1000 W, so the static cap is the
conservative floor of the dynamic scheme rather than an alternative. The daemon's entire
contribution is the one-card-busy case, worth perhaps 5% throughput, which is rare for a
serving fleet that puts one seat per card and common only for a training window.
|
||
|
vh
|
b538fde6f0 |
caps: fv-ml1 250W / ana-ml3 200W — and nvidia-smi -pl caps BOARD power, not wall power
Operator set fv-ml1 at 250 W per card (83% of TGP, ~5% throughput) and ana-ml3 at 200 W (67%, ~10-15%). Records the term that decides whether 250 W actually clears a 15 A feed, because it is easy to drop: a power limit bounds BOARD power, and the wall sees that divided by PSU efficiency. Four cards at 250 W is 1000 W of board; add 180-300 W of host components and divide by ~0.90 and the plug sees ~1310-1445 W, against a 15 A circuit's 1440 W NEC continuous derating -- an inference box serving all day being a continuous load. So 250 W lands ON the limit rather than under it, where 200 W would give ~1090-1220 W with real margin. The deciding term is the host draw, which is still an estimate, so the procedure is: set 250 W, verify at the plug under four-card load, fall back to 200 W if it reads near 1440 W. A cap is a claim; the ammeter is the verification. Two consequences recorded alongside. Caps bound sustained draw and not transients -- the enforcement window is short but not instantaneous -- and while a breaker's thermal-magnetic curve forgives brief overload, a UPS's overload protection does not. So 250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA unit, which it exceeds even capped. And ana-ml3's 200 W across only two cards is deliberately conservative at 400 W total, relaxable if Anaheim's measured headroom beats its trip history. |
||
|
vh
|
2da0c76d99 |
correct the hardware: fv-ml1 is 4x Blackwell Max-Q 300W, ana-ml3 is 2x Ada RTX 6000 — and four cards is a breaker problem
Operator clarification, and it separates two boxes I had been conflating. fv-ml1 is 4x Blackwell RTX PRO 6000 Max-Q at 300 W each (Max-Q being the reduced-TGP SKU; the Workstation Edition is the 600 W part), 391 GB VRAM, deployed and currently dark. ana-ml3 is 2x Ada Generation RTX 6000 at 300 W, 96 GB VRAM, not yet deployed. The 200 W cap directive is ana-ml3's. With the TGP known, the outage stops being a vague 'undersized' and acquires a mechanism: two Max-Q cards at 300 W is ~600 W of card, plus a host carrying 566 GB of RAM, drives, fans and PSU conversion loss at perhaps 200-350 W, against an Eaton 1500 VA's real ~900-1200 W. That lands at or just over the rating, which is precisely what explains a full day of service on one card and failure minutes into the second. The host term is the only one being guessed; idle-at-the-plug measures it directly. It also surfaces something that is not a UPS question at all. Four cards at 300 W plus ~300 W of host is ~1500 W against a 15 A circuit's 1440 W continuous derating, so four cards uncapped is marginal on the breaker with no UPS in the path. Capping therefore belongs at fv-ml1 as well as ana-ml3, or fv-ml1 needs a 20 A feed -- and worth noting today's incident only ever had two of the four cards working. ana-ml3's placement constraints sharpen too: sm_89 has native FP8 but no NVFP4, so the in-house NVFP4 quants stay at FV, and at 96 GB total it cannot host the Flash-Next seat at all -- that needs 74 GiB resident on a single card, and the offload moves the n-gram table rather than the experts. |
||
|
vh
|
8fcc26e2c9 |
policy(gpu-power): cards are RTX 6000 Ada at 300 W — 200 W is a mild cap, plus two sm_89 placement consequences
Corrects the SKU: RTX 6000 Ada, 300 W, not the ~600 W initially recalled. That makes 200 W a cap to 67% of TGP -- the favourable part of the concave perf/watt curve, roughly 10-15% of throughput -- rather than the severe 33% cap a 600 W part would have implied, and it very likely sits above the card's enforceable floor, so the check becomes a formality rather than a gate. The protective value is worth stating: four cards at 300 W uncapped is ~1200 W, which is roughly the neighbourhood that overwhelmed a 1500 VA unit at FV with only TWO Blackwell cards drawing. Capping to 800 W makes a repeat of today a non-event. Two consequences that follow from Ada independent of power, and both are placement constraints rather than details. sm_89 has native FP8 but NOT NVFP4, which is Blackwell-only -- so the in-house NVFP4 quants that most of this fleet runs will not be accelerated on that colo's cards, and its seats want FP8 W8A8 builds or the NVFP4 checkpoints stay at FV. And it unparks the triton-backend item, which is a hard no on Ampere because fp8e4nv is unsupported on sm_86 and was explicitly deferred to Ada; sm_89 has what it needs. VRAM is 4x48 = 192 GB against fv-ml1's 391 GB, so big-model placement stays at FV. The Flash-Next seat needs 74 GiB resident on one card and would not fit a 48 GB Ada card even with the n-gram table offloaded -- the offload moves the table, not the experts. |
||
|
vh
|
3e61d7d4e0 |
policy: cap GPU power limits at build time — 200 W for the other colo's cards
Operator directive, and the right generalisation of the FV outage: decide the power envelope first and size the cards into it, rather than installing cards and discovering the constraint by tripping it. Four cards at 200 W is 800 W, which fits a real circuit with a real UPS and headroom. Records three things to settle before it is a plan. First, 200 W may sit below the card's enforceable floor: nvidia-smi -pl is bounded by Min Power Limit, often around half of TGP on a high-TGP part, and a sub-floor request is refused -- quietly, depending on how it is scripted. Run nvidia-smi -q -d POWER before any build planning depends on the number. Second, the 600 W figure wants confirming against the actual SKU. The Ada parts do not land there -- RTX 6000 Ada is 300 W, L40/L40S 300/350 W, 4090 450 W -- while 600 W is Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or the figure may be a two-card total. Read it off the device rather than a spec sheet. Third, the workload asymmetry is in this fleet's favour: decode is memory-bandwidth-bound and tolerates a cap far better than training does, with a concave perf/watt curve where 60-70% of TGP costs roughly 10-15% of throughput. A cap to a third of TGP is deeper into the steep region; measure it on the first card rather than predicting, and expect prefill-heavy and training work to pay more than a serving seat. And persist the cap. A hand-set limit holds until the next reboot and then silently stops holding, which is the worst shape available given that the thing rebooting the box is likely to be the power event the cap existed to prevent. |
||
|
vh
|
b6335bf6ad |
runbook(fv-outage): the circuit case — split power survives a trip on battery, but only if the colo handoff does
Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits into two halves with different answers. A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a load-bearing supply and goes back to being what it is for. What it does NOT cover is the colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just tripped, the outcome is a firewall running on battery with nothing upstream to talk to and the drive happens anyway. Added as a question for the facility, because it decides whether split power delivers remote diagnosis or merely feels like it does. Records the case where none of it matters: removing an undersized UPS does not remove the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which is what makes that single ammeter reading the load-bearing measurement of the visit. Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room the dial has, and persist any cap -- one that evaporates on reboot will hold right up until the next power event and then silently stop holding. |
||
|
vh
|
00b842bb9b |
runbook(fv-outage): operator ruling — undersized UPS; NAT demoted; ammeter protocol for the visit
Operator's reasoning, accepted and better than the hypothesis-space argument it replaces: the NAT change went effective, was verified bidirectional, and then ran correctly for twenty minutes before the site died the moment GPU load was applied. A working config change does not spontaneously fail under an unrelated physical variable. The load correlation is tight; the NAT correlation is merely adjacent in time. Undersized UPS is the only candidate that explains the trigger. NAT material retained as record, and the power.log/uptime check demoted from decision point to free confirmation. Adds the measurement protocol, since the operator is bringing a PDU and an ammeter. The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU, 566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is the primary instrument and power.log only cross-checks the GPU share. Four states to capture (idle, one card, two cards, four cards), and capture PEAK rather than average: UPS overload protection responds to short-term overload, so an average-only reading that hides transients will mis-size the replacement exactly the way the present unit got mis-sized, and must be recorded as a floor rather than as the draw. The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the cutover's own open question -- that the FV circuit was likely specced against half the real draw, back when every record still said the box had two GPUs. |
||
|
vh
|
59ddedd980 |
runbook(fv-outage): a NAT change 34 min earlier means power is not established — and power.log settles it for free
Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34 minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding, and nobody should buy hardware on it until the discriminator below has been read. On the evidence that change is the wrong shape to have caused this, and it is recorded as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a single source /32 and a single destination /16 cannot stop the gateway, the BMC or the public WAN address from answering inbound; no routes, filter rules, WAN settings or subnet advertisements were touched; pfctl -sr came back byte-identical; and it was verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up. Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50, a different subnet. They correctly declined to claim BMC health, but the observation is void rather than negative and should not be reasoned from. The discriminator costs nothing and is already on disk: power.log is written locally to /tank every 10 s by a shell loop on the box and does not depend on the network. Entries past 06:56Z mean the machine never lost power, which makes this a routing fault and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory outright. So the first action on site is now to READ, not to fix. The two hypotheses lead to completely different remediations and only one of them needs a new UPS. |
||
|
vh
|
524da702ef |
flash-next-mtp-bench: run on GPU 3 / :8023 without touching the production seat
Operator-directed so the campaign could run in parallel with live gen-large testing. Strips the production-restore tail entirely -- the only container the driver can create or remove is fn-mtp-bench, and the sole remaining compose call is a read-only config --format json to derive argv. Adds per-card power+memory sampling every 10 s to power.log, because two cards under load was the risk being accepted and a record beats an argument. That power log is now the only surviving measurement of what the load drew before Fountain Valley went dark; it lives on /tank rather than in the container. |
||
|
vh
|
312725ddfb |
memory: snapshot — Flash-Next seat on one card, and the FV outage that followed
Durable capture so tomorrow's session does not have to reconstruct either half. Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA -- 74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144 context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end. The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes the paused worker-based #53899 and designs out its entire bug family; text_config.ple_embedding_dtype is the load-or-fail discriminator for any community build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin nearly OOM with no visible failure; MTP is off pending measurement here rather than written off, because the recipe's number is cross-harness and tested k=3 only while the head is one layer run autoregressively; and a container once reported (healthy) with no published port at all, because the healthcheck runs inside the boundary it was trusted to validate. Then the outage. Records it as will-not-self-recover, so no session wastes effort polling a dead site, and carries the three things that change the visit: bypass the UPS rather than using its surge-only bank (both banks share one 12 A inlet -- the surge bank bypasses the inverter, not the current rating), recover power.log before anything else because it is the only load measurement that exists anywhere, and bring seats up one at a time because ten restart:unless-stopped containers loading at once is the largest transient the box can make into whatever just failed. Also records what is still half-done: the stale homepage labels on the 10 containers that died before they could be recreated, which the staged bring-up fixes as a side effect, and the eight drifted stacks plus three untracked host-only stacks that were deliberately left for a deliberate reconciliation. |
||
|
vh
|
d79f10457a |
runbook(fv-outage): UPS overload as leading hypothesis, site-visit bring-list, no-local-fallback correction
Operator's read is that the UPS the box was plugged into overloaded and died, and it fits better than the breaker-trip theory: a UPS's output rating sits far below the circuit's, so it is the first protective device to give -- which explains why the site let go at TWO cards loaded rather than four, and why the ~25 W firewall died with it. Records the operationally important consequence: a tripped UPS resets, an overloaded one can kill its output stage permanently. If it is dead, nothing on site can be reset back to life, so the visit needs the means to BYPASS the UPS or it is wasted. Elevates recovery of /tank/.../power.log to the first action on site. It sampled all four cards every 10 s up to the cut, lives on /tank rather than in a container, and is the only measurement of what the load actually drew -- without it a replacement UPS gets sized by guesswork. Also states that no load figure exists yet, only idle. Corrects an earlier claim of mine in this session: there is NO local fallback for the 19 dark aliases. Probed -- every free local model is on fv-ml1, and irv-ml1 runs no chat seat at all, only TTS/ComfyUI/arbo/clipper work on two partly-occupied Ampere cards. The only non-fv chat backends are paid. Any paid coverage must go under a new opt-in alias name rather than a silent repoint of summarizer/gen/classifier. |
||
|
vh
|
969a1b64a2 |
runbook: FV site dark 2026-09-13 — outage facts, blast radius, staged recovery, OOB design gap
Written while the site is down so recovery does not have to be reconstructed later. Records what was measured rather than what is suspected: every FV address including the BMC is unreachable while all three other sites answer, the campaign's last log line was off_A rep 2 at 06:56:04Z, and the site was dark by 06:58:40Z. Names three candidate causes with the evidence that would distinguish them, because the instrument that could have settled it -- the per-card power log -- died with the box. The two-card-load hypothesis fits the timing and the two prior Anaheim breaker trips on this same chassis, but it is circumstantial and is recorded as such. Carries the recovery hazard that matters: every seat on the box is restart:unless-stopped, so resetting power alone brings ten vLLM containers up loading at once -- the largest transient the box can produce, into a circuit that may have just tripped. Staged sequence given, gen first and flash-next last. Also records the OOB gap the outage exposes: OPNsense-as-subnet-router protects against box-down/gateway-up, and not at all against the site-wide loss that actually happened, because the BMC's only path out is through that same gateway. |
||
|
vh
|
7e62a07341 |
flash-next-seat: full 262K context, KV pinned at a measured 14 GiB, gen-large on the gateway
Operator-directed: raise context to the model's native maximum and take as much KV as the card safely allows, and expose the seat through LiteLLM as `gen-large`. max_model_len 131,072 -> 262,144 KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens) concurrency 2.54x@128K -> 2.14x@262K ⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted against 403 MiB free. The request completed, so nothing failed visibly; that is one step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak. The reason the first estimate was wrong is worth keeping, because it is not obvious and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting for deep-prefill activation, so pinning bytes switched off the protection that the pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu memory" line is computed from a profile measured at max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request survives; open #54764 compounds it, since PLE short-conv prefill pads every request in a batch to the batch-MAX query length. max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because the peak tracks chunk size, not context length. Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One alias on purpose -- a single alias cannot trip the shared-config enable_thinking mutation footgun, which needs two over the same (model, api_base). Sampling is the checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and repetition_penalty are left unset because the checkpoint declares no canonical value for them. Verified registered for both the infra-ops admin key and the shared all-agents key, since a new model behind a scoped allowlist 403s silently. Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's 4xH100 recipe number, which is a cross-harness comparison and not evidence about a TP=1 Blackwell seat. |
||
|
vh
|
3132a16ca0 |
fv-ml1: finish the renumber the cutover missed -- 16 dead dashboard links
Every fv-ml1 link on the Homepage dashboard was broken. Measured against the
live dashboard API before the fix: 16 entries pointing at the dead 10.250.50.54
and zero at the live 10.251.50.54, covering gen, M.O.G.-SEC, Scriberr, Embed,
Rerank, Reward, Coder, Dockge and six dormant seats.
The miss was structural, not careless. fv-ml1-rename-sweep.sh works from an
allowlist assembled from files that mention the HOST, and a homepage.href label
mentions only an IP -- so every stack whose sole stale reference was a label
fell outside it. The allowlist now covers those 24 files, and records how to
derive the list next time (grep the old address, subtract history) rather than
enumerating from memory.
History is still untouched, and the exclusions are now written down with the
reason each one keeps the old address: recorded benchmark results, whose
base_url is part of a measurement's provenance; the one LiteLLM comment
preserving a retired hand-test endpoint; and the cutover runbooks, where the old
address is the subject matter.
Two bugs found while applying it, both fixed here:
- deploy-stack.sh rejected any stack name containing a dot, so qwen3.5-122b,
qwopus3.5-122b and mistral-medium-3.5 could not be deployed by the script at
all. The check exists to stop path traversal, which means rejecting ".." and
"/" -- not every dot. Traversal is now rejected explicitly and tested.
- stacks/scriberr/.env.example allowed CORS only from the dead IP and from
scriberr.ana.internal, which no longer resolves; the box is at the fv site
and DNS already carries scriberr.fv.internal. The live .env had both stale
origins, i.e. an allowlist with nothing reachable in it.
Host side, applied separately: canonical pushed for the 16 stacks whose only
difference from the host was this renumber, and an in-place address-only fix for
the nine whose host copy has genuinely drifted or has no canonical copy, so that
drift survives for a deliberate reconciliation instead of being clobbered. Every
compose.yaml on fv-ml1 now reads 10.251.50.54. The labels themselves only take
effect at container creation, so the running containers still need recreating.
|
||
|
vh
|
f964a47262 |
flash-next-seat: Qwen3.8-Flash-Next on fv-ml1 GPU 2 with the n-gram table in host RAM
First seat whose weights do not fit its card. The model is 176B total -- a 125B main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The table is a pure embedding lookup, so it lives in pinned host RAM and the GPU reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing. Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source precision. Chosen over better-liked builds because its provenance states protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP tensors -- and because it declares text_config.ple_embedding_dtype, which is the field vLLM reads first when selecting the PLE weight format. Builds that ship an FP8 table without that declaration resolve to the unquantized path and fail on load; the README records the check. Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit. Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is paused upstream and is not the path here -- its deadlocks, ptrace gate and stale-output-under-graphs bugs all came from the separate worker process that UVA does not have. Five deliberate departures from the other seats on this box, each from a measurement rather than a preference, all annotated in place: - no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100 (8-36% less throughput, 32-173% more latency, ~36% acceptance) - modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the ignore list keeping attention, shared experts, PLE and MTP out of W4A4 - KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426) - mamba-cache-mode stated explicitly: the model raises on mode "all" - 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919 make depth the risky axis, and sizing to the KV pool has never fixed a depth-driven crash on this hardware Nothing is wired into LiteLLM. Pointing an alias at this seat changes what existing callers receive and is a separate decision. |
||
|
vh
|
a5691ce796 |
restic(esh-vm-db): a backup that exited 0 nightly while keeping an April dump
Work by a parallel session on 2026-09-12; committed here with the rest of the day's changes. Rationale in persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md. The only visible symptom was a systemd-failed unit from a Sep 6 repository network timeout after boot. The real fault was quieter and much worse: the pre-backup hook logged failures as WARN and returned zero, so pg_dumpall could fail every single night -- it used TCP localhost and wanted a password nobody supplied -- while restic dutifully backed up the stale April 23 dump still sitting in the staging directory and reported success. Mongo was fine, which is part of why it went unnoticed. Postgres now dumps over the /var/run/postgresql socket with peer auth and -w, and both database failures now fail the backup rather than masking it, while still preserving any prior per-DB dump rather than truncating to nothing. An ERRORS counter replaces the warn-and-continue path, and the staging directory is overridable via RESTIC_STAGE_DIR so the new test can exercise it. Adds backup.contract.md, retry.conf and test_pre_backup.py -- three red-green regression tests covering the failure modes above. systemd drop-ins on both jobs add network-online ordering plus Restart=on-failure with a 5m delay and a 3-per-hour limit, which addresses the original boot-timeout symptom. Verified against a real run: snapshot bc5eeaff at 07:01 PDT with a fresh 3.46MB PG dump, retrieved from the repository with decompression and completion marker checked (not a full restore). Repository check passed, 99 snapshots. The old hook and stale dump are preserved root-only at /var/lib/restic/repair-20260912. |
||
|
vh
|
3e727dbeb5 |
memory: snapshot — FV cutover executed, BMC online, four GPUs not two
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1 live on 10.251/16 and serving, BMC recovered after finding it was tagging 802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB count that invalidates prior sizing assumptions. Also records two things that cost real time and would cost it again: OPNsense write APIs need an X-CSRFToken scraped from a script block, not a hidden form input, and a 403 that coincides with the box going down looks exactly like a successful reboot. Auto-archived 8 settled entries (Recent decisions: 8). The index stays over the 300-line cap at 385 because the guards hold -- nearly every remaining dated entry is under 14 days old or carries open deferred work, and an over-cap file that keeps live decisions beats a scannable one that lost any. |
||
|
vh
|
48e34e1849 |
playbooks: repoint the four remaining live fv-ml1 targets
homepage-regroup, mog-sec-move-to-gpu0, pull-hf-repo and serve-qwen3.5-122b all carried runnable 'scripts/elway ana-ml2 --playbook ...' instructions or the old 10.250.50.54 address. Each would fail today against a dead name and a dead IP, so these are corrections rather than cosmetics. homepage-regroup is renamed to match; the other three keep their names, which never carried the host. |
||
|
vh
|
87d7555fcf |
fv-ml1: the playbook + homepage body edits the rename commit missed
A zsh glob failure in the staging step of |
||
|
vh
|
17586abb19 |
homepage: retire the fv-ml1 docker relay, its precondition is now met
stacks/ana-ml2-proxy existed for one reason, stated in its own README: ESH could not reach the box's Docker API cross-site because of a return-route gap, so ana-docker relayed :2354 -> :2375 from the same subnet. The README named its own retirement condition -- "once it has a working return route to ESH, point Homepage back at it directly and compose down this stack". That condition is met. FV is on the mesh, vb-gateway accepts routes, and the NH3 UDM, ESH UDM and Anaheim FortiGate all carry 10.251.0.0/16 statics toward their local scale nodes. Verified before removing anything: esh-docker-vm opens a TCP connection to 10.251.50.54:2375 directly. Homepage now points at 10.251.50.54:2375 and discovers 112 services across 20 groups, so nothing regressed. The socat container is stopped and removed from ana-docker; the compose file and README are deleted here rather than renamed, because renaming would preserve a moving part that no longer earns its place. |
||
|
vh
|
91bda3c480 |
fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports. |
||
|
vh
|
fde7834635 |
docs(fv): on-site playbook for the colo build, and the redundancy it does not have
fv-ml1-cutover.md covers what to change; this covers the on-site hour, where the binding constraints are limited time and not locking ourselves out of a site an hour away. The load-bearing correction is that fv-ml1 running tailscale is NOT a second independent path. Its default route is the OPNsense LAN address, so its node needs egress through OPNsense to reach the control plane; OPNsense down means fv-ml1 is unreachable by construction. An earlier version of this plan claimed otherwise and proposed proving it by rebooting the firewall, which would have failed on site with the operator standing at the rack. He caught it. What the fv-ml1 node does buy is narrower and still worth one command: it covers control-plane lockout -- a bad admin rule, a WebGUI on the wrong interface, a lost password -- where the box still routes but cannot be managed. It does not cover data-plane failure. Those are different failure classes and conflating them produced the wrong test. The gate is now a deny-admin-from-mesh test run while the console is in arm's reach, which exercises the bypass that actually exists. Deliberate deviation from the cutover runbook: flat /24 with the BMC at 10.251.50.50 rather than a mgmt VLAN at 10.251.250.50. The VLAN needs OPNsense config plus switch tagging during the scarcest hour, for something addable remotely in ten minutes. Also records the DNS chicken-and-egg (the mesh needs public resolvers because the fleet AdGuard is unreachable until the mesh is up) and the interface auto-assignment trap measured on the lab VM the same night. |
||
|
vh
|
2dd459d2e5 |
memory: Worldtree memory-split flip protocol, and all three deployments measured writable
worldtree-dev's U6 reader refuses at boot if it cannot append and read back <memory root>/reader/canary.jsonl. Per-euid subdirectories are lazy and only warn, so that root canary is the single boot-blocking check -- which makes the memory root's writability by the container uid the precondition worth knowing before a flip rather than during one. Protocol agreed with worldtree-dev: neither memory.reader.enabled nor memory.writer.enabled gets flipped on any deployment without infra-ops confirming that writability first. Both ship dark until the operator schedules the tracer skeleton. Measured tonight on corviduo-dev, all three pass. Also retracts a wrong prediction I sent earlier in the thread: personal runs as root, not uid 1000, and pinned is the only uid-1000 deployment -- it passes regardless because /data/state is owned 1000:1000. Recorded with the caveat that a permissions reading is a claim about its own date, so the probe gets re-run immediately before any flip rather than cited from tonight. |
||
|
vh
|
ddfa1a6e5b |
feat(beszel): commit the fleet agent rollout — configs, playbooks and runbooks
Beszel agents are installed and verified across the fleet but the artifacts that produced them were never committed, so the deployment existed only on the hosts. Adds the per-host agent environment files (PORT, NICS, EXTRA_FILESYSTEMS and the hub's PUBLIC key), the systemd unit, the guest install script, the Synology compose, and the elway playbooks for native, guest-stage, guest-install and Synology paths. The two dated memory detail files covering the priority-1 and priority-2 waves ship alongside, per the convention that memory lands with the work it describes. No credentials here. The KEY= value in every host env is the Beszel hub's public ed25519 key, identical across all nine and public by design; the agent README says so explicitly. The nh3-nas sudo password referenced in the runbook prose lives in Vaultwarden and the helper scripts named there never contained it. ⚠ Overlapping VMIDs across hypervisors are a standing trap and are recorded in the priority-2 notes: pfi-pve 105=postgres and 100=pbs-ana, nh3-pve 105=pbs-nh3. ⚠ PBS-NH3's export was ~75.5% used at capture; resource checks are not job success monitoring and should not be read as such. |
||
|
vh
|
88e171bea6 |
fix(esh): Plex hardware transcoding on the Arc A580, and the two ways it hid
Plex never hardware-transcoded on esh-pve-nas LXC 105 despite correct passthrough, cgroups, group membership, authenticated HuC firmware, a lifetime Plex Pass, HardwareAcceleratedCodecs=1, and the Arc already selected as HardwareDevicePath. Root cause sat below all of that: intel-media-va-driver 22.3.1 (Apr 2023, stock jammy) predates Arc/DG2 support and exports only __vaDriverInit_1_14, against the libva 2.22 that Plex bundles and loads via RPATH rather than the system one. A half-finished prior attempt at the same fix was also present -- libva and libva-drm hand-installed at 2.22 with libva-x11 left at 2.14, breaking every X11 VA-API consumer on va_fool_postp. Fixed with Intel's client-GPU repo: iHD 24.3.4 (__vaDriverInit_1_22, an exact ABI match) plus a consistent libva 2.22.0.2-87 set, which also brings the orphaned manual install back under dpkg. The repo track is rolling, so the six packages are pinned in /etc/apt/preferences.d/intel-gpu-pin and apt-mark held; verified by a simulated upgrade that moves 152 packages and touches none of them. Two findings worth more than the fix: pct snapshot refuses on a guest with a bind mount and still exits 0, so a script guarding a change with it proceeds without the rollback point it believes it has. The ZFS dataset snapshot is the working path, verified by reading it back. A synthetic Plex Transcoder invocation is not a valid test of Plex's transcode path. Plex bundles its own libc among 61 libraries; the harness produced three distinct failure modes that were artifacts of not reproducing that runtime, and it failed identically before and after a fix that worked. With no positive control its negatives carried no information. Only a forced transcode settles it, and PASS is recognisable by Plex naming the device. The original empty decoder/encoder line was an absence of evidence rather than evidence of failure -- TranscodeSession was 0. Jellyfin LXC 107 has the same stale stack and the same Arc available; left alone per the operator, and it ships its own ffmpeg so this may not transfer verbatim. |
||
|
vh
|
e01484f6d0 | memory: consolidate backref lines to true archival counts (was triple-counted) | ||
|
vh
|
19aeaedeba | memory: snapshot — fv-ml1 cutover prepped, Anaheim recovered, BabyYarros complete; archived 5 settled entries | ||
|
vh
|
1d435c9e33 | memory: ana-ml2 -> fv-ml1 Fountain Valley relocation + leave-rack-dark decisions | ||
|
vh
|
8400f3aef5 | scripts: pre-script the ana-ml2->fv-ml1 rename sweep (scoped, dry-run default, history-safe) | ||
|
vh
|
ce04f9dbde | docs: finalize fv-ml1 cutover — confirmed addresses, piggyback DNS, exact commands | ||
|
vh
|
a13ab598ae | docs: fv-ml1 cutover runbook (ana-ml2 -> Fountain Valley colo) | ||
|
vh
|
1405363649 | memory: Anaheim recovery — crowdsec/traefik fail-closed 403 fix, ana-ml2 still needs power | ||
|
vh
|
870f42960e | memory: BabyYarros complete — both arms trained + evaluated, voice moved, beats 9/10 | ||
|
vh
|
5558d9c7d3 |
R49/BabyYarros: voice + beat evaluation tooling and the seat-free delta_cb
Adds the Yarros-side evaluation the training exists to justify: does the adapter move arbitrary prose toward Yarros, and can the instruct arm still expand a beat to a paragraph on direction. Yarros-flavoured voice prompts (modern/neutral/ romantasy tiers so any Yarros voice in the modern tier is adapter-attributable, not prompt-supplied) and a Yarros-register beat SYS on the chat generator. voice_distance.py is the honest slice of adjudication that needs no seat: Burrows's Delta over character bigrams against held-out Yarros. Its first cut mis-framed the noise floor — it used the same-author distance (held-out vs itself) as the between-arm significance threshold, which is the target, not the threshold. Fixed to the measured floor: the within-arm seed spread, which is this metric's sampling variance at this sample size, computed from the two seeds already generated rather than assumed. Result on the built corpus, ordering base-125-tuned < instruct-tuned < base-unadapted, both adapters clearing the 0.046 measured floor (base +0.157, instruct +0.076), and the ordering corroborating the independent held-out loss ordering (Base below Instruct). One seed-pair per arm, so it corroborates rather than settles; the full frozen adjudication still needs a romantasy control panel, a second seed, and the gen seat for the beat-incumbent leg. |
||
|
vh
|
61ddcfd9a6 |
CLAUDE.md: remind this session it IS infra-ops, not a party to page
The recurring incident-time failure: during an outage the session reaches for althing to notify infra-ops, and since ALTHING_HANDLE is infra-ops the send mails itself, the herald pokes about new mail from infra-ops, and the session triages its own words as a peer report. Documented the identity, the check ($ALTHING_HANDLE before addressing an ops message), and the one legitimate use of the infra-ops thread during an incident (a durable memo to a future infra-ops session on recovery, not a page to a peer). |
||
|
vh
|
fa6e2c8228 |
memory: Sentinel-R3 pulled, MTP-grafted, quantized — a better-fit pen-test seat candidate
Records the comparison (Sentinel-R3 is a real agentic-pentest SFT on the same Qwen3.8-27B base vs M.O.G.-SEC's persona-on-stock-weights, on a tool surface that matches our own harness), the HF check (M.O.G.-SEC repo unchanged), and the graft: Sentinel ships no MTP head, so the verbatim base head was grafted in, which is the correct lineage but leaves acceptance unverified because the head now sits on an SFT-finetuned body rather than stock weights. The quant is done and structurally verified; serving and the acceptance/A-B gate are blocked on GPU space and are the operator's next call, along with the proprietary-license posture. Detail in the artifact's PROVENANCE.txt. |
||
|
vh
|
5ccb6c3118 |
memory: concurrent training on pfi-gx10 is net slower than serial
VRAM said yes and the box said no. Two 4B LoRA jobs fit easily -- 36 of 121 GiB with 98 free -- but both dropped from a 37.10 s/it solo baseline to ~85 s/it, 2.29x each, so combined throughput fell below serial. The GB10 is past its memory-bandwidth roofline and pays a contention penalty on top of the split. Worth keeping because the intuition it corrects is a common one: free GPU memory is the number everyone reaches for, and on this box it does not predict whether a second job is affordable. The control is what makes it a finding rather than an anecdote -- killing the second job returned the first to 37 s/it on the next step. |
||
|
vh
|
38eb40ace4 |
BabyYarros: stage the 4B-Base comparison arm, both concurrent and chained
One arm is not a comparison. On the Brontë corpus Base won held-out loss (2.814 against the instruct arm's 2.908) while instruct won instruction-following, and that trade is the open question for Skaldsong. It has to be re-measured on this corpus rather than carried over: BabyYarros is 12% larger and contemporary. Two entry points because the right one depends on a measurement rather than an argument. The box has 98 GiB free against an 18.4 GiB training footprint, so VRAM is not the constraint -- but VRAM was never the binding constraint on a GB10 that onboarding measured at 6x slower than ana-ml2 where compute predicts 2.7x, and where batching was not a throughput lever. If the box is already at its bandwidth roofline, a second job splits the same bandwidth and buys nothing. So: launch-yarros-4b-base.sh takes an explicit --allow-shared-gpu that bypasses the GPU-clear guard, and chain-yarros-4b-base.sh waits on the instruct run and refuses if it produced no adapter. The bypass is an argument and never a default, because the guard's normal job -- stopping a chain from firing into a live run -- is what keeps one lost run from becoming two. A shared-GPU launch stamps its own log with a warning that its s/it is not comparable to a solo run, since the harness is part of the number. Both carry the same gate guard as the instruct arm: refuse to start unless the leak gate report on disk says PASSED. |
||
|
vh
|
969d649a24 |
memory: BabyYarros unblocked — gate passes 0/325, instruct training live on gx10
Records what the gate cost to build and what it found: a second typography defect (small-caps epigraphs and drop caps, which the D1 "no unwrap needed" note was right about in the wrong direction), back matter inside the prose of all five works naming real people, a possessive-only entity that was unrenamed and unreported at once, a character lost to an in-book article typeset in lowercase, and a phrase class the unigram scan structurally cannot see. Also records the two things that will matter next time: every new detector flag is opt-in and the Brontë map was re-derived and diffed after each change, because one intermediate version silently dropped four real names from Shirley; and the gate's sensitivity floor is part of its result, not a footnote. |
||
|
vh
|
7b0580dcbe |
BabyYarros: the leak gate passes, and it found three defects nobody was looking for
The gate is new. There was no committed instrument for "does any of the author's own proper nouns survive the rename" -- the Brontë number was produced by hand -- so leak_gate.py is now that instrument, and it runs both directions every time: the same scan over the unrenamed source as a positive control, and a nonce string as a negative one. A detector that only ever sees renamed text cannot distinguish absent from blind. Run against BabyYarros as built it reported 212 surviving entities, not the 86 recorded earlier, because it scans the whole corpus rather than each work separately and it counts the sub-threshold entities rename never looked at. Three findings came out of closing that. The corpus had a typography defect of its own. The D1 notes correctly say no unwrap was needed; a different defect was there instead. The Empyrean books set their chapter epigraphs in small caps and the extractor rendered the run as uppercase while leaving the large initial as a separate token, so the corpus carried "M AJOR A FENDRA'S G UIDE TO THE R IDERS Q UADRANT" -- 106 lines, ~700 splits -- plus 52 drop caps like "T he flight field". That is where the entities called IDERS, UADRANT, NAUTHORIZED and seventeen bare single letters came from. A split initial next to an uppercased run is enough to recover the original mixed case, so the restore is exact rather than approximate: a word with a split initial was capitalised, an all-caps word without one was lowercase. Back matter was inside the prose. The builder splits on chapter headings and nothing follows the last one, so every work carried its acknowledgments, newsletter pitches and cover-artist credits -- 4,555 words naming the author's agent, editors and children, in a corpus whose entire purpose is that no identifiable name survives. And the gate passed at 0 of 314 while Afendra was still in every copy. The name never appears unpossessed, so it keyed as an apostrophe form, and rename and the gate both skip those as contractions -- unrenamed and unreported at once, which is the worst failure shape available. Baxter escaped a different way: wilder renders an in-book news article entirely in lowercase, putting the cap/lowercase ratio at 0.13 against a 0.05 bar. Then a second class the unigram scan structurally cannot see. Riders Quadrant, Flame Section, War Games and Fourth Wing -- the book's own title -- are built from ordinary words the detector correctly refuses to call names. The gate now audits recurring capitalised 2-3grams against an explicit allow list, and rename applies a phrase map after the entity pass. Every new detector flag is opt-in and off by default, and the Brontë entity map was re-derived after each change and confirmed identical in keys, surfaces and every field. The stoplist was built by reading each surface in context, which is why it is short: Violence is Xaden's nickname for Violet, and Continent, Presentation, Barrens, Originals, Montserrat, Athena, Aura, Curator and Sage are all in-world. A plausible-looking guess would have excluded most of them. Final: 0 of 325 entities and 0 of 91 audited phrases survive in any of 30 copy files, both controls passing. The sensitivity floor is stated in the gate's own output -- 3 occurrences for a name, 5 for a phrase -- because a negative without one is unfalsifiable. |
||
|
vh
|
16c144fcda | memory: snapshot — BabyYarros blocked on the leak gate; R49 sweep complete; mog-sec settled | ||
|
vh
|
6dba912324 |
BabyYarros: corpus built, gender resolution fixed, rename blocked on leak gate
Located the source: five Rebecca Yarros works in the Kvasir licensed library, with rights recorded as gated. Built D1 at 208 chapters and 780,744 words, which is 15% larger than the Brontë corpus. No unwrap step was needed because Kvasir's cleaner already emits flowing paragraphs, so the hard-wrap defect that cost a re-cut on Brontë does not exist here. The alphabet was re-derived rather than inherited: 23 non-ASCII letters across three forms, against F02's 4 on a smaller sample. Same ASCII-fold conclusion from a different measurement, which is the reason to re-derive per corpus. The interesting finding is a new pathology. In a rotating first-person POV corpus, every book's narrator gets the wrong gender. Measured against six names verified in the text, the pronoun resolver called Violet male, Leah male and Landon female -- three of eighteen wrong, and all three are the narrator of the book where they were misgendered. A narrator is "I" in her own book, so her name appears mostly inside the other lead's dialogue surrounded by his pronouns. This is Brontë's "Jane called male" amplified by rotating POV. Title-first resolution, which fixed it for Brontë, is nearly blind here because contemporary romance uses given names rather than honorifics. What works is the POV header: resolve each name from the chapters it does not narrate. Validated at 9 correct, 9 held, 0 wrong against the previous 7, 8 and 3 wrong, and the instrument refuses to write unless it beats what it replaces. Re-pointing rename.py surfaced three bugs, two of which would have silently corrupted the corpus. Gender came only from honorifics and the entities file's gender field was ignored, so the POV fix had no effect until wired through; that took wilder from 1 gendered entity to 13. The pool labels were hardcoded in a print statement, so any non-Brontë preset crashed. And the collision-filter log claimed it dropped names colliding with Brontë entities regardless of which corpus it filtered against -- the logic was right but the message named the wrong corpus, which is how a reader later concludes the filter ran on the wrong thing. D3 is blocked and nothing has been trained. The leak gate shows 86 of 232 renameable source entities surviving where the Brontë run reached 0 of 203. It decomposes into detector false positives that need a stopword filter rather than renaming, genuine misses among worldbuilding proper nouns, and a third class whose cause is not yet established. Training before the gate passes means fitting in-copyright text with 86 identifiable source entities intact, in a corpus F02 already flagged as small enough for leak to be a real concern. |
||
|
vh
|
e15c5ee5ea |
Instruct probe: voice and instruction-following coexist
Trained the same corpus onto Qwen3-4B instruct rather than -Base, with seed, steps and token count held so the carrier is the only variable. The chain picked checkpoint-150 by loss automatically, applying the lesson that cost a re-cut on 4B-Base. The central risk did not materialise. The assistant prior did not block the voice: curly quotes land at 16 of 18, identical to the 4B-Base tuned arm, against 1 of 18 on the unadapted control, and task-leak is 0 of 18 where the base carrier leaked 4. Instruction-following also survived raw-text training -- 10 of 10 on-beat through the chat template, the same as the untuned control. The cost is length discipline rather than comprehension. In-band dropped from 10 of 10 to 6 of 10 and the median went from 124 to 140 words. Training on Victorian prose made it wordier, which is a soft degradation and not a break. Held-out sits at 2.908 against 4B-Base's 2.814, and it plateaus without turning where the base carrier overfit at step 75. The assistant prior competes for capacity, so the instruct carrier absorbs less rather than overfitting more. What raw-continuation training does not fix is the plot furniture. The tuned instruct arm renders the beat and then drags the referent -- "He licked her clean... my master thus, my husband thus", turning the dog into a man, because the corpus is about masters and husbands. Another beat ran to 247 words and gave the narrator a list of duties. That is precisely what instruction-pair training addresses, since pairs teach render-this-and-stop where continuation teaches keep-writing. The probe de-risks the instruction-pair path without substituting for it. One metric note against future misreading: ran_on reports 10 of 10 on both arms and is uninformative on this job, because a single paragraph contains no blank line for it to find. |
||
|
vh
|
d9cf486004 |
Settle the Skaldsong architecture question with a ten-format negative result
Skaldsong's requirement is beat-sentence in, paragraph out, stitchable. That is narrower than free-form continuation and it fails differently, so it was tested directly rather than assumed from the earlier prompt work. The adapted 4B carrier cannot do it. Ten prompt formats across three seeds each -- bare, paragraph-break, labelled, epigraph, one-shot, one-shot unlabelled, three-shot, "it happened in this way", "I remember the whole of it", and the letter prompt's label-and-begin move applied to a beat -- produced thirty samples and not one that reliably renders the beat. They drift, they frame, or they truncate. The cause is structural rather than a shortfall of the adapter: "write a paragraph about this sentence" is an instruction, and a completion model has no mechanism for "about". It continues the text it is handed. Two formats failed worse than drifting and leaked pretraining task data. The paragraph-break format emitted an NLI multiple-choice item, and label-begin emitted a grammar-correction exercise. A standalone sentence followed by a blank line looks exactly like a dataset entry, and style adaptation does not remove base-model task artifacts. The same five beats through the gen seat with a style instruction and no adapter landed all ten samples inside the requested 90 to 140 word band, every one on its beat, with no drift into a following scene. The voice, however, is generic literary pastiche -- abstract-noun-heavy, fond of aphoristic openers, and over-written -- rather than Brontë. Voice without direction against direction without voice. The product needs both, which makes rebuilding the corpus as instruction-response pairs on an instruct carrier the evidence-backed option rather than more prompt cleverness. This applies to Yarros identically, because the carrier question is orthogonal to the author, so the next corpus should not re-run this experiment. Stitching also has its own failure mode, shown in the booth's third panel: the paragraphs were generated independently and the point of view slides between them, so by the fourth beat the narrator is both watching the girl carry the animals and carrying them herself. A real stitcher has to feed prior paragraphs back as context, which means the instruction-pair corpus wants multi-paragraph continuity examples and not only isolated beat-to-paragraph pairs. |
||
|
vh
|
8787daf04f |
Find the prompt recipe that works on a completion carrier
Two operator prompts settled the question the handoff prompt opened. The winner labels the artifact and then begins it: "This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw--". Two of three seeds delivered the actual event in first person, and one is the strongest output of the sweep -- the dog following her to the mill, the kitten caught in a hedge, the dog carrying it out and laying it in her lap before licking it. Coherent arc, correct subject, no gloom-override, no meta-frame. It also reproduced the Gutenberg underscore-italics convention. It works where the handoff failed for a specific reason. The handoff could be satisfied by narrating compliance, because the letter did not yet exist. Naming the artifact and starting it leaves nothing to narrate around, so the only continuation is the artifact. The chapter-heading prompt refuted a hypothesis of mine. I predicted that rendering the title in the corpus's own conventions would make it land harder than the operator's inline form. It did the opposite: both corpus-form seeds ignored the title and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter about the story. Corpus chapter titles are short and decorative, so a long descriptive one in that slot reads as decoration to skip. A label only instructs if the model treats that slot as load-bearing. That prompt also surfaced an unnoticed consequence of the D2/D3 rename pipeline: given "Alice", the adapter produced "Alexander the Alexander, as he was known in Little London". The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Callers cannot reliably name their own characters at prompt time. That is not a defect of the rename, which exists to prevent memorising Brontë's cast, but it is a usability constraint worth stating. |
||
|
vh
|
e47295d2ce |
Re-cut the 4B arms from checkpoint-75 and test an in-fiction instruction
The 4B tuned arms now come from checkpoint-75, the actual loss minimum at 2.813826 confirmed from loss-series.json rather than from reading the training log. Only the tuned arms needed re-cutting; the base arm never touches the adapter. One surprise in the comparison: step-75 and end-of-run differ on typography rather than voice. Curly quotes read 16 of 18 against 17, collapse is 0 either way, but the hard-wrap ratio is 0.33 at step-75 against 0.12 at the end. Further training washes the residual line-break habit out while held-out loss gets worse, so best loss and best typography are different checkpoints. Neither is near the original 0.85 defect and the corpus's own residual from preserved verse is 0.25. The operator's second prompt embeds the instruction inside the fiction: Abernathy tells the tale badly and then asks the narrator to retell it in a few short paragraphs. Across six seeds the model acknowledged the handoff every time and never once performed it. It wrote "I told it, briefly, to his satisfaction", and "So I wrote it out, and kept it in my pocket-book", and one seed negotiated the brief in character, objecting that it could not condense the story into a few short paragraphs because there were too many points to touch. The reason is structural rather than a failure of the adapter. In a novel, "she retold the story" is an ordinary sentence, so the likeliest continuation of a request is narration of compliance rather than compliance. A trailing paragraph break did shift the behaviour -- one seed opened in the narrator's own quoted speech -- so typography is a real lever, just not a sufficient one. That is direct evidence on the instruct question raised earlier: if the product is asking for a scene and getting the scene, no amount of in-fiction framing substitutes for a post-trained instruction-follower, which favours rebuilding the corpus as instruction pairs over more prompt cleverness. |
||
|
vh
|
25eb65e23b |
Complete the R49 carrier sweep: 4B closes the continuity gap and overfits
The three rungs now sit on the same unwrapped corpus, seed, step count and token count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B, 3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but still real. 4B answers the question the rung existed for. Scene-level continuity holds: on the office-kitchen prompt it produces a named character with motivated dialogue, a spatial layout the narrator navigates, and a physical description, all in one passage, where 1.7B wrote pretty but eventless prose about opening doors and looking at stars. On the letter prompt it opens the letter, promises to quote it, and then quotes it across a paragraph break. Voice saturation is also the best of any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0 of 18 against 4 of 18. Two findings that change earlier conclusions. 4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816, 2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both smaller rungs plateaued without turning, so the optimal epoch count shrinks as the carrier grows and my earlier "one epoch is right for this corpus" holds only for the small end. The consequence is operational: the adapter directory holds the end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as a recoverable checkpoint only because save_steps was set. The voice arms were cut from the end-of-run adapter, so the booth understates 4B slightly. The tone-override also appears to close. On the operator's frame prompt asking for a wonderful story, 1.7B held the frame on every seed but killed the animals on two of four; 4B kept them alive on both seeds, and one of them had the narrator doubt the story he was told and supply a parallel childhood memory to explain the doubt. That is a narrator with an interior position on the tale. Two samples per arm, so directionally right rather than established. |
||
|
vh
|
4cee4b179d |
Launch the 4B rung and record that voice transfer carries subject matter
Rung 3 of the sweep is training: Qwen3-4B-Base, one epoch, seed 4919, the same unwrapped corpus, 159 steps at roughly 37.8 s/it for about 100 minutes. Carrier size remains the only variable across the three rungs. A two-arm voice test is chained behind it, gated on the adapter existing, so the rung is readable when it lands rather than needing a second round trip to become informative. The finding worth keeping came from an operator prompt rather than from my test set. Asked for a wonderful story about a stray dog licking a calico kitten, framed as reported speech, the 1.7B adapter held the frame across all four seeds and was unmistakably Victorian throughout -- and two of the four turned the premise into animal death, with the cat licking the puppy to death and the narrator's interlocutor answering that he wished they were all dead. That is not a coherence failure. It is Charlotte Brontë's preoccupations arriving alongside her sentences: Jane Eyre opens on a beaten child, Helen Burns dies, and Villette is saturated with grief. Train 680k words of that into a small carrier and a request for something wholesome reaches for mortality. My nine test prompts were all emotionally neutral and structurally could not have surfaced this. The operator's prompt found it on the first try, which is worth recording as a property of the test set rather than of the adapter. The implication for the regime is that "voice transfer" includes tone and subject, so wanting the voice without the gloom is a corpus-selection or prompt-framing problem rather than a training-length one. Also observed: one seed closed its anecdote and began "CHAPTER XIX. THE CHILD'S CLOSET", having learned book structure unprompted. The base control on the same prompt went modern and essayistic, so the shift is the adapter rather than the carrier. |
||
|
vh
|
9e18fd58ec |
Complete R49 rung 2 and booth the three-way voice comparison
Both rungs now sit on the same unwrapped corpus with seed, steps and token count held, so carrier size is the only difference and the effect is attributable: held-out 3.329 at 0.6B against 3.018 at 1.7B, a gap of 0.311 nats. The chained 0.6B rerun closed the confound the unwrap opened. Two things in those numbers need stating or they will be misread. First, the original wrapped-corpus 0.6B reached 3.172, which looks better than the unwrapped 0.6B's 3.329 and is not. Different corpus means a different held-out set, and the wrapped version's 5.7% newline tokens are near-deterministic after a 70-character line, so they deflate the loss with cheap wins. Removing them removed the easy tokens. It is a measurement artifact, not a regression. Second, a correction to my own earlier claim: I twice described the 0.6B run as still descending and undertrained at 3.172. Its series reads 3.176, 3.173, 3.172, 3.172 -- it flattened. All three runs plateau, so one epoch is about right for this corpus rather than short. The three-way booth puts 1.7B base, 1.7B tuned and 0.6B tuned side by side on the same nine prompts and seeds. The base arm is the control that matters: curly quotes go 0 of 18 on 1.7B base to 15 of 18 on 1.7B tuned, and worksheet-or- explainer collapse goes 3 of 18 to 0 of 18, so the shift is the adapter rather than the larger carrier. Hard-wrapping fell from 0.85 to 0.18, confirming the corpus unwrap carried through into the adapter. Sense partially returned. The 1.7B arm produces locally coherent sequential Victorian prose where the 0.6B produced word salad, but scene-level continuity still breaks mid-passage. One observation held loosely: curly quotes are slightly lower at 1.7B than 0.6B, which would fit a bigger model's stronger priors resisting the adapter at the same rank. That is untested and is not offered as established. |
||
|
vh
|
b8435ceb6f |
Set mog-sec's context to a measured ceiling so it refuses instead of dying
Operator: "loading up the context killed sec again." That reproducer is what finally made the failure legible, and it showed the previous four fixes had all been aimed at the wrong quantity. What the KV pool can hold and what the card can process at depth are different numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were governed by the transient needed to process a prefill chunk against a quarter million tokens of resident KV, which scales with depth and not with pool size. Each change helped and none fixed it. Bisected against the real reproducer, with a non-repeating prompt because prefix caching would let a repeated one hash to cached blocks and never prefill deep: 113,247 prompt tokens SURVIVED (27 s) 200,088 prompt tokens SURVIVED (174 s) ~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted The sustainable ceiling therefore sits between 200k and 285k with gen idle, and gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin under the proven-good depth rather than sitting at the measured edge. The ceiling's purpose is the refusal. Verified after the change: a 149,073-token request serves in 41 s, and requests at both 200k and the ~285k depth that killed the engine now return a clean 400 naming the limit in under a second with the seat untouched. A seat that refuses what it cannot serve is strictly better than one that dies trying. Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was never actually deliverable on a shared card; it had simply not been exercised at depth until today. The probe is committed rather than described, so the ceiling can be re-measured when the card's tenancy changes. |
||
|
vh
|
935b35ac2e |
Unwrap the Brontë corpus and launch the 1.7B rung
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible. |
||
|
vh
|
8842ffe1fe |
Pin mog-sec's KV in bytes at 16.48 GiB and take it to 320k context
Operator: "yes, pin the kv and take it to 320k." The real finding, which took three crashes and two failed attempts to reach: --gpu-memory-utilization does not bound actual usage. It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line showed mog-sec running 0.9 GiB over its 47.48 GiB reservation -- 26.44 consumed plus 3.53 peak activation plus 0.89 CUDAGraph plus 17.52 KV equals 48.38 -- and gen was over by 0.33 on the same card. That overage came out of the shared card's slack, which is what kept OOMing after the utilization drop. The fix is the one vLLM printed itself: --kv-cache-memory=17697765376, its own recommended figure to fit inside the requested budget. Same discipline erp-seat already uses, and for the same stated reason -- an explicit figure is reproducible where a ratio silently yields a different cache depending on what else is resident at start time. The KV pin and the context length are coupled. 16.48 GiB yields about 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM refuses to start rather than crashing later; pinning the KV while keeping 384k was never an available combination. 327,680 leaves 1.15x, up from 1.03x. Verified: the engine now logs "reserved 16.48 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling", KV 375,901 tokens, GPU0 down to 90,561 MiB from 91,313, RestartCount 0, and both sec and sec-reasoning return 200 through the gateway. Also records the BabyBronte eyeball A/B, whose result is the operator's own: the voice transferred and the sense did not. Curly quotes went 1 of 18 to 18 of 18 and worksheet collapse 3 of 18 to 0 of 18 between arms. That voice is separable from coherence at 0.6B is the premise the lightweight-adapter regime rests on, so this is the informative outcome rather than a disappointing one. A corpus-prep defect surfaced with it: the tuned output is hard-wrapped at about 70 characters because the Gutenberg source kept its line breaks and the adapter learned the typography too. Cost: 320k of context instead of 420k, on a seat whose crashes happened at 151k. |
||
|
vh
|
77224619ee |
Give GPU0 real headroom: mog-sec to util 0.50 and 384k context
sec/sec-reasoning crash-bounced twice in ten minutes, not once in thirteen days. My earlier read of "rare, not chronic" came off a RestartCount of 1 and was wrong; the operator pushed back and the second and third failures arrived while that recommendation was still on screen. The memory entry making that call is replaced rather than left standing. Cause is unchanged from the first diagnosis: mog-sec at 0.52 plus gen at 0.38 reserve 0.90 of the card, leaving about 4.6 GiB, and vLLM's utilization figure covers weights and the KV pool but not transient activation memory. A request about 151,700 tokens deep scheduling a further 15,700-token chunk asked for 1.04 GiB with roughly 600 MB free. Dropping utilization alone does not work, and fails in a worse way: a single 420,000-token sequence needs 17.88 GiB of KV, and at 0.50 the pool is 17.4 to 17.5 GiB, so vLLM refuses to start at all and the seat crash-loops during startup instead of during a request. The context length and the crash were directly coupled -- 420k was only reachable at the utilization that left no transient headroom. So both moved: 0.50 and 393,216. 384k rather than vLLM's suggested maximum, deliberately. It estimated 406,352 on one boot and 409,840 on the next, because the available-KV figure drifts about 0.1 GiB boot to boot; pinning the edge value fails to start on an unlucky boot. 393,216 sits 3% under the lower estimate and leaves roughly 0.7 GiB of the pool unspent, which is the transient headroom the change exists to buy. Verified after: KV 405,612 tokens, concurrency 1.03x at 393,216, and both sec and sec-reasoning return 200 through the gateway. num_speculative_tokens is documented as NOT the lever. The crash window logged 17.6% draft acceptance with positions 5 through 7 at 1.5 to 4.9 percent, which reads as an obvious cut from 7 to 3; across 180 samples the median acceptance length is 3.12 of 7 and median draft acceptance is 30.4%, so the crash window sat near the minimum and cutting would cap the workloads accepting nearly the full draft. Cost: 384k of context instead of 420k, an 8.5% reduction on a seat whose crashes were happening at 151k. |
||
|
vh
|
8ab794d89a |
Record the mog-sec GPU0 OOM crash and the spec-decode n=1 near-miss
sec/sec-reasoning on ana-ml2 :8019 took a CUDA OOM on GPU0 at 13:20 PT and auto-recovered in three seconds. First fatal error since the container was created on 2026-08-28. A request already 151,728 tokens deep scheduled a further 15,696-token chunk, needed a 1.04 GB allocation, and found about 600 MB free. EngineCore died, the API server exited 0, and restart: unless-stopped brought it back; the dockerd journal records manualRestart=false, so nothing restarted it by hand. Not caused by the same-day char-rp-fast swap. That work was entirely on GPU1 -- erp-seat is pinned device_ids ["1"] -- and GPU0 read 93,257 MiB both before it started and after it finished. The standing condition is that GPU0's two seats reserve 0.38 plus 0.52 of the card between them, leaving about 4.6 GiB, and vLLM's utilization figure covers weights and the KV pool but not all transient activation memory. Long-context prefill chunks with speculative decoding live in what is left. Also recorded as a specimen: the crash window logged a 17.6% draft acceptance rate with positions 5 through 7 accepting 1.5 to 4.9 percent, which reads as an obvious case for cutting num_speculative_tokens from 7 to 3. Across 180 samples of the same counter the median acceptance length is 3.12 of 7 (range 1.83 to 6.75) and the median draft acceptance is 30.4% (range 11.9 to 82.1). The crash window sat near the minimum. The single window pointed the opposite way from the distribution, and acting on it would have capped the workloads that were accepting nearly the full draft. No configuration changed. One self-healing failure in thirteen days does not justify shrinking a KV pool on a seat configured for 420k context. |
||
|
vh
|
9a916a759f |
Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here. |
||
|
vh
|
1a5bc2ddf1 |
Land the MeroMero v2-31B NVFP4A16 quant and record the pinned-transformers trap
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme. |
||
|
vh
|
b8dbe71a1c |
memory: snapshot — R49 1-epoch pilot complete and awaiting adjudication; MeroMero A4B quantized, v2 blocked; althing 3.6.2 rolled
Ships the two pending code changes alongside the memory that describes them: train_voice_lora.py gains --eval-steps/--save-steps (the 3-epoch pilot overfit with per-epoch eval and save_strategy="no", so the minimum was neither visible nor recoverable), and the althing post-office compose is pinned to 3.6.2. Index rewritten: in-flight compressed from 176 lines to the live state, three detail files added for R49 D1-D3, the MeroMero four-failure chain, and the althing rollout. Seven closed pre-08-27 entries archived; the guards held back the rest because they carry open deferred pointers. |
||
|
vh
|
44c853cd20 | Repair ana-docker database backups and bound CI build caches | ||
|
vh
|
eb75713c1b | Wire Beszel fleet filesystems, GPU telemetry, dashboard and alerts | ||
|
vh
|
20bbb95113 |
fix(r49): loss-series collector silently dropped every eval record
The LossLog callback filtered on `"loss" in logs`. Trainer emits eval under `eval_loss` with no `loss` key, so every eval record was discarded and loss-series.json showed zero eval points while the log a metre away carried `eval_loss: 3.198`. An artefact that omits data which demonstrably exists reads as "no eval was run" rather than "the collector dropped it", which is the failure mode that costs someone a re-run. Collector now accepts either key. Seed 2 gets it from launch; seed 1 is already running with the old code, so recover_eval_series.py parses the eval records back out of its log -- the data was printed, not lost -- and the chain runs that recovery before cutting the generation arms, so the artefact is complete before anything reads it. The two seeds therefore differ in logging code but not in training math: the callback only affects what is recorded, never what is computed, so the weight trajectories remain comparable. Noting it because a difference between the two arms whose spread sets the decision threshold is worth stating even when it is provably inert. |
||
|
vh
|
8fff722f2c |
feat(r49): incumbent arm generated against the concrete gen seat, not the stale H02 name
brokkr-smithy corrected H02's incumbent naming: qwen3.6-35-a3b-heretic was retired from the gateway roster on 2026-08-15 and is not what Skaldsong would call today. Verified against the gateway and the seat itself -- alias `gen` resolves to hosted_vllm/qwen3.8-27b-uncensored on ana-ml2:8015, container vllm-gen, 262,144 ctx. The arm targets that. 24 records, style-prompted on the same prompts and sampler as the other arms. Alias resolved at run start AND end and confirmed stable across the run, per the fleet rule that an artefact records the backing model rather than the alias. Two things recorded rather than glossed: The harness is NOT matched to the other arms and the artefact says so. Base and adapted arms are local transformers on gx10; the incumbent is a served NVFP4 27B reached over the gateway, and it is an instruct model receiving a style instruction where the others are base models receiving none. That asymmetry is the comparison H02 asks for -- prompted imitation against trained voice -- but it must not be reported as if the harnesses were identical. The gateway echoes the ALIAS in each response's `model` field, so a row read on its own would have recorded "gen" as provenance -- the same class of mistake that inflated an exposure count 4.7x on this fleet. Rows now carry alias_echoed_by_gateway beside backing_model_resolved and its date, and the generator was fixed at source rather than only in the emitted file. Sanity: median 392 completion tokens, zero records opening with markdown or meta-commentary, output reads as continuation prose. The style prompt was written to be a fair incumbent rather than a strawman, since this arm is what the adapter must beat. |
||
|
vh
|
375244ad05 |
feat(r49): D2/D3 complete and the H02 pilot is training on gx10
Entity resolution, deterministic rename augmentation, packing and the pilot trainer. Qwen3-0.6B-Base is training now: 507 steps, 11.2 s/it, ~1h35m. D2 -- gender resolution is TITLE-FIRST, and that is a change from F02's method rather than a port of it. F02 used pronoun proximity and recorded that it is structurally blind to the first-person narrator, whose name appears mainly in dialogue surrounded by other people's pronouns. Measured here, proximity called JANE MALE -- the narrator of Jane Eyre and the single worst entity to get wrong. Titles have no such blind spot: Miss Eyre, Mrs. Fairfax, Mr. Rochester, Madame Beck, M. Paul, and a 19th-century novel is saturated with them. Measured: 16 entities resolved, zero wrong, every ambiguous case landing on HELD -- shared family surnames like Helstone and Pelet genuinely belong to both a man and a woman and hold as they should. Held means ungendered, not unrenamed. A HELD entity is still renamed, from the gender-neutral surname pool, because the operator's Yarros directive was "rename all proper nouns" and holding a place leaks it -- Thornfield appears 100 times in Jane Eyre and is as author-specific as Riders Quadrant was. Substituting a neutral token makes no gender claim, so no gender claim can be wrong. D3 -- pool is French + English per the operator, weighted per work by setting: Brussels novels 60% French, Yorkshire novels 25%. Locales restricted to fr_FR/fr_BE/en_GB/en_IE; en_US and en_AU carry modern surnames that are wrong register for the 1840s. The pool is filtered against Brontë's own 75-letter alphabet, so French accents stay and Czech/Latvian marks do not. Two collision defects found by running the leak gate rather than trusting it: `Burns` and `Marie` were drawn as replacements while being Brontë characters -- F02's collision filter was built against Yarros and does not carry -- and then `Pierre-Yves` passed a whole-string filter while `Pierre` (Mademoiselle St. Pierre) is a Villette character. The filter now compares by COMPONENT. Final gate: 0 of 203 source entities survive in any of 24 copy-files. Trainer records what the run RESOLVED to rather than what it requested -- attention implementation, dtype, device, corpus sha and harness cleanliness are read back off the live objects. transformers 5.x has dropped warmup_ratio, caught by reading the signature after the first launch failed on it; the 3% warmup is computed into warmup_steps instead. |
||
|
vh
|
ba8dac2c80 |
feat(r49): D1 corpus built and green — Charlotte Brontë, 680k words, 951k tokens
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
|
||
|
vh
|
6cbc9c7a2c |
fix(memory): kill the recurring "check on the run" cron and record how to find it
CronList showed job 12bdea3c firing hourly at :37 with the verbatim "check on the
run ... inform brokkr when ready" text; CronDelete removed it and the list is now
empty. No crontab entry, user systemd timer, or background shell was re-seeding
it -- the only on-disk hits are conversation transcripts.
This is at least the second kill: the operator killed the same loop on 09-09
("kill the check, no training is running") and that session's handoff warned
against starting one. The warning was insufficient because a cron-fired prompt
arrives as an ordinary user turn with no marker, so it is indistinguishable from
the operator typing it -- three turns went into re-verifying an idle fleet and
answering a timer carefully.
The memory entry now says what to DO rather than what not to start: run CronList
before answering that prompt, and treat a second identical arrival as a signal to
check the cron list rather than answer again. Worth the prominence because the
instruction chain ends in three outward-facing acts -- an althing message to
Miranda, standing up a serving seat, and cueing brokkr -- every one of which
carries something false when no run exists.
|
||
|
vh
|
e08ac702c9 |
chore(althing): herald to 3.6.1 on nh3-dev only, operator-scoped 1 of 7
uv tool install --force --reinstall from ~/development/althing (clean tree, HEAD == origin/master 719bc5f, pyproject declaring 3.6.1). althing-core 3.6.0 -> 3.6.1; herald PID 2457743 -> 3690918 with the old process confirmed gone; version read back from the running install; postbox status reachable/push/alive; journal clean since restart. postbox, althing-listen and althing-route moved on the same install. Six boxes stay queued and the plugin hop (0.1.6) is separate and not done, so this is deliberately not a "fleet done" claim. Reported to forseti per box. Records one verification trap worth keeping: POKE_TEXT in post_office_herald.py is a four-part implicit string concatenation, so a line-oriented grep for the new poke sentence returns empty on a CORRECT 3.6.1 checkout. The first pass came back empty for both the new text and the old, which reads as "new absent, old removed" -- two wrong conclusions from one wrong pattern, detectable only because both cannot be true at once. Check the effective value: reconstruct the concatenation or import the module and print the constant. |
||
|
vh
|
08c852792b |
chore(erp-tune): purge superseded merged models (~499 GB), keep run 6 and the v6 quant
Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.
pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final, merged-run02, merged-run03,
merged-run03-s{025,050,075}, merged-test,
nvfp4a16-test 8 dirs -> 354 GB
gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.
The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.
Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.
The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
|
||
|
vh
|
9f30c12da7 |
docs(r49-prep): close the "is there a newer small base?" question against the registry
The operator asked brokkr-smithy to look for a smaller Qwen3.6/3.8; brokkr reported
none exists and that both 27Bs are the same hybrid kernel path one size up. Verified
independently against the HF API rather than adopted, and it holds -- with two facts
brokkr's note did not carry.
1. No official Qwen3.6 or Qwen3.8 below 27B. The listing is Qwen3.6-{27B, 35B-A3B}
and Qwen3.8-{27B, 2.4T-A95B, Flash-Next}.
2. Neither family publishes a -Base checkpoint AT ALL. Every Base newer than Qwen3
is Qwen3.5. Since the regime requires a non-instruct carrier, that rules the
3.6/3.8 lines out on its own, independent of size.
3. Qwen3.6-27B and Qwen3.8-27B both report model_type qwen3_5, 64 layers as 16 full
+ 48 linear-attention, vision tower present -- confirmed as stated.
So the dense Qwen3 ruling is not a compromise against a better available option; it
is the only dense Base option. Also recorded as insurance, not a proposal:
Qwen3-8B-Base and Qwen3-14B-Base exist, so the sweep can extend past 4B without
changing family if the carrier curve has not flattened there.
|
||
|
vh
|
04950c2881 |
feat(training-probes): re-measure the R49 name pool under the Qwen3 tokenizer
brokkr-smithy flagged that R49 F02's name-pool token splits were measured with the Qwen3.5-2B tokenizer, so the dense-Qwen3 carrier ruling invalidates them. Measured rather than left on their critical path; handed over as input to their re-check, since the dictionary and the adjudication are theirs. The multi-token property strengthens on the chosen carrier: pool multi-token 88.0% -> 90.3%, mean tokens 2.33 -> 2.46. A smaller vocabulary fragments more, so Qwen3's 151,936 splits names into more pieces than Qwen3.5's 248,320. The operator's requirement that names be multi-token, so the drafter reconstructs them from the prefix instead of recalling one embedding, is better served after the ruling. Positive control: the Qwen3.5 column reproduces F02's published figure on the same pool and tokenizer (F02 89% / mean 2.35; here 88.0% / 2.33), so the instrument recovers a known-true value before being asked about an unknown one. The pool is deduped across locales, which reconciles male_given and female_given exactly against the dictionary's own totals block. |
||
|
vh
|
36f1b70a88 |
chore(erp-tune): purge intermediate checkpoints (~74 GB); R49 carrier settled on dense Qwen3
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
|
||
|
vh
|
70f7c0e4a2 |
docs(training-playbook): §3.11 gains a fourth check — hybrid linear-attention carriers without a fused kernel
The base-viability pre-flight had three checks (fits / MoE expert mapping / LoRA support) and would have passed Qwen3.5-0.8B-Base clean while it trained 2.6x slower than a dense model 2.3x its size. Check 4 closes that: read `layer_types` for a linear_attention majority AND probe for mamba_ssm / causal_conv1d / fla / kernels. It is the intersection that is slow -- a hybrid shape with the kernel present is fine, a dense shape does not care. Carries the measured table (gx10 GB10, n=10/arm, spreads 0.6-2.6%), plus the two things a hybrid Base checkpoint brings that a dense one does not: a vision tower and MTP head that target_modules="all-linear" would train on text, and the module rename that AutoModelForCausalLM introduces relative to the vLLM serving class; and unsafe cross-document packing, since SSM state ignores the attention mask. Section heading corrected from "three greps" to "four checks". The example was made runnable and verified on the box rather than shipped untested. |
||
|
vh
|
7db6c44bcd |
feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus the operator's "keep the adapter" ruling made durable. Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq 4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on: Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96). Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect. Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused linear-attention kernel is installed on the box. Grad checkpointing is not the culprit (19%, and saves 2.6x memory). Batching is not the lever for either family -- both sit at this box's roofline at batch 1. Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B 4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune the regime exists to replace, so the carrier family is now an open decision with a recommendation for the dense Qwen3 line -- the design doc's original pin. Two further Qwen3.5 findings, both measured rather than read off the config: the Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that target_modules="all-linear" would train on text) and an MTP head, both dropped for free by loading through AutoModelForCausalLM -- which renames modules relative to the vLLM serving path, so adapter binding needs the sampled-target-changed check on the serving side; and cross-document packing is unsafe because SSM state ignores the attention mask, breaking the per-copy name-consistency invariant the design doc calls sacred. Neither exists on dense. Adapter disposition, per the operator's ruling: all five gx10-resident ERP adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter matching the layout runs 01-03 already used, byte-totals identical both sides and sha256 matching on every adapter_model.safetensors. /tank/* is deliberately excluded from ana-ml2's restic sources, so the profile gains one documented carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run to expand to exactly those eight paths. Nothing is training and nothing is queued. |
||
|
vh
|
92cf09e397 | memory: in-flight rewritten for the close — Pfish-6 on ana-ml2 only, gx10 is a training box with no seat, run 7 purged (~139 GiB), no new run planned | ||
|
vh
|
5ea35d86f4 | memory: run 7 purged (~139 GiB), pfi-gx10 declared an experimental/training box with no serving seat, Pfish-6 on ana-ml2 is the sole standing seat | ||
|
vh
|
625796b894 | memory: snapshot — run 7 retired, Pfish-6 (run-6 NVFP4) is the standing seat on both boxes at 262k ctx; gate failure was a detector bug (adjective 'minor'), fixed cc42d76; ana-ml2 measured 4.1x faster than the GX10; three operator decisions left open | ||
|
vh
|
b8956b58fa |
feat(erp-seat): Pfish-6 replaces run 7 — run-6 NVFP4 quant on both boxes, max-num-seqs 8->32
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is retired (its gate failure was the detector bug fixed in cc42d76, but the run was independently poor). - served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6 - ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact - SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails, 'kernel does not support current device'); Marlin is correct for a weight-only scheme, and vLLM's 'no native FP4' warning is about the scheme not the card - the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s, 3.2x the old ceiling, with single-stream latency unchanged - head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower |
||
|
vh
|
202e7f60e1 | memory: run-7 gate failure was a DETECTOR BUG (adjective 'minor' in a HARD rule) — operator-diagnosed, fixed cc42d76, all service restored, retention wired; the lesson is that rigor downstream of an unexamined premise is not rigor | ||
|
vh
|
a4888f07f8 | memory: point at the run-7 decision brief (kept booth, 5-question ask, answers readable via booth answer) | ||
|
vh
|
31c280c5d2 | memory: run-07 incident CLOSED — alias removed and verified at routing, exposure quantified (2,793 tok into the operator's own Open WebUI session, no peer agents), alias-vs-artifact filter 4.7x, three reporting errors disclosed | ||
|
vh
|
b135adce99 |
docs(ops-lessons): filter on the artifact, not the name pointing at it — measured 4.7x inflation
Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363 rows, counting by the artifact gave 77, because the alias had carried three different models that day. Wrong in the direction that looks careful. |
||
|
vh
|
98da4ec1f1 |
docs(ops-lessons): §11 fourth variant — the instrument read a surface mid-transition
From a near-miss brokkr-smithy-dev caught and declined to send: a post-fix verification read raced a gateway restart and reported the alias still present. A disagreement between two observers is not a finding until the boring explanation is ruled out, and during an incident the pressure runs the other way. |
||
|
vh
|
4b4911e07e |
fix(litellm): remove the trial alias — run 7 failed the safety gate
Alias-present-with-backend-down is a different and worse state than alias-removed: it re-arms silently the moment anything is served on 10.250.50.54:8021 again, and the next thing served there inherits a fleet-reachable name without anyone deciding it should. Flagged by brokkr-smithy-dev; same writer/reader-contract family as ops-lessons §11. Commented rather than deleted so restoring is uncommenting, not reconstructing. Verified: /v1/models returns 33 entries, trial absent, a caller gets a clean 400 'Invalid model name' instead of a connection error to a port that could re-arm. Disposition of the artifact and the alias is the operator's. |