7b5fd91d3cef754dd2f8ffa11772986f183bee55
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
7b5fd91d3c |
docs(gemma4-erp-tune): root-cause the 8.6% MFU — attention on Ampere kernels, 29.9% padding
Run-01 was killed at step 19 by operator instruction to root-cause before spending a ~13.9h window. Two independent methods now agree on where the step time went, and neither was the hypothesis the consult panel converged on. Scaling fit (3 points, 2 params, residuals <3ms over an 8x range): A = 6.87e-4 s/token, B = 8.85e-8 s/token^2 quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384 No fixed term was needed, which refutes launch-bound outright. Profiler kernel table (device rows only): attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 dense GEMM 2,774.0 ms 7.9% other 5,739.0 ms 16.4% The attention kernels are sm80 — Ampere-generation CUTLASS running on an sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS. Correctness cleared separately: the sliding mask asserts at max 1024 allowed/row, so the 25 windowed layers were genuinely windowed. The same probe found that right-padding is what pins the 5 global layers to an explicit 4D mask and off the is_causal fast path — measured at 9.4% slower for 24% less loss work at fixed width. The largest available win is not the attention kernel. The corpus is 29.9% padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall clock, no new dependency, unchanged peak memory. Bucket size turned out not to be a diversity knob — roots per accumulation window are flat across a 256x range, so the global micro-batch shuffle does that work alone and the bucket should be tight. Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic home (sibling to the quantization playbook), the four probes under scripts/training-probes/ with raw output kept for re-derivation, and a §6 to the sizing doc carrying the Gemma-4-specific numbers and round-2 restart parameters. Measured negatives recorded so they are not re-chased: grouped_mm (0.9% slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever), FA4 on sm_120. Round-1 state preserved: 609MB encode cache, order manifest, truncation report, resume script. No checkpoints — it died at step 19 and the first was due at 100, so the lora_B inert-adapter gate never ran and moves to the restart. |
||
|
vh
|
872c2c562f |
memory: the MFU hunt — two hypotheses measured and killed, consult dispatched
Records what has actually been ruled out rather than what is suspected. The hardware is fine: a plain dense GEMM at the same shape reaches 97.1% of the benchmarked 313.8 TFLOPS peak. The Python expert loop is not the cause, which was my hypothesis and I was confident in it. transformers' grouped_mm experts backend runs 0.9% SLOWER than eager with bit-identical output and identical peak memory, and torch 2.13 has the kernel available, so it is not falling back for lack of one. MoE is not the bottleneck at all. Isolated at real shapes the block runs at 26.5% of peak with 36% of its time in pure gather/scatter, and a dispatch-free bmm version would reach 80.9% — but the whole MoE contribution is only about 10% of a step. Making it free buys 7%. So roughly 90% of the time is unaccounted for. The leading untested hypothesis is that the five full_attention layers use global_head_dim 512, above FlashAttention-2's 256 cap, which would push SDPA onto a slow backend for O(n^2) attention at sequence 16384. Also records that the earlier 5% MFU figure was wrong in two ways — unpadded tokens and a guessed peak — and that the operator caught it. Padding is real but secondary at 29.9%. Consult dispatched to brokkr-smithy-dev for the frontier-dwarf panel. |
||
|
vh
|
07743c6aff |
memory: snapshot — tune training unattended, MFU root-caused to a Python expert loop
The in-flight section is rewritten around the run itself rather than the decisions that led to it. The sizing and seat-call bullet collapses to a pointer now that both are executed; its detail lives in docs/pfi/gemma4-erp-tune-sizing.md. Adds the measured MFU finding: 27.1 TFLOPS against a benchmarked 313.8 TFLOPS peak, root-caused by reading the source rather than inferring — transformers runs the Gemma-4 experts in a Python loop, 128 experts across 30 layers, roughly 11,500 iterations per optimizer step under gradient checkpointing. Padding is a secondary 29.9% tax. Records that my first estimate of 5% MFU was wrong in two compounding ways: divided by unpadded tokens, and compared against a guessed peak rather than a measured one. The operator pushed back on the number and was right to. The fused MoE kernel is deferred work with a tracking surface — park id 47 — per the snapshot rule that deferred decisions go in Recent decisions with a pointer, never into the volatile in-flight section. Also records the resume trap: the original launch command begins with rm -rf on the output directory, which would destroy both the encode cache and every checkpoint. resume-run-01.sh exists so that cannot happen. |
||
|
vh
|
d6dfd61c91 |
memory: the ERP tune is running — override granted, 12 defects fixed first
Operator overrode the corpus gate for one run on 2026-08-25, with the grant staged beside the recipe rather than asserted in chat. It deliberately does not flip any root's training_eligible flag, so the signal that made the run stop in the first place survives intact. Records where the run lives, what it is configured with, how to restore the fleet, and the two lessons that generalise past this project. The first is inert gates. Two turned up in one evening — auditcore, whose CSAM hard-drop never fired across 42,662 records, and validate_vision_keys, which compared model.state_dict() against itself and could not fail on any input. Both read as guards. The question that catches them is not whether the check passes but whether it can fail. The second is an invariant enforced on one code path and not its sibling. That was my own bug: INV-T9 requires a window to hold at least one complete assistant turn, and I enforced it where the window is cut but not where it fits, so a trailing user-only remainder became a zero-loss window and killed the first launch. Same shape as the inert gates, in code I wrote an hour earlier. Two further foot-guns worth the space: enable_input_require_grads is mandatory beside gradient checkpointing on a frozen base, or every adapter stays at its initialisation and the run completes successfully having learned nothing; and the upstream Gemma-4 template forward-scans to suppress a closing turn marker before another assistant message, so incremental rendering cannot tile against it and assistant runs must be merged first. |
||
|
vh
|
33433e0d1e |
docs(gemma4-erp-tune): replace the estimates with measurements — they were 3x optimistic
Ran the loss path on the real checkpoint on GPU0 with synthetic tokens.
The arithmetic held for parameter counts and was badly wrong for
activation memory.
naive CE bsz1 seq 8192 81.93 GiB
naive CE bsz1 seq16384 OOM
chunked CE bsz1 seq16384 65.66 GiB
chunked CE bsz2 seq16384 79.71 GiB <- the run config
chunked CE bsz4 seq16384 OOM
The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the
~5 GiB estimated: the estimate modelled gradient checkpointing as
storing layer inputs plus a modest recompute peak, and the real MoE
recompute peak with top-8-of-128 routing and its scatter/gather buffers
is far heavier. Dense-model intuition does not size an MoE run.
Two predictions landed exactly — 205 target modules and 74,342,400
trainable params at r64 — which is why the rest of the model of the
thing is still worth trusting.
The headline is that chunked CE at seq 16384 costs 16 GiB less than
naive CE at seq 8192, so chunking is what makes brokkr's 16384
recommendation reachable rather than an optimisation on top of it.
max_seq_len moves 8192 -> 16384 on his truncation finding: the cap
drops 6.2% of samples but 22.4% of tokens, concentrated entirely in
dialogue, which is 60% of the mix.
Also records the four harness changes this required (eitri-smithy
62b556b), including the inert-adapter trap: without
enable_input_require_grads() alongside gradient checkpointing on a
frozen base, no gradient reaches the adapters, every one stays at its
initialisation, and the run completes successfully having learned
nothing.
|
||
|
vh
|
47ec3d1a97 |
memory: the ERP tune is blocked on a corpus gate only the operator can clear
Every clean-v1 CLEANROOT carries training_eligible: false with two named blockers, and the recipe states plainly that nothing in it is Charter §3 training-eligible. I initially read scoped_grant: operator-2026-08-22 as authorization and told brokkr-smithy-dev I was proceeding. That was wrong, and the person who wrote the field corrected it: the grant governs INV-4 one-way tier inheritance — the adapter is permanently internal-erp-rnd and never distributable — not training clearance. The stage-2 detector is measured-inert rather than merely unvalidated. auditcore v3.7.2 returned its hard-drop exit code zero times across 42,662 raw RP records, its printed verdict ignores its own printed threshold, and it passed a record a blind audit had already identified as sexual content involving a participant the text marks as a child. Verified the one thing that decides whether that specific record reaches training: pippa-5083 is present in kept-manifest.jsonl (4,551 rows) and absent from recipe-dedup-kept.jsonl (20,473 rows), which is the survivor list the harness gates on. The substitute lexical screen caught it. That is one known instance caught by a stopgap and says nothing about what the screen misses. Both brokkr and I recommend stopping. Neither blocker is hours of work. |
||
|
vh
|
c9943b1507 |
docs(gemma4-erp-tune): whole-card placement — gen moves to GPU1, sec stands down
Operator chose a third placement over the two the sizing offered: rather than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no co-tenant and gen never goes dark beyond its own restart. Revised run parameters, since a whole card changes them: - micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving 888 optimizer steps instead of 444. At one epoch the step count is worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert per step against ~512 at micro-batch 1 — a far healthier GEMM on 704-wide experts. - Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock but pushes activations to ~24 GiB per sequence, which forces micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow. - Scriberr stays on GPU1. The previous revision suggested moving it to GPU0, which was correct only while training was going to live on GPU1. Records the ordering constraint in both directions, the elway identity requirement, and that sec's aliases should be allowed to fail at the gateway rather than be substituted with another model. |
||
|
vh
|
9d70100867 |
feat(ana-ml2): elway playbooks to open and close the ERP/RP tune window
Operator call: rather than train beside gen on GPU0, move gen to GPU1 and
stand sec down for the night, so the tune gets a whole 95.60 GiB card and
the fleet's general seat never goes dark beyond its own restart.
Order is load-bearing in both directions and the playbooks enforce it.
gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction
of TOTAL card memory: 42,091 MiB must be FREE at startup or the engine
refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up, so
recreating gen onto GPU1 first would take the main seat down and leave it
down. mog-sec stops first and a hard gate checks the freed memory before
gen is touched. The close playbook mirrors it: gen must vacate GPU1
before mog-sec starts, since mog-sec needs 50,901 MiB of its own.
Close opens with a gate that refuses to run while a process is still
resident on GPU0, so it cannot evict a training run mid-flight.
Override with --var allow_busy_gpu0=true.
Three defects found and fixed while landing this, all worth keeping:
- Verifying GPU residency via `docker inspect --format {{.State.Pid}}`
never matches. vLLM V1 runs EngineCore as a child of the container's
pid 1, and it is the child that holds the memory and that nvidia-smi
reports. Match by cgroup instead.
- A step's `sudo: true` does not extend to its when/creates/changed_when
guards, which run as the login user. The root-only .env made an
unsudo'd grep exit 2, so the GPU-id flip SILENTLY SKIPPED. The
effective-value assert is what caught it.
- That assert originally grepped the config YAML for -\s*'?1'? and failed
against compose's double-quoted `- "1"`. Parse the JSON with jq; an
assert that fails for the wrong reason is worse than no assert.
elway must be invoked as infra-ops@10.250.50.54 rather than the ana-ml2
ssh-target, which resolves to lkraven and has no NOPASSWD sudo.
|
||
|
vh
|
c507db9ac0 |
docs(gemma4-erp-tune): size the run against the checkpoint — QLoRA is structurally unavailable
The proposed shape was QLoRA r64. It cannot be run as specified. The checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj [128,2816,704] — no .weight suffix, so they are parameters, not modules). bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params / 42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit saves ~3.1 GiB of 48.07 and does not error while doing it. Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per additional 8192-token sequence. Two sizing items were absent from the brief and both are load-bearing: - vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping 30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at ~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a 95.6 GiB card — it starts, then OOMs on the first long sample. Fused or chunked linear CE is mandatory and must be smoke-proven before a window is booked, since Liger may not carry a Gemma-4 MoE patch. - v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the full-attention layers). A v_proj target silently produces no adapter there, and k_proj adapts K and V simultaneously. 45.96M trainable at r64 across q/k/v/o. Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr. Recommend standing down sec (2 aliases, last request ~5h ago) rather than gen (7 aliases, 765 busy-engine log lines in 24h). Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at effective batch 16, which makes the wall-clock-checkpointing amendment concrete rather than hypothetical. Package as a uv venv on /tank: root is 91% full (36 GB) with /var/lib/docker on it. |
||
|
vh
|
9d0e628643 |
memory: correct the vLLM version claim — ana-ml2 runs a spread, and 0.27.1 is on disk
The snapshot recorded "ana-ml2 now runs vLLM 0.26.0". That is true of the char-rp seat's pin and false of the box, which the operator caught immediately. Measured per running container: gen is on nightly-311b3513 reporting 0.27.2rc1.dev150, mog-sec on nightly-e9d1398d reporting 0.26.1rc1.dev1102, and rerank-a3 / coder / reward / embed still on 0.24.0. char-rp and the trainee bench stack are pinned to v0.26.0. So there is no single "the version" for this host, and stating one invites exactly the wrong retest. The correction improves the LoRA question rather than complicating it: vllm/vllm-openai:v0.27.1 is already on disk and unused — a TAGGED release, not a nightly, roughly four months past the 0.24.0 where the silent-no-op was diagnosed. That is the right target for a decision test: no nightly variance, no pull. The retest instruction in both the decision entry and the handoff now names it. |
||
|
vh
|
668e590e7d |
memory: snapshot — char-rp on the Gemma-4 MoE, abliterated trainee staged, QLoRA sizing next
Captures an evening that ran from an OOM crash-loop to a measured trainee base. The durable lessons, none of which CLAUDE.md can carry: --gpu-memory-utilization sizes the KV cache and does not cover CUDA context or graphs, which is half of why a seat that fit on the 21st stopped fitting on the 24th; the other half is that gen's footprint GROWS WITH UPTIME (38.5 GiB fresh against 45.6 GiB after three days, same container, same flag), so headroom arithmetic against a long-running gen measures a moving number. The stale-chat-template trap turned out to be endemic across third-party Gemma-4 derivatives rather than a one-off, and it is silent in both directions — wrong prompt when serving, train/serve skew when tuning. And a benchmark finding was retracted because 12% on a five-option task is below the 20% chance floor: a below-chance score indicts the instrument before the model, and a preflight can be thorough while aimed in the wrong direction. Records the serving decision for the tuned model with its history intact: LoRA-on-NVFP4 is preferred if it works, merged weights the expected fallback, but the archived root-cause says the objection was never NVFP4-specific — vLLM 0.24.0's LoRA application was a silent no-op proven quant-agnostic, and ana-ml2 now runs 0.26.0. Retest before designing around merge; the answer changes what Eitri's harness must emit, and he is still early. Auto-archival moved 5 entries (Recent decisions) to archival-memory.md; the guards held back the rest of the 78 age-eligible candidates because their bodies carry open deferred-work language, per the keep-when-unsure rule. Index sits at 286 lines, above the ~250 target and reported rather than forced. |
||
|
vh
|
5415fd4b30 |
docs(gemma4-charrp): abliteration measured in isolation — close to free, but it MOVES capability
Second bench window, operator-authorised after an initial decline and reversal. Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned upstream template, same 192 items, CoT off. Abliteration was the only axis that moved, which is what the previous run could not claim. Net core cost is 0.6 points — but the headline understates what happened. Capability MOVED rather than degraded: five items lost on contradiction detection, four gained on spatial composition, nearly cancelling. A gain was not predicted by anyone, least of all on that axis. The decision this was authorised to settle: llmfan46 stands as the trainee base. No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap between the builds is smaller than the gap this measurement failed to find. Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are ~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than the measurement supports and only marginals were run; and this says nothing about quantization, because the stock-NVFP4 T2 figure came from n=16 against n=32 here — different item counts mean different item sets, so that comparison is n-confounded and is not being made. Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench stack already existed — itself the residue of debugging a 35-restart crash-loop caused by the production compose hardcoding --quantization compressed-tensors. The fix outlasted the incident. gen restored and verified through the gateway; char-rp remains down deliberately; bench stack env reset to the heretic base for the post-tune gate. |
||
|
vh
|
019ccff7e8 |
feat(gemma4-trainee-bench): BF16 bench stack; record that gen's footprint grows with uptime
Adds an ephemeral stack for serving the BF16 trainee base on :8016 under the char-rp aliases, so the abliterated base can be measured on the same battery and the same gateway routes as the served seat with no harness edit. It is a separate stack rather than another variable on gemma4-charrp because that compose hardcodes `--quantization compressed-tensors` for the NVFP4 build. Pointing it at unquantized BF16 weights crash-loops immediately — `TypeError: CompressedTensorsConfig.__init__() missing 3 required positional arguments: 'target_scheme_map', 'ignore', 'quant_format'` — vLLM trying to read a quantization config out of a checkpoint that has none. 35 restarts before it was caught. `restart: "no"` here so a bench seat cannot resurrect itself and block gen's restore, and no homepage labels so it leaves no permanently-offline dashboard card. It cannot coexist with gen and says so: 48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before any KV cache. Running it means gen is stopped. THE MORE USEFUL FINDING is in the meromero env note: gen's memory footprint GROWS WITH UPTIME. Measured today at 46,726 MiB (45.6 GiB) after ~3 days up, and 39,424 MiB (38.5 GiB) immediately after a restart — same container, same --gpu-memory-utilization 0.43, ~7 GiB apart. That is the missing half of this afternoon's crash-loop: the char-rp seat "fit on the 21st and stopped fitting on the 24th" because nothing about char-rp changed and gen crept up underneath it. Headroom arithmetic done against a long-running gen is measuring a moving number, so the note now says to measure against a freshly-restarted one. Operator's requested end state reached and verified through the gateway: gen and summarizer both 200, char-rp down deliberately to hold GPU0 headroom for the upcoming trainee run, bench seat stopped. |
||
|
vh
|
14ff4a3f57 |
docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap
The operator directed that the ERP/RP trainee base be a low-damage abliterated instruct build rather than the stock checkpoint. Two are now staged under /tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB / 25.8B shape with only transformers_version differing in config: gemma4-26b-a4b-it-heretic-bf16 llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100 gemma4-26b-a4b-it-abliterated-bf16 TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset "Low damage" was treated as a measurable claim rather than a description: the field spreads from KL 0.09 to 0.4118 and the table is in the README so the next choice is made on numbers. Fleet anchor for reading them — our own abliteration work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so both staged builds sit inside an already-validated band rather than past it. huihui-ai is rejected despite its reputation: no published metrics, its own card calls the method a crude proof-of-concept, it abliterates both thinking and non-thinking modes, and its parameter count runs ~738M over upstream. The operator's independent read matched. The more durable finding is the chat template. NOT ONE third-party Gemma-4 derivative pulled here ships upstream's — three independent repos carry the identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third 365-line variant, and only the RedHatAI NVFP4 build matches upstream's 6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager slipping, and it is now recorded as a class rather than as the single incident that surfaced it during the A16 control staging. That matters twice over and silently both times: serving a mismatched template renders a different prompt, which is why production pins it; and training through `base/chat_template.jinja` means training on a different prompt format than production serves — train/serve skew with no error, presenting as a tuning failure. brokkr-smithy-dev has been warned on the training side while the harness contract is still early enough to amend. |
||
|
vh
|
8d6a9390de |
docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed
Supersedes what commit
|
||
|
vh
|
3446367d5e |
feat(gemma4-charrp): pin the chat template; A16 control run executed and reverted
The template is now passed explicitly, defaulting to the A4 build's chat_template.jinja. That is a no-op for what is served — the A4 build ships that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once trailing newlines are normalised — and it permanently closes the class of bug found while staging the control: the A16 build ships a stale 266-line template against upstream's 390, with the thinking path built differently and no `thinking` property in its tokenizer_config response_schema. Serving each build with its own template would have moved a second axis. The control ran on the operator's greenlight and has been reverted. Seat is back on the W4A4 build, healthy, RestartCount 0, both aliases verified through the gateway — char-rp returns content with reasoning_content empty, char-rp-reasoning returns both. Result, since it settles a question this repo's config now encodes: activation precision does NOT explain the contradiction-detection deficit. Contradiction detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16 — one item — against gen's 81% on identical items. Every other task is identical across the two builds and the core difference is 2.6 points carried almost entirely by two single items. brokkr-smithy-dev pre-registered that a null result would be the robust branch, because a hidden third axis would tend to create a delta rather than suppress one, so the conclusion survives the residual doubt neither side could close without a dequantization pass. The practical upshot for future scheme choices: W4A4 costs less on this workload than the caution warranted. The caution was still correct to have. Displaced production for 3.7 seconds of measurement plus two container recreates. The A16 build and the BF16 tuning base both stay on disk with the runbook in the stack README, so re-running is a two-minute flip. |
||
|
vh
|
1bd90eaacc |
docs(gemma4-charrp): stack README — the three model dirs, the A16 control runbook
The stack had no README and now carries three model directories that look interchangeable and are not: the BF16 QLoRA base that cannot be served here, the W4A4 quant that is served, and the W4A16 build that exists solely as an activation-axis control. Writing down which is which, and why, before someone "simplifies" the compose to the BF16 path and rediscovers the OOM. Also captures the A16 control procedure end to end, including the two confounds found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit activations, and the stale chat template the real one ships — and the fact that overriding the template is safe because the tokenizers are identical. Both sides have now cross-checked this: brokkr independently diffed every non-quantization config field of both builds against the upstream BF16 and found only transformers_version differing. Residual risk recorded rather than hidden: config identity is not weight identity and nobody has done a dequantization pass. The Gemma-4 flags are documented as architecture-level rather than checkpoint-level, since that is why they survived the seat swap unchanged, and the enable_thinking:false pin is called out as mandatory rather than stylistic — without it every plain prose turn lands in reasoning_content with a null content and every consumer breaks. Notes the non-termination defect with thinking on (32 of 96 calls truncating at 12k tokens, all 16 constraint items among them, reasoning sound right up to the point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not applied to a seat whose production mode is thinking-off. No live change: the seat is still serving the W4A4 build. Displacing it for the control run is an operator decision and is still open. |
||
|
vh
|
24e8826219 |
docs(gemma4-charrp): the A16 control needs a chat-template override, not just a path swap
Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's
battery found a second axis hiding inside what was supposed to be a
single-variable control.
The A16 build ships a STALE chat template. Verified by hash against the upstream
weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI
A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is
not cosmetic — upstream and A4 open the thinking path with
`{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off
it, while the A16 template has no such set and guards with
`enable_thinking is defined and enable_thinking` instead. tokenizer_config.json
corroborates: A4's response_schema carries a `thinking` property, A16's has only
role and content. That build was quantized from an older revision of the
checkpoint.
Served with its own template, the A16 arm would render a different prompt for
identical messages, and a contradiction-detection delta could be attributed to
activation precision when it was the template. That is the same failure class as
the misnamed-A16 repos — a field nobody validated, believed because the name
looked right — one layer further down, and it would have produced a result that
looked like a finding.
Overriding is safe because the tokenizers agree: vocab identical at 262,144
entries, added_tokens identical, so the same template over the same vocab
renders the same token ids. Everything else pre-flights clean — both artifacts
complete with no missing shards, generation_config.json byte-identical.
Seat NOT flipped; displacing production for the bench window is the operator's
call and is still open.
|
||
|
vh
|
f509668e45 |
docs(gemma4-charrp): record the A16 activation-axis control and how to run it
brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction detection with CoT off, against gen's 81%, while state tracking, deterministic constraint following, long-context state and the confabulation control all sat at 100%. That is not general degradation — it is the shape 4-bit input activations produce on the most reasoning-dense task, which is exactly the confound flagged when the W4A4 quant was chosen. They have recorded the finding as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an A16 build to separate the two. No quant run was needed: a genuine A16 build of the same checkpoint already exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 (prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations null, 17 GB). Same weights, same loader as the live seat, one axis moved. ⚠ Two other repos would have answered the question wrongly and the note in the env template says so: bg-digitalservices and ManniX-ITA both publish "Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits 4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's 17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the model. Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom that crash-looped the predecessor this morning. Port and both aliases are unchanged either way, so no consumer config moves. |
||
|
vh
|
27155c0f3b |
feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every gateway route are unchanged, so no consumer sees a difference in addressing: `char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now include chain-of-thought, which makes throughput more critical rather than less — the user waits through the whole reasoning block before the first visible token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7. Both artifacts are on disk and they are NOT interchangeable. The BF16 weights (/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the engine would die at allocation exactly the way the predecessor did this afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB), chosen over the other -it quants because it is compressed-tensors (nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than -it, and the thinking channel lives in the instruction-tuned weights. Smaller weights at the same 0.47 memory budget buy a much larger KV pool: 27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4. The gemma4 tool-call parser, reasoning parser and the enable_thinking:false default all carry over unchanged — they are architecture-level, not checkpoint-level. The --chat-template override does NOT carry over: MeroMero pointed at a jinja hand-patched against that checkpoint, and this model ships its own. Verified that dropping it did not reintroduce the failure that flag existed to prevent — non-thinking prose lands in content with reasoning_content empty, and the thinking alias populates reasoning_content with content carrying the answer. ⚠ Scheme differs from the incumbent and the bench should say so: this quant declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16. Faster, and not like-for-like on the activation axis. meromero-charrp is retained stopped in `created` state and relabelled to AI - Dormant, per the house rollback pattern. Both stacks want :8016, so rolling back means stopping the gemma4 seat first. |
||
|
vh
|
850e0c3351 |
fix(meromero-charrp): drop GPU0 budget to 0.47 — the seat was OOM crash-looping
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the time it was looked at, taking both the `char-rp` and `char-rp-reasoning` gateway aliases down with it (they resolve to the same seat on :8016 — hosted_vllm/char-rp and hosted_vllm/char-rp-thinking). Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the engine gets through weights, torch.compile and CUDA-graph capture looking entirely healthy, then dies at KV-cache allocation with `torch.OutOfMemoryError: ... 195.19 MiB is free`. GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV cache and does not account for CUDA context, graphs and non-torch overhead. This seat was at 0.51, so the pair was committed to 0.94 of the card with about 0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today. 0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 -> 23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the pool still holds 1.4x a full-length sequence. What is lost is concurrent long requests, not context. Verified through the gateway rather than at the container: char-rp returns 200 with content, char-rp-reasoning returns 200 with both content and reasoning_content populated. Seat is healthy with RestartCount 0. The arithmetic and the "check used_memory, not the flag" warning are written into the env template, because the next person to raise either budget needs to lower the other in the same change. |
||
|
vh
|
35adc4a043 |
feat(homepage): rebuild on Australis Skyfall — dual theme, light mode shipped
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.
The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.
build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.
⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.
So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.
Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.
Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
|
||
|
vh
|
39da1d4a97 |
feat(homepage): recategorise on "do I open this?", collapse the API groups
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
|
||
|
vh
|
f6f2f69649 |
fix(homepage): uniform 4-column grid, hold the status gutter, unleak AI Systems
The board's card width changed at every group boundary because `columns:` was being tuned per group under the 2026-08-18 "columns = member count" rule. That rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that group's CARD WIDTH, not its density. Measured on the live board, Notes rendered a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything else 360px. All 20 groups are now `columns: 4` and every card renders at 360. `.service-name` reserved a 78px status gutter with `padding-right` and relied on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the gutter was spill room the title printed straight through. Six cards on the AI tab rendered their name underneath their own status pill, measured by testing the title text node's box against the status cluster's. The intended ellipsis never fired either: it is painted by whichever block's own line overflows, and that is the anonymous box around the bare title text node, which does not carry `overflow`. The gutter holds by wrapping now, and the description opts back out of it with a negative margin since the pill only ever covers the first line. Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox in 2026-08-18, arriving this time from a container label. Relabelled to `AI - Audio Tools`, where the other ASR seats already live. The `homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of the name and now carries the constraint, plus the fact that a labelled container is discovered from any of the five configured engines and must not also be listed in services.yaml. Also: descriptions clamped to three lines so a five-line outlier stops dragging its row 50px taller than its neighbours; icon ramp overridden off slate-400 -> slate-700, which was sinking the bottom half of every glyph into the card fill; bookmark groups and Jellyfin's trailing stream rows brought into the card and eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates them. The icon override sits on `html[class]`, not `:root`: Homepage sets the same variables on `.theme-slate`, which is on the <html> element, and a class beats `:root` on the same element. Verified with Playwright against the live board — per-group card width, card height spread, and a geometric title-vs-status collision check, before and after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/ |
||
|
vh
|
32349b7653 |
memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed
Rewrites the in-flight section, which had gone stale in one place (speaches is stopped, not live) and did not reflect a session in which everything opened was also closed. Three detail files carry the bulk: the FortiGate's public surface reduced to nothing including the ACME listener, Scriberr's deployment and the three upstream defects it required working around, and the ESH DNS repair that also made the IPv6 naming scheme real on three hosts. The tunnel entry in the decisions log described a cipher change as proposed and pending; it has since been attempted, found impossible in the form intended, and resolved differently, so the line now points at the outcome rather than the intention. New decisions cover the cipher adoption and the finding that the per-flow ceiling belongs to the UniFi gateway's software cipher rather than the firewall, the administrative closure, the transcription deployment, the DNS repair, the retirement of an ASR service whose only consumer was abandoned, and a database VM that until now had no fleet identity and no vaulted credential at all. Three approaches are recorded as abandoned: an AEAD cipher the far end cannot express, an upstream DNS setting the gateway accepts and ignores, and a scheme to claim unused delegated prefixes that founders not on the prefixes but on having to rebuild a firewall policy to use them. |
||
|
vh
|
d419b11d43 |
docs(ipv6): close the NH3 multi-prefix question by operator ruling
The gateway has no IP-passthrough mode, confirmed by the operator with admin access to it, so the inexpensive path of letting the UDM take the delegation whole and carve it natively is unavailable at this site. What remains is a separate DHCPv6 client presenting several identities, which needs recabling to reach the gateway, splits routing so that one device handles v4 and another v6, and above all moves IPv6 off the UDM's zone firewall, leaving the entire policy to be rebuilt elsewhere before any host could safely hold a globally routable address. The operator has declined it and the LANs stay without v6. The index entry also still asserted the superseded single-delegation conclusion as fact; it now carries the corrected reading alongside the ruling, so the finding and the decision not to act on it arrive together rather than the first inviting a retry of the second. |
||
|
vh
|
bd209951ac |
docs(ipv6): correct the NH3 delegation finding — eight /64s, not one
The note concluded AT&T delegates a single /64 and that reading the address pattern as a /60 was a mistake. A later finding in the same session overturned that by reading the BGW's own statistics page, and that correction was never written down; it survived only in the transcript and surfaced again while assessing whether more prefixes could be claimed. The gateway holds the /60 and rations it, keeping half for itself and delegating the top eight /64s one at a time. Both observations agree: a prefix-id only carves within a delegation already held, so the earlier test could not have moved a lone /64 regardless. The limit is that UniFi solicits once, which makes the requester the ceiling rather than the carrier. |
||
|
vh
|
d127e29fac |
docs(ipv6): retire the next-candidates line now that all three are done
Replaces it with why the remaining segments have no eligible hosts: two are appliance-only and three have no IPv6 enabled yet, pending the firewall-policy pass that SLAAC on a client segment would require. |
||
|
vh
|
4e83395ddf |
docs(ipv6): all three esh-server hosts now carry the segment name
esh-pve-nas and esh-vm-db join esh-docker-vm on 4411:b105, at :50:55 and :50:60 respectively, each applied by the same prefix-deriving if-up.d hook so the last two groups read straight off the IPv4 address. Two obstacles are recorded because both will recur. The Proxmox node had link-local only despite every relevant sysctl appearing correct, because its bridge carries per-interface forwarding and the kernel ignores router advertisements on a forwarding interface unless accept_ra is explicitly two rather than one. The fix takes the advertised prefix while declining the default route, so the hypervisor gains an address without any change to how it routes; this was verified after applying, with the v6 default route count still at zero. The database VM refuses key authentication for the privileged accounts and its unprivileged login cannot escalate without a password, so the hook went in through the QEMU guest agent from the hypervisor, which executes as root inside the guest. The document notes the base64 indirection needed to get a multi-line script through intact. |
||
|
vh
|
ffb7fba346 |
docs: give the ESH IPv6 naming scheme a home, and make it real on one host
The scheme has existed since August as a single line of persistent memory, which a snapshot then deleted. It is a naming convention rather than temporal state, so it now lives in docs/pfi as a proper document, and the memory entry is reduced to a pointer at it. The document carries the full table, the address structure, the reasoning about which slots can and cannot hold a name, and the recipe for applying one to a host. It also corrects the conclusion the original note ended on. That note held that these names could never appear on the wire, which is true of everything UniFi is able to assign but not of what a host can assign to itself, and the distinction is the whole difference between a joke and an address. AdGuard on esh-docker-vm now holds the esh-server name, at 2607:73c0:402:1d02:4411:b105:50:45, where the segment identity and the IPv4 address are both legible. It is applied by an if-up.d hook that derives the prefix at runtime rather than hardcoding it, backgrounds itself with a retry so it cannot stall interface bring-up, and adds nothing to the existing interface configuration. This is load-bearing rather than decorative. The gateway advertises an IPv6 resolver to clients, macOS prefers it over the IPv4 one, and it previously pointed at an address derived from that host's MAC. |
||
|
vh
|
41091eef8f |
docs: restore the ESH IPv6 naming scheme, dropped by a snapshot
The six ESH LAN hexspeak names were recorded in |
||
|
vh
|
6217d3993e |
fix(scriberr): force uv to copy rather than reflink when building envs
Scriberr builds each model backend's Python environment with uv at container start. uv's default link mode reflinks or hardlinks out of its cache, which fails on this overlayfs over ZFS combination and reports it as a failure to clone a metadata file with errno 11, resource temporarily unavailable. The wording points nowhere near the cause. The damage was partial and therefore easy to miss: WhisperX and PyAnnote came up fine and the application served normally, while the Parakeet and Sortformer backends were quietly absent. Setting UV_LINK_MODE to copy trades a little disk and build time for environments that actually materialise, and the occurrence count for that error is zero on the following start. The knob is exposed through the environment so it can be moved back to the default if a future host does support reflinks. |
||
|
vh
|
efddb4e511 |
feat(scriberr): stand up transcription on ana-ml2, pinned to GPU1
Scriberr transcribes audio and video locally with WhisperX and speaker diarization, and it lands on ana-ml2 rather than ana-docker because the work is GPU-shaped: ana-docker offers eight cores already shared with fifty containers and thirty-seven gigabytes of disk, against ninety-six cores, terabytes on /tank and idle capacity on GPU1. The reservation names device 1 explicitly, since GPU0 is fully committed to the gen seat, and the container is confirmed to see that card alone. The image is built from source, which is not a preference. These are Blackwell cards at sm_120; the published CUDA image covers Pascal through Ada only, and the blackwell image the upstream README documents has never been published at all. The path upstream actually ships for sm_120 is Dockerfile.cuda.12.9, carrying CUDA 12.9 and cu128 torch, so that is what gets built. The compose header says so, because the obvious cleanup is to swap in the published image and that would silently drop the deployment to CPU. Two configuration details are load-bearing and documented where someone would go to change them. The application runs as uid 10001 rather than the usual 1000: that Dockerfile moves its user aside for Ubuntu 24.04's own uid-1000 account and chowns /app accordingly, while the entrypoint's remapping covers only the data directories, so at 1000 the process cannot open its database and restarts forever behind a SQLite error that reads as though the machine were out of memory. Secure cookies stay off while the service is reached over plain HTTP, or sessions are dropped by the browser and login appears to loop for no visible reason. Storage is bind-mounted onto /tank because model weights run to several gigabytes and the root pool on that host is nearly full. Also adds the scriberr service alias to internal DNS, following the existing alias convention so consumers name the service rather than the box. |
||
|
vh
|
22ae9cd480 |
ops(ana-gw): disable ACME and retract the all-port VIP claim
The ACME client is unbound from wan1, which removes the HTTP-01 challenge listener that had been holding port 80 open irrespective of allowaccess. An external sweep of fifty-five ports against the WAN address now finds nothing open at all, while the internal interface still serves the GUI and SSH and retains its certificate, which remains valid until late October and simply stops renewing. The previous note's claim that four virtual IPs were unrestricted all-port static NATs is withdrawn. A FortiOS virtual IP can be scoped either by an explicit forwarded port or by a service binding on the object, and the earlier parse inspected only the former. All four use the latter, and the custom services behind them are narrow. None of the fourteen is unrestricted. Ground truth from outside is recorded in place of further config reading, listing what each public address actually exposes. Three configured mappings answer nothing at all and are noted as tidy-up candidates for the migration rather than as exposure. |
||
|
vh
|
f39b66d2e1 |
ops(ana-gw): port 80 on the WAN address is the FortiOS ACME listener
The previous note attributed the open port to an ISP transparent proxy. That was wrong. It terminates on the FortiGate: system acme is bound to wan1, and FortiOS opens port 80 there to answer HTTP-01 challenges regardless of what allowaccess permits, which is why the port stays open with the interface set to ping only. Every non-challenge request returns a fixed 403 whose body reads ACME Access Only. No DNAT is involved; of the fourteen virtual IPs only two land on that address, neither on port 80. The wrong conclusion came from a sniffer filtered on dst host, which matches inbound packets alone and so excluded the replies being looked for. Filtered bidirectionally the box is plainly seen emitting SYN-ACK on port 80. The note records the rule. Two consequences follow. The earlier warning that certificate renewal would fail without http in allowaccess is retracted, since FortiOS opens the challenge port itself. And the listener is not an administrative surface, though its value is now marginal with WAN administration closed, so the note records how to remove it and leaves that decision open. Also captured: four virtual IPs are all-port static NAT and map every port of their external address. |
||
|
vh
|
7bc9754e40 |
ops: adopt AES-128 on both Anaheim tunnels and close public admin
Both tunnels now negotiate AES-128 for ESP, applied make-before-break so neither dropped waiting on its far end: the FortiGate was widened to accept the new cipher alongside the old one first, then each UniFi gateway was flipped. Single-stream throughput moves from 245 to 270 on the NH3 tunnel and from 268 to 304 on the ESH tunnel. Both network objects were diffed against pre-change snapshots and the only field that moved on either is the ESP cipher. The proposal lists are left accepting AES-256 as well. The peers offer only AES-128 so the extra entries are inert, and retaining them means a gateway reverting cannot strand a tunnel. With that up, the WAN administrative surfaces are closed. The interface is back to permitting only ping, and the infra-ops account is again restricted to RFC1918 space. Ports 443 and 22 were confirmed closed from two separate sites and management over the tunnel still works. The close was issued over the tunnel rather than over the WAN, since withdrawing SSH from the interface while connected through it would sever the session mid-command. The box now has no out-of-band path, which the memory records explicitly. Also captured: the two UniFi vault items have different shapes, one a bare key and one a documentation note requiring extraction, which produces an opaque nginx rejection if missed, and the ESH key's first confirmed write. |
||
|
vh
|
6edebe4864 |
ops: forwarding through a downstream WireGuard terminator is free
FortiOS has no WireGuard, so any WireGuard site-to-site has to terminate behind the edge, and ana-wg already fills that role. The earlier 767 Mbit/s figure was taken with traffic terminating on that box, which left the forwarding case unmeasured and overstated what a real deployment would see. Measured properly, transit from NH3 through the tunnel and onward to ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight, so the forwarding hop costs nothing and the full threefold gain over IPsec survives. The container uses around a fifth of four cores at that rate, against the UniFi gateway spending a third of its own four to move a third as much. The note records what still needs deciding rather than presenting this as ready to build: Anaheim hosts have to route to the terminator directly or pay a hairpin through the edge, which was not measured; the terminator introduces a failure mode the edge did not have, with a fallback route as the mitigation; and the NH3 end needs its own terminator, where a Linux host matches the measured figure while the existing UniFi WireGuard server would likely land lower. The test tunnel, its peer and the temporary route were all removed. |
||
|
vh
|
062215e81a |
ops: the tunnel's per-flow limit is the UDM's software AES-CBC
Varying only the cipher settles what inference could not. AES-128 has ten rounds against AES-256's fourteen, so a software-bound path must speed up when the cipher gets cheaper while a hardware-offloaded one will not move. Run as A/B/A on a single stream, AES-256 returns 232, 245 and 243 Mbit/s and AES-128 returns 282 and 275, a gain of about a fifth at identical CPU. The FortiGate's offload engine is therefore not the constraint, and the operator's reading of the UniFi side as software crypto without acceleration is correct. Two earlier claims of mine were wrong and are corrected in place. The observation that the UDM sat at seven percent CPU was a sampling artifact, since UniFi refreshes that statistic on the device report interval and the windows used were four seconds; a sustained run reads thirty-five percent, around one and a half of four cores. The per-session offload hypothesis is refuted rather than merely unproven, because it predicts no response to a cipher change. The remaining figures follow from this. One stream is one core, eight streams engage about three, and aggregate stops responding to cipher choice once several cores are working because the path bounds it. CBC chaining is what makes this expensive, as it serialises blocks and prevents the AES instructions from pipelining, which is also why the same gateway manages far more over WireGuard. The cipher was restored and the network object verified unchanged against its pre-test snapshot. |
||
|
vh
|
8ecffa1fab |
ops: WireGuard over the same path carries one stream at 767 Mbit/s
The operator asked for a WireGuard test between ana-wg and NH3 across the public internet, which is the arrangement that separates the path from the crypto since neither gateway performs encryption in it. A single stream returns 767 Mbit/s against 245 for IPsec over the same wire, and eight streams return 763, so one flow already saturates the path and there is no per-flow penalty whatsoever. That settles several things at once. The limit is not the circuit, the NH3 uplink, the ISP or the physical path, all of which carry 767 on one flow. Ranking the implementations, Linux WireGuard shows no per-flow penalty, UniFi's WireGuard shows roughly 1.4x, and IPsec on this pair shows 2.8x. Latency under a single bulk stream tells the same story, rising to 12.7 ms on WireGuard against 102 ms on IPsec. This overturns the earlier conclusion that changing transport was not worth pursuing, which compared eight-stream figures and so understated the gap for single-stream work by a factor of three. A WireGuard site-to-site terminated on ana-wg now looks worth considering, and the note records the open questions around failover and policy. Attribution between the FortiGate and the UniFi gateway remains unresolved, since both perform IPsec in the slow measurements, but the remedy does not depend on which one is responsible. The test tunnel was removed, ana-wg is back to its original three peers and the generated keys were shredded. |
||
|
vh
|
d42e9d8712 |
ops(ana-gw): establish the per-flow cap is IPsec-specific, not capacity
The operator asked whether the 80F is simply out of capacity. It is not. Routing a single flow between two Anaheim VLANs through the same box, with no tunnel involved, sustains 940 Mbit/s, which saturates the link, and eight flows over that path return the same figure. There is therefore no per-session ceiling in the plain forwarding path and the roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone. Measuring the second tunnel with the same probe supports this. The ESH side, terminating on different gateway hardware at nearly half the round-trip time, returns 268 Mbit/s on one stream against NH3's 245, and 715 against 692 across eight. A window-limited path would have been substantially faster at the shorter round-trip, so the agreement between two dissimilar far ends is further evidence of a rate cap. This also corrects the previous note, which named the FortiGate on reasoning that could not separate the box from the protocol, since every slow path was both. That separation is still not established: both tunnels terminate on UniFi gateways running common firmware, so the cap could belong to either side. The note records the tunnel topology that would decide it and flags the bearing on the pending replacement. |
||
|
vh
|
cf0cb2cbb3 |
ops(ana-gw): pin down what limits a single stream across the tunnel
The earlier note described the constraint as per-flow serialisation without evidence. Pinning SO_SNDBUF and sweeping it shows what it actually is: throughput holds flat between 224 and 247 Mbit/s across a thirteenfold range of in-flight data while round-trip time scales with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions are absent throughout. That is a fixed service rate with a standing queue ahead of it, so socket buffer, window scaling and congestion control are not the lever and should not be pursued. The same measurements surface something with wider consequences than throughput: a single bulk stream lifts tunnel latency from 6.9 ms to 102 ms average, 136 ms peak, with no loss. Interactive traffic sharing the Anaheim link degrades sharply whenever anything moves bulk data, and parallelising transfers makes that worse rather than better. Localisation rests on the FortiGate appearing in both slow paths and in neither fast one, with aggregate throughput over the same security association reaching 692 Mbit/s. Per-session NPU offload fits the shape but was not confirmed on the device; the note records the test that would settle it and flags the bearing this has on the pending FortiGate replacement. |
||
|
vh
|
e41d19f1cb |
ops(ana-gw): close out the Anaheim tunnel cipher question
The AES-GCM cutover was authorised and attempted, NH3 side first. It cannot be completed: UniFi's manual site-to-site IPsec implements no AEAD cipher. Eight GCM spellings were rejected with api.err.InvalidPayload while an otherwise identical body carrying aes256 returned rc:ok, which isolates the enum rather than the request as the cause. The accepted set is aes128/aes192/aes256/3des. Both Anaheim tunnels terminate on UniFi gateways, so this blocks the ESH tunnel on the same grounds. Measuring while testing also retires the premise. NH3's uplink is a 1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather than the ~550 previously recorded from a 4-stream run. Compared with WireGuard over the same gateway and uplink, the deficit narrows from 2.3x at one stream to 15% at eight, so moving this link onto a different transport is not worth doing. The constraint is per-stream, around 245 Mbit/s, with both endpoints idle under load. Parallelising bulk transfers remains the mitigation and is worth 2.8x at no cost; NFS nconnect is the equivalent lever for single-stream mounts such as /mnt/smithy on ana-ml2. FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list, which is inert while the peer offers only CBC and avoids a further renegotiation. The UDM network object was diffed field-by-field against its pre-change snapshot and is unchanged. |
||
|
vh
|
5af362e9d0 |
ops(ana-gw): restore WAN admin access ahead of the FortiGate cutover
Re-open the ana-gw admin GUI on wan1 so the Anaheim edge can be managed remotely if the cutover goes wrong, reversing part of the 2026-08-12 lockdown. Two config changes, nothing else (verified by diffing pre/post `show full-configuration`): - wan1 `allowaccess ping https` — https only; http, ssh, and fgfm stay off, and wan2 is untouched. - `infra-ops` trusthost widened to all routable IPv4; the `admin` account stays locked to 10.0.0.0/8 so the guessable username remains unreachable from the internet. Verified end-to-end from two sites: a real `/logincheck` POST returns AUTH OK over the public path, on a browser-trusted Let's Encrypt cert for ana-fw.phasefinal.com valid through 2026-10-27. Two FortiOS behaviours worth recording, both of which cost time here: a trusthost whose base address is 0.0.0.0 is silently treated as unset (so there is no writable "any" — only decomposed ranges), and trusthost is enforced before the TCP handshake, so a blocked source sees a filtered port rather than a refused login. Follow-ons captured in memory, not actioned: ACME renewal for the admin cert needs port 80 on wan1 (next attempt ~2026-09-27), and a ~5 SYN/s source in 179.51.184.0/21 now draws SYN-ACKs at no measurable CPU cost. |
||
|
vh
|
b6340519bc |
memory: snapshot — Anaheim tunnel at 25% of circuit; selene retired; hrafn CI fixed
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired. |
||
|
vh
|
0ad332bb4a |
feat(booth): per-row link removal + render the link board as real UI
The standing link board is the one MULTI-WRITER booth -- every agent session appends operator-facing URLs to it. "Delete the folder" was the only granularity available, so removing one dead link meant hand-editing markdown. It is 32 rows and only grows. booth links row number, entry id, raw row booth unlink 3 by row number booth unlink 8b40e0a5 by entry id (what the UI's x posts) POST /b/<name>/unlink form field `entry` = content id ROWS ARE ADDRESSED BY CONTENT ID, NEVER BY POSITION. The board is append-only and multi-writer: another session can post between listing it and clicking x, and an index would then delete a neighbour. An id either matches the row you saw or matches nothing. A row number typed at the CLI is resolved to its id BEFORE anything is deleted. Appends and prunes now take the same flock on .links.lock, so a post cannot be lost inside a prune's read-modify-write. UI: a booth carrying links.md renders as rows -- description, URL, provenance, copy button, per-row x -- instead of a markdown blob. links.md is filtered out of the gallery so it does not appear twice; the header counts LINKS not files; the empty-state and the one-click "Wipe now" both stand down for a board (same rule as the kept lane: nothing durable is one click from gone). booth/links.py extracted, STDLIB ONLY. The CLI needs this logic and must not require the service venv -- importing app.py drags in FastAPI, so deleting a line from a text file would have needed a web framework installed. THREE BUGS FOUND BY TESTING, all in the shell wrapper while the module was correct throughout -- module-only tests would have caught none of them: - `[ "$n" -eq 0 ] && echo ...` as the LAST statement made `booth links` exit 1 whenever the board had rows. `unlink`'s index lookup calls it inside $( ) under `set -e`, so a successful listing killed the caller and the removal silently did nothing while reporting success. - ids are 8 hex chars and roughly one in forty is ALL DIGITS; those were read as row numbers, resolved to nothing, and removed nothing. Now disambiguated by the id's actual shape, not by "is it numeric". - filtering links.md out of the gallery left `items` empty, so a full board rendered "This booth is empty" and an empty <div class="gallery"> under 32 visible rows. 87 tests (was 76): parser tolerance of hand-written prose, content-id stability across concurrent appends, removal precision, UI branch behaviour for board/normal/empty booths, and subprocess CLI tests pinning the two shell bugs. Deployed to nh3-dev and verified against the live 32-row board read-only; board file byte-identical afterwards. |
||
|
vh
|
4be880f36c |
feat(booth): kept boards can be deleted from the UI; document the TTL-reset trap
Kept boards had no delete path in the UI at all. The kept lane deliberately omits the wipe control -- a one-click wipe next to the durable stuff is a footgun -- but "deliberate" had been implemented as "impossible": the only routes out were ssh or a hand-written API call. Now it is two deliberate acts. A `release` control on kept cards drops the sentinel, the board moves to the ephemeral lane, and the existing x wipes it from there. Release is reversible -- POST /b/<name>/keep pins it again. POST /b/<name>/unkeep release the pin POST /b/<name>/keep pin it (round-trip, so release is not a one-way door) FOUND WHILE TESTING, and it invalidates the previously-documented workaround: removing the sentinel BUMPS the booth directory's mtime, and booth age is the newest mtime in the tree -- so a released board's clock RESETS from 10,000s to 0s and it survives another full TTL. The old comment said "remove the sentinel first (it rejoins the sweep)", which is true but means the board lives another 24h, not that it gets reaped. Unkeep-and-wait is a delay, not a delete. test_releasing_a_board_RESETS_its_ttl_clock pins that behaviour deliberately so nobody re-derives the workaround. Release is what unlocks the x; the x is what deletes. CLI: `booth rm` already worked on kept boards but said nothing about it. It now announces "(was KEPT -- durable board)" so wiping something durable can never look identical to wiping run output. Not a block -- a CLI user naming a booth is being explicit. 5 new tests (67 pass). Verified live on nh3-dev: release renders on all four kept boards, the ephemeral lane keeps its x, and the links board is untouched with its sentinel intact. |
||
|
vh
|
b8a535507a |
feat(judge-bench): keep the judge harness; warn about the 7-way alias collision
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced. |
||
|
vh
|
ca3c984f93 |
feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design
Benchmarked selene against gen on selene's own job: 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a neutral JSON prompt and Selene's native Atla template. 288 calls, all free local. neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) gen won on both templates and selene's BEST sat below gen's WORST. Selene was given its own fine-tuned template as a fairness check; it gained one point, not the three it needed. Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters most. gen returned tie correctly on the JSON template. Selene also compressed the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only win was ~3x latency, unexercised at ~60 calls/day with zero queueing. TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately: - chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says consumers bind the capability, not a concrete model. Sampler profile copied from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served config matches the benchmarked condition. - selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A served-name is a contract about what the model IS; answering it with a different model hides a material change behind a stable string. Operator ruling: "never repoint a named model at a different model's endpoint -- that is intentionally misleading." Verified: the gateway now returns HTTP 400 "Invalid model name" for it. Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role moved onto an existing seat rather than allocating anything new. Canonical litellm config synced from the host; ana-ml2 README and recommended-model-settings updated. compose.yaml kept for reference, not deployed. |
||
|
vh
|
a896c0a5a9 |
feat(playbooks): sweep world-readable secret-bearing .env files to 0600
Eight stacks on ana-docker kept secret-bearing compose .env files at mode
0644 -- readable by every local account on a box with four interactive
users. Verified as a real exposure, not a theoretical one, by reading one
as `nobody`. Six other stacks already used 0600, so this converges on the
existing house pattern rather than inventing one.
Swept (all verified mode 600, unreadable by nobody, compose config still
rendering as the deploy user, container untouched):
vaultwarden traefik beszel gitea-runner
miniflux news-digest searxng vor
Post-sweep the host has zero secret-bearing .env readable by nobody.
Checked before writing the playbook, and re-checked by the playbook itself
so a bad case cannot be swept in by accident:
- every target is owned by lkraven, the deploy user, so 0600 preserves the
deploy path (asserted by rendering compose config AS lkraven, not root --
checking the mode proves the bits changed, only rendering as the deploy
user proves the next deploy still resolves its variables)
- none is bind-mounted into a container. vaultwarden looked like it was;
that `- .env` is under `env_file:`, not `volumes:`. A genuinely
bind-mounted .env is read by the container UID and 0600 could break it,
so the playbook refuses that case.
Two gotchas recorded in the file:
- elway's variable regex eats any bare identifier in braces, so a docker
--format Go template containing {{end}} or {{println}} fails as an
undefined variable. Grep the raw inspect JSON instead.
- the `ana-docker` ssh alias resolves to lkraven, who needs a sudo
password; run elway against infra-ops@10.250.50.70 for NOPASSWD.
|
||
|
vh
|
9642952a54 |
docs(hrafn-ci): mirror the now-canonical vh/hrafn pipeline; record two failed runs
claude-bot holds write on vh/hrafn as of 2026-08-23, so the canonical copy of the pipeline moved there and infra-ops maintains it directly instead of routing patches through the repo holder. The files here are a verified mirror (byte-identical to live at 22e0eb9d75a6). Live state: run 9922 green, and both new content assertions executed rather than merely existing -- verify 4/5 host tree matches shipped context (a99ce748a0c9...) verify 5/5 image source matches host source (06f209fd0641...) The CI-computed context hash matching on the host is the end-to-end proof that the converge lands what CI ships. Its absence is what let the frozen-source bug survive every green deploy. Also records why the HEAD == GITHUB_SHA assertion was added and then removed: it needed the git binary (run 9920, exit 127), and installing git flipped actions/checkout@v4 from its node implementation to the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout code path and broke a working pipeline; it guarded a hypothesis that proved wrong, so it went rather than getting ca-certificates bolted on. |
||
|
vh
|
b38c369313 |
fix(hrafn-ci): staging dir inside the rsync target froze host source silently
Root cause of nevermore-claude's report that v1.0.0 deployed green while the host kept serving 0.1.0. The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from the destination (absent from the source listing) DURING the transfer, destroying the source mid-copy. Reproduced exactly: before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE Deletion worked; the copy silently did not. So the directory looked converged while host source stayed frozen at the first manual rsync, and because the build's COPY inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. The provenance guarantee was false. Nothing caught it because the verify steps asserted the marker, health, and a 200 from /readyz -- all of which pass on a frozen host. None measured content. Fixes: - stage at /tmp/hrafn-deploy-stage, outside the target - CI computes context_sha256 over the shipped file list; the playbook recomputes it on the host post-converge and fails on mismatch - compare the running container's src/**/*.py against the host's, catching a SHA tag naming layers the image does not contain - checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner workspace fails the job rather than shipping a stale tree Declined --no-cache: a cache hit is correct when the context is genuinely unchanged, and the new assertions prove the property directly rather than brute-forcing it. The container-vs-host check compares only *.py -- `pip install .` generates src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against the live container before shipping (12 host files, 18 in container, 0 content differences). |
||
|
vh
|
bb19a96f39 |
fix(playbooks): compare image IDs, not the created-from tag; make the repin idempotent
Two defects the worldtree-pinned case exposed in the guard written an hour ago. 1. The guard compared the container's .Config.Image STRING against the tag being pinned. That string is only the tag the container was CREATED from, which can differ from what it actually runs: worldtree-pinned was created from `:latest` back when that tag pointed at 446e5807, and `:latest` has since moved to b19afd71d7cc. So the guard refused an instance whose pinning was correct and necessary. Now it resolves the target tag to an image ID and compares that against the running image ID -- asserting the thing actually cared about, that this tag names the bytes now running. It also fails closed when no such local tag exists. 2. The sed step reported CHANGED unconditionally, so a re-run on an already-pinned instance claimed work it had not done. Gated behind a `when:` that skips when the line is already correct; a second run on demo now reports "2 ok, 0 changed, 2 skipped / overall: OK". Applied to worldtree-pinned under worldtree-dev authorization. That instance needed a `docker tag` first -- its image was DANGLING (no repo tags, kept alive only by the running container), so the fleet's frozen reference was one `docker rm` from garbage collection. Tagged as :446e5807bf43, then pinned. All three instances now render a SHA with no floating tag anywhere: worldtree -> :ae88a057c0ed worldtree-personal -> :f63529168c13 worldtree-pinned -> :446e5807bf43 Nothing restarted -- pinned still Up 3 months, its start time unchanged. |
||
|
vh
|
064181a8fb |
feat(playbooks): re-pin Worldtree WORLDTREE_IMAGE off the floating :latest tag
Both corviduo-dev Worldtree instances carried `WORLDTREE_IMAGE=.../worldtree:latest` in their .env while running SHA-tagged images built the same day. The local :latest tag pointed at b19afd71d7cc, built 2026-06-14 -- 69 days stale. Any `docker compose up` on either instance, by anyone, silently downgraded that service by 69 days: the same footgun behind the 2026-06-15 outage. Applied under worldtree-dev authorization (Worldtree #410): demo -> gitea.phasefinal.com/vh/worldtree:ae88a057c0ed personal -> gitea.phasefinal.com/vh/worldtree:f63529168c13 Both runs verified compose config resolves every service to the pinned SHA with no :latest remaining, and that no container restarted. The edit is inert by design -- it changes what the NEXT recreate pulls. The playbook guards against pinning a SHA that is not the one actually running, which would re-arm the same hazard with a different image. Also records the worldtree-pinned case, deliberately NOT changed here: that instance runs a DANGLING image (sha256:446e5807, no repo tags), kept alive only by the running container. It has no tag to pin to, so it needs `docker tag` first -- and an untagged image referenced only by a container is one `docker rm` from garbage collection, which for the fleet's frozen reference instance is worth fixing on its own merits. Out of the authorized scope (demo + personal); raised with worldtree-dev. Stopgap: the durable fix is the deploy workflow stamping the deployed SHA into .env at each deploy, queued repo-side with worldtree-dev. |
||
|
vh
|
11b9d1891e |
fix(hrafn-ci): make the deploy converge instead of accrete
The first CI run shipped clean but revealed a design gap in the playbook: unpacking the context tarball in place overwrites tracked files and never removes anything. Leftovers from the pre-CI hand-rsync (tests/, docs/, ROADMAP.md, persistent-memory.md, CLAUDE.md, LICENSE) survived the deploy and had to be cleaned off ana-docker by hand. That is the same failure class that produced the mess in the first place: a deploy that only ever adds cannot return the host to a known state. - unpack to a staging dir, then rsync --delete onto the compose dir - protect host-owned .env and .deployed by name - add .env.example to the context tarball so converge does not delete it - record in the workflow that the tar list is now AUTHORITATIVE: anything omitted is removed from the host on the next deploy Re-validated with `elway --dry-run` (9 steps, 3 verify, parses clean). Not yet in vh/hrafn -- infra-ops has no write access there, so this is offered to the repo holder rather than pushed. |
||
|
vh
|
b001d0cb2e |
feat(hrafn): author the CI deploy that replaces the hand-rsync path
hrafn was handed to infra-ops for uptime ownership with no CI deploy and no commit provenance -- the image was always local/hrafn:v1 and the whole working tree lived in the compose directory. These two files fix both. They are authored here because infra-ops owns hrafn's uptime, but they belong in vh/hrafn; claude-bot is not a collaborator there, so they are handed to the repo holder rather than committed directly. - playbooks-deploy.yaml -> vh/hrafn playbooks/deploy.yaml - gitea-workflows-deploy.yaml -> vh/hrafn .gitea/workflows/deploy.yaml Design calls recorded in the README: the build context travels as one tarball rather than per-file upload steps (nevermore's pattern fails open when a new source file has no matching step), and the playbook refuses to deploy unless .env exists at 0600 -- a guard prompted by it arriving 0644 with a live bearer token in it. Validated with `elway --dry-run`, which caught a real interpolation bug during authoring. No new Actions secrets needed. |
||
|
vh
|
b6924de728 |
feat(hrafn): register the browser-fetch service and tighten its env perms
nevermore-claude handed hrafn to infra-ops for uptime ownership. Intake: verified health independently, brought it into the inventory, and fixed one security defect found during the check. - canonicalize stacks/hrafn/ (compose.yaml + .env.example + README) - list hrafn in the ana-docker running-stacks table - README records the load-bearing bits: shm_size 1gb is required for Chromium, playwright and the base image are version-coupled, SSRF denies private targets by default, and restart:unless-stopped does NOT act on healthcheck failure Security fix applied on the host, not in this tree: /opt/docker/compose/ hrafn/.env was 0644 with a live 57-char bearer token in it, readable by every local account (verified by reading it as `nobody`). Tightened to 0600; owner unchanged so the deploy path still works, confirmed via `docker compose config` as lkraven. Container untouched and still healthy. Known gaps recorded rather than silently accepted: no CI deploy, and the image builds from an rsync'd working tree living in the compose directory, so the running image has no commit provenance. |
||
|
vh
|
7bf17dd39e |
fix(filezilla): set restart: unless-stopped and canonicalize the stack
The filezilla container on vm-esh-nas shipped with no restart policy. It was SIGTERMed during the 2026-08-18 host shutdown and never came back, staying down four days while dockge/dozzle-agent/beszel-agent all recovered on their own. - add `restart: unless-stopped` (applied on the host, container recreated and verified serving 200 on :5800 with all four /NAS mounts visible) - promote the previously un-canonicalized compose into stacks/filezilla/ so the policy is version-controlled rather than host-only - record the missing-restart-policy failure mode in the host README |
||
|
vh
|
837fa362fc |
memory: snapshot — sec rebuilt on DFlash2 + newer vLLM; quant tokenizer defect fixed fleet-wide
Rewrites the in-flight section (140 lines to 18) now that the AI-seat work has settled, promoting the durable calls into the dated decision log where they will not decay: DFlash2 measured and promoted to sec's compose stack, the calibration-truncation defect fixed at source in the quant pipeline, sec retuned after a runtime OOM, the Max-Q spread disproved by telemetry, ESH dual-stack with the Cityside static proven unprovisioned, and speaches live with the weak-gate finding. Carries forward the two things a fresh session must not get wrong: the sec degeneration question is confounded because engine and drafter changed together, and neither the 2k nor the 10k observation is evidence. Auto-archived one entry (Recent decisions: 1) to archival-memory.md. Eight further candidates were held back by the guards -- open deferred work or referenced by in-flight state. |
||
|
vh
|
6e82899ba7 |
fix(mog-sec): retune to 0.52/420K after a runtime OOM at 0.55/480K
The 0.55 configuration booted cleanly, passed every startup check, and then killed the engine on an ordinary request of roughly 20k tokens: torch.OutOfMemoryError allocating 614 MiB in the GDN causal_conv1d path with 443 MiB free. The container self-recovered via restart: unless-stopped. Context length was not the cause. gpu-memory-utilization is not a hard reservation. Weights and KV are reserved up front, but activation memory is allocated on demand and the profile that sizes it uses dummy data at max_num_batched_tokens. The profile predicted 3.62 GiB; the process sat about 1 GiB over its entire budget even at idle. With six vLLM containers sharing GPU1 there was no slack left to grow into, and a modest allocation lost the race. 0.52 leaves roughly 7.7 GiB free against 4.6 GiB at 0.55. Records a measurement that governs how max-model-len can be chosen at all: the KV pool varies between boots. Two consecutive boots at 0.52 produced 425,663 and 453,600 tokens, a 6.6 percent swing, because the pool is sized from free memory at startup and that depends on what the other tenants hold at that instant. 420,000 sits under the lower observation so the seat starts in both conditions. Sizing to the higher one would boot today and refuse tomorrow, which is why the value was measured across boots rather than estimated from one. Concurrency of about 1.0 to 1.08x means one request at full length. Shorter requests still batch normally, since vLLM schedules by tokens rather than by max-model-len. |
||
|
vh
|
8389470898 |
feat(mog-sec): promote the DFlash2 configuration into the compose stack
Operator approved after real-use testing. The experimental standalone container is retired and stacks/mog-sec is canonical again, with restart: unless-stopped so the configuration survives a reboot. Cutover verified against the container it replaces: KV pool 526,617 tokens at 1.10x concurrency, identical; zero restarts; both gateway aliases serving; DFlash2 confirmed drafting at k=7 with 231 draft tokens over 33 drafts; vision working at 2048x2048. One variable was deliberately dropped rather than carried over. The previous stack hardcoded PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, the validated container never set it, and the quant playbook records expandable_segments corrupting retained tensors in another context. The compose now defaults it empty via MOG_ALLOC_CONF. Promoting the stack as it stood would have shipped a variable the tested configuration did not have. The speculative config moves into a single MOG_SPEC_CONFIG carrying the whole JSON, because the two shapes are not interchangeable: dflash requires a model pointing at the drafter and MTP must not have one, so a method-plus-tokens template cannot express both. Also parameterised: MOG_DRAFT_MODEL, MOG_MM_PROCESSOR_KWARGS, MOG_MAX_NUM_BATCHED_TOKENS. The mm-processor image cap is now mandatory rather than incidental. The model's own preprocessor declares 4096x4096, which expands to 16384 image tokens and kills startup on builds that enforce the image-token count check. Adds the .env.example this stack never had, carrying the measured rationale for each value and the one-line rollback. |
||
|
vh
|
20ac53052b |
memory: retract the MTP-head degeneration hypothesis; n=1 was never evidence
Operator ruling: the multi-turn degeneration lives in the un-fixed vLLM, not in the weights. The hypothesis that sec's stock-graft MTP head causes it is withdrawn. Two failures produced it. First, a false dichotomy treated as a deduction: having verified gen and sec run an identical engine, I concluded config was eliminated and therefore the weights were responsible. That does not follow. An engine bug present in both seats is not exonerated by the seats being identical; it only means the engine cannot explain a difference between them. It can still explain the failure. Second, and more instructive, the difference being explained may not exist. The premise was a single operator observation made during a session with many concurrent changes. That cannot carry a causal claim, and it became the load-bearing support for a root-cause narrative it could not hold. The same caveat now attaches to the coherent-to-10k observation on the new build: same n, same uncontrolled conditions, opposite direction. The comparison is weak at both ends, so the file no longer presents either sighting as a result. What survives as measured fact is unchanged and still recorded: sec's MTP head is byte-identical to the uncensored base across all 15 tensors, gen's was abliterated in-band, and acceptance differs slightly. None of that is shown to cause degeneration. Adds the generalisable lesson: an observation made while many things are changing cannot support a causal conclusion. It is the inverse of the warning already in the gen-seat compose file, which guards against trusting a negative result from a synthetic probe; this guards against trusting a positive sighting from an uncontrolled session. |
||
|
vh
|
ab3a0ca5bc |
docs(quant-playbook): acceptance is not throughput -- always run the depth control
Measured 2026-08-22 on one target with one instrument: raising MTP num_speculative_tokens from 3 to 7 improved accepted length from 2.753 to 3.041 per forward pass while throughput fell from 114.9 to 74.0 tok/s. Reporting acceptance alone would have recommended a 36% regression. The cause is architectural rather than model-specific. A single-module MTP head has no depth of its own, so vLLM runs it autoregressively and k draft tokens cost k sequential forward passes. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. Records the comparison rule that follows: match k when comparing two speculative methods, or the measurement is of depth rather than method. A parallel-drafting drafter at k=7 against an autoregressive MTP at k=3 is not a method comparison. In the case that produced this, the depth control showed most of the apparent acceptance advantage was depth, while the throughput advantage was real and came from parallel drafting -- our MTP was better at position 0 and still lost overall. Only the measured, model-agnostic result is recorded here. The DFlash2-specific findings, the hypotheses that remain unproven, and the wrong turns taken along the way live in persistent-memory.d/2026-08-22-dflash2-spec-decode.md with explicit epistemic labels, deliberately kept out of the playbook. |
||
|
vh
|
9f87b7c4e5 |
memory: DFlash2 spec-decode measured; sec running on it (experimental, confounded)
Records the 2026-08-22 session with measured results, hypotheses, and wrong turns explicitly separated -- the operator held this back while it was in flight specifically so conjecture would not enter the record as fact. MEASURED. DFlash2 works on an abliterated + NVFP4 compressed-tensors target on Blackwell sm_120, which the model card does not claim (it tests stock BF16 on H200). gen 2.753 -> 3.254 accepted tok/forward and 114.9 -> 131.9 tok/s; sec 2.676 -> 3.252 and 110.5 -> 130.0. The drafter is model-agnostic across two different finetunes to 0.06%, but is EAGLE3-style coupled to its target's hidden states, so the weights file is shareable while the 3.85 GB of VRAM is per-seat. The k=7 MTP control is the load-bearing result: raising MTP depth improves acceptance and collapses throughput to 74.0 tok/s, because a single-module head run autoregressively costs one forward pass per draft token. Without that control the obvious recommendation would have been wrong. CONFOUNDED. sec no longer degenerates at 2k, but the engine advanced 259 commits and the drafter changed at the same time. Isolating it means running MTP k=3 on the new build. Also recorded: #51113 is present in both builds and is therefore necessary but insufficient, since sec ran it and still degenerated. HYPOTHESES, labelled as such: that sec's stock-graft MTP head causes the degeneration, and that NVFP4 explains the gap against published acceptance figures. Neither is proven. WRONG TURNS, recorded so they are not re-derived: version strings are not lineage, Docker Hub push timestamps are not source freshness, and the claim that 1M context needs YaRN absent from config is false for the sec quant. Operationally important: sec is serving from a standalone container rather than its compose stack, which is stopped but unmodified. Rollback is two commands and is written down. |
||
|
vh
|
0755ba7d00 |
fix(quant): stop baking the calibration truncation cap into the shipped tokenizer
load_calib tokenizes with tok(..., truncation=True, max_length=seqlen). For a
fast tokenizer that mutates the Rust backend's truncation state in place, and
the subsequent tok.save_pretrained() persisted it, so every mixed-NVFP4 build
shipped a tokenizer.json carrying
"truncation": {"direction": "Right", "max_length": 2048, ...}
against a source whose value is null. Every prompt was clamped at the
calibration length, permanently.
It hid because older transformers does not enforce the text-vs-ids count
check. On a newer one the seat dies at startup with a message that names
images and never mentions tokenizers:
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384].
The cap also silently limited image resolution well before it killed
anything -- at 2048 the largest servable image is about 1448x1448, since
(edge/patch)^2 / merge^2 image tokens have to fit under it.
Fix saves a pristine tokenizer re-read from the source rather than the
mutated calibration object, and then asserts truncation is null so the
defect fails the build instead of shipping again.
Playbook gains section 3.14 with the symptom, the cause, the audit one-liner
and a table of which builds were affected, plus a fourth mandatory post-step.
The transferable lesson is called out: this is the third case of an artifact
carrying config authored against an older transformers that a newer one
begins enforcing, so an image bump is a config-compatibility event rather
than just a version change.
|
||
|
vh
|
ad21302474 |
memory: ESH internal IPv6 live on two LANs; Cityside v4 static proven unprovisioned
Operator-directed IPv6 enablement on ESH-UDMPM: esh-userland (VLAN 10, SSID PVC) and esh-server flipped from ipv6_interface_type=none to pd against the delegated 2607:73c0:402:1d00::/56. Verified esh-docker-vm egresses over v6 as its own address with no NAT, against v4 which still egresses as the shared CGNAT pool. No disruption through either write; originals backed up. NH3 stays v6-off deliberately -- its single /64 is reserved for network meshing, so nh3-dev having no v6 egress is correct rather than a fault. The v4 static is NOT provisioned, and this is now proven rather than inferred. A full gateway reboot -- verified by observed down-transition and an uptime reset from 140,846s to 183s -- forced a fresh DHCP DISCOVER and Cityside returned the identical address. Recorded alongside it: a DHCP renew could never have surfaced a new address in the first place, since renewal preserves the lease by design. Next step is a carrier ticket, not more testing. The v6 firewall posture was audited and is correct -- default-deny inbound for both IP versions, only four hand-written policies, none WAN-facing. My earlier warning that v6 removes CGNAT's accidental shielding was overstated and is retracted in the record. The audit's limitation is stated: it read rules, it never probed from outside, because no v6 vantage exists. Flat-zone lateral-movement finding parked as id 44 rather than actioned -- operator's call, moot until the mesh and segmentation design settle. |
||
|
vh
|
c7e21879ae |
memory: speaches ASR live on irv-ml1 — and no_speech_prob is a weak gate
Records the deployment plus the finding that outlives it: no_speech_prob alone does not carry a hallucination threshold on this stack. Silence and room tone both hallucinated 'Thank you.' at no_speech_prob under 0.11, where a conventional >0.6 gate passes them through; avg_logprob separates ~6x better. Any future consumer asking for a Whisper hallucination gate should be pointed at the composite, not the single field. Also banks the digest-pin rationale (the VAD flag is an upstream-declared unstable variable name, so a floating tag can move gate semantics silently), the two deployment gotchas, the docker-vs-native GPU index inversion on this host, and Eyra's future diarization VRAM claim on the same A6000. Adds speaches to the irv-ml1 stack table alongside parakeet, with the reason both exist. |
||
|
vh
|
aa5863c9a3 |
feat(speaches): OpenAI-compatible faster-whisper ASR seat on irv-ml1 A6000
Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves
large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204,
fp16, both resident, ~5.9 GB VRAM against 20 GB still free.
Sits alongside the existing parakeet stack (:8765) deliberately: parakeet
is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob
concept, so it structurally cannot serve this consumer.
The load-bearing requirement -- segments[].no_speech_prob surviving
response_format=verbose_json -- is VERIFIED on both tiers.
Measured finding worth more than the deployment: no_speech_prob alone is a
WEAK hallucination gate on this stack. Pure silence and pink room tone both
produced the classic Whisper 'Thank you.' hallucination while no_speech_prob
stayed under 0.11 -- a conventional >0.6 threshold passes both through.
avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs
-0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556.
Consumers should gate on a composite, not no_speech_prob alone. Table in
the README.
VAD pinned OFF at the consumer's request (they VAD-gate upstream on the
capture edge). Consequence stated plainly in the README: with VAD off this
service will transcribe silence into text and is not defending itself.
Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off
setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly
marks unstable. A floating tag could rename it on any bump, silently
restoring VAD and moving no_speech_prob semantics under a calibrated gate
with no error and no log line.
Two deployment gotchas recorded: PRELOAD_MODELS only loads models already
cached (it does not download -- use POST /v1/models/{id}), and the bind-
mounted cache needs a hub/ subdir or every /v1/models call 500s with
CacheNotFound while /health still returns 200.
|
||
|
vh
|
ff5ce212da |
memory: OWUI credential state — password reset, admin-scoped API key vaulted (operator-ruled)
Two OWUI credential facts worth surviving a context reset:
- The reported lockout was NOT a broken account. The vaulted password returned
200 from POST /api/v1/auths/signin immediately before the reset, so the cause
was client-side. Test the vaulted credential against signin before concluding
an account is broken -- this is the second lockout here traced to an assumed
rather than verified credential.
- The operator-minted API key is ADMIN-scoped, inherited from the owning user,
and that is the INTENDED shape per operator ruling: infra-ops manages the
instance, so the key needs the rights the management work requires. A
least-privilege dedicated-user split was offered and declined. Do not
re-propose it.
Both credentials vaulted at esh-docker-vm/open-webui-{admin,api-key}; key verified
against /api/v1/auths/ (admin identity) and /api/models (33 models), with an
unauthenticated 401 control. Plaintext copies shredded.
Also folds in an unadjudicated drift found in passing: the running image is the
floating ghcr.io/open-webui/open-webui:main, not the v0.11.0 the stack notes
claim. Replaces the now-resolved 'operator's OWUI password change' follow-up.
|
||
|
vh
|
b8e5022a1a | memory: snapshot — three AI seats settled (orcarouter gen / MeroMero dual-mode RP / sec pen-test), Open WebUI deployed | ||
|
vh
|
5e47a59b32 |
feat(meromero): MeroMero-v2 dual-mode (prose + streaming CoT) live on one seat — no re-quant
The multi-turn Gemma-4 CoT problem is solved. One MeroMero-v2 seat, one weight set,
two aliases: char-rp (prose) + char-rp-reasoning (streaming chain-of-thought).
The winning stack, traced from vLLM source by the four-arm brokkr/dwarf panel:
- vllm/vllm-openai:v0.26.0 — ships transformers 5.14.1 natively, below the
head_dim guard, so Gemma-4-31B loads with no pin and no custom image. It also
carries the #48217 streaming pre-arm fix.
- A patched chat template whose enable_thinking:true branch force-opens a BARE
<|channel> (not <|channel>thought\n -- full-open defeats _preprocess_feed's
injection). --chat-template override, no re-quant.
- Two served-names char-rp / char-rp-thinking; --reasoning-parser gemma4;
default enable_thinking:false. LiteLLM char-rp -> prose, char-rp-reasoning ->
the thinking served-name with enable_thinking:true.
Verified: streaming CoT split 6/6 direct on :8016 and 3/3 through the gateway;
char-rp prose clean on both transports with no trailing-token leak.
Two hard-won facts recorded in persistent-memory:
- STREAMING ONLY. Non-streaming can't split -- extract_reasoning never receives
prompt_token_ids so the pre-arm can't fire (a vLLM one-shot bug unchanged
across v0.24-0.27). Fine here: Lobe/OWUI stream. Upstream PR #49797 fixes
non-streaming too, landing ~v0.28.0 -- then it's a clean image bump.
- KEY-NAME TRAP: vLLM streams reasoning in delta.reasoning; LiteLLM normalizes
to delta.reasoning_content. I lost two false-negative test rounds to this.
Canonical: stacks/meromero-charrp/ (compose + patched_chat_template.jinja) and
stacks/litellm/conf/config.yaml. Rollback is the .env image line + dropping
--chat-template.
|
||
|
vh
|
76834777a4 |
feat(open-webui): deploy env-declarative chat bake-off on esh-docker-vm:3211
Stand up Open WebUI v0.11.0 parallel to lobe-chat (:3210) as an operator-approved candidate replacement. Env-declarative config (ENABLE_PERSISTENT_CONFIG=False = the deploy is the config source of truth), model picker auto-tracks the LiteLLM gateway with no pins, background task model pinned to summarizer, TTS routed direct at the dots gateway (:8198). Gates verified on the box: G1 (declarative config, both directions), G2 (picker == live gateway roster, no hand-listing), G3 (models/sync genuinely reconciles create+delete), G5 (task model pinned). G4 (TTS) handed to tts-dev. Credential posture (G6): fresh capped LiteLLM key open-webui-esh (all-proxy-models, $50/1mo budget) rather than inheriting the uncapped lobe-chat-esh key; signup locked off after admin creation. Secrets vaulted under esh-docker-vm/open-webui-*. Note: the API-key toggle env var is ENABLE_API_KEYS (plural); a var only reaches the container if compose names it in environment:. lobe-chat is untouched. |
||
|
vh
|
f01ee28cea |
memory: gemma4 CoT test on v0.27.1 blocked by config incompatibility (reverted)
Tried serving the MeroMero NVFP4A16 quant on vllm/vllm-openai:v0.27.1 to test the
per-request enable_thinking hypothesis. Couldn't even load it:
- v0.27.1's stricter transformers raised AmbiguousGlobalPerLayerAttributeError
on the Gemma-4 config's head_dim (now a per-layer attribute).
- Setting allow_global_per_layer_attribute_access on text_config downgraded that
to a warning, but then gemma4.py load_weights asserted:
"Attempted to load weight (512) into parameter (256)". Gemma-4-31B is
genuinely heterogeneous -- some layers have head_dim 512, not a uniform 256 --
so forcing the global value built wrong-shaped parameters. The guard was right.
So the MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot
load on v0.27.x without a config migration or a re-quant against newer transformers.
The per-request-kwargs hypothesis was never reachable -- the model wouldn't load.
Consequence worth carrying: the eventual gen-seat move to v0.27.2 stable must
re-verify config-compat for any Gemma-4 seat of this vintage.
Fully reverted: config.json restored (flags removed), image back to latest
(v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified. char-rp stays
on v0.24.0; MeroMero CoT remains undelivered. For RP-with-CoT use gen-reasoning, or
re-quant MeroMero against v0.27.x (real work, unproven payoff).
|
||
|
vh
|
7ebbcec5bb |
memory: vLLM release status for #51113 — no stable yet; gemma4 CoT is decoupled
Checked upstream. Latest stable is v0.27.1 (Aug 11) and it does NOT contain #51113 -- the v0.27.0..v0.27.1 compare is only "DSpark Markov heads" plus three CI commits. The #51113 cherry-pick (khluu, Aug 9, "v0.27.0 cherry picks" milestone) is staged on the release-0.27 branch but no v0.27.2 stable has been cut; only v0.27.2rc1 / the pinned nightly carries the fix. So the standing follow-up (move gen off the nightly) still waits on a v0.27.2 stable cut. Key decoupling: #51113 fixes the Qwen3.8 MTP + prefix-caching accuracy drop (#43559). It is irrelevant to Gemma-4 seats, which have no MTP head. So the meromero/gemma4 CoT test does NOT need the nightly and does NOT need #51113 -- it can run on the latest stable v0.27.1, which is newer than the seat's current v0.24.0 and where per-request chat_template_kwargs is expected to work. |
||
|
vh
|
b84ad888d6 |
memory: CORRECT the MeroMero-CoT finding — not a hard wall, not MeroMero-specific
My prior entry claimed the gemma4 reasoning parser is process-wide and the CoT
was architecturally blocked. Reading the actual code proves that wrong:
- Template (chat_template.jinja:347-352): thinking is a PER-REQUEST toggle.
enable_thinking:false prefills an empty <|channel>thought<channel|> to
suppress; true omits the prefill so the model can emit a real thought channel.
- Parser (vllm/reasoning/gemma4_utils.py): splits on <|channel>/<channel|> tag
presence, "works with or without enable_thinking" -- NOT a process-wide flag.
The stale compose comment I trusted cited an old parser API (vllm/parser/
gemma4.py:439) this container doesn't run.
So the mechanism is gemma4-GENERAL (any gemma4 finetune on this template family),
and the two-served-name gen pattern should work. What actually failed the test was
that meromero runs vllm/vllm-openai:latest (v0.24.0), where per-request
chat_template_kwargs.enable_thinking didn't take effect -- whereas the gen seat's
pinned nightly demonstrably applies it. That's a version/plumbing issue, not the
model and not the architecture, and it would hit any gemma4 finetune on that image.
Unverified fix (needs a GPU window): re-serve meromero on the nightly image with no
process default and per-request enable_thinking. Currently reverted to known-good.
|
||
|
vh
|
a260b57974 |
memory: MeroMero-v2 CoT via vLLM gemma4 parser is not achievable (tried + reverted)
Operator asked to serve MeroMero (char-rp) with reasoning as char-rp-reasoning.
The model genuinely reasons (GRPO-trained with thinking; ships Think/NoThink
SillyTavern presets), but two facts kill the vLLM path:
- it's byte-identical to the weights we already serve (same zerofata repo,
chat_template sha 94899c0f matches our on-disk quant), so re-quanting adds
nothing; and
- the gemma4 reasoning parser's enable_thinking is PROCESS-WIDE, not per-request.
The seat's --default-chat-template-kwargs sets the parser state machine; a
per-request chat_template_kwargs.enable_thinking reaches the template but not
the parser (unlike the qwen3 parser the gen seat uses). So the two-served-name
gen/gen-reasoning trick can't work.
Verified empirically: added a char-rp-thinking served-name and removed the process
default; enable_thinking:true per request still produced pure prose, 0
reasoning_content. Per the seat's own comment, a dedicated thinking-on process
dumps all prose into reasoning_content with null content -- no clean split.
MeroMero reasons only client-side in SillyTavern, not through vLLM.
Fully reverted to known-good: single served-name, process default restored,
char-rp prose verified end-to-end. GPU0 both seats healthy, gen untouched.
Kept one change: MEROMERO_GPU_MEM_UTIL 0.52 -> 0.51. 0.52 no longer boots because
orcarouter gen (the GPU0 neighbor) is marginally bigger than the old heresy seat
(free 49.02 < needed 49.38 GiB); 0.51 fits at KV 2.00x @ 262K.
For real RP-with-CoT: gen-reasoning (orcarouter, uncensored, thinks) or a dedicated
Qwen-based RP-thinking seat, which needs a GPU slot.
|
||
|
vh
|
3d30a6530b |
feat(lobe-chat): pass every chat-capable model through the picker
Operator: "pass everything through, lobe is a test surface for me." The picker goes from 11 curated seats to all 25 chat-capable models the gateway serves, paid passthroughs included. Membership is derived from LiteLLM's declared `mode` rather than by taste: chat (15) and completion (1) are in, and the nine GLM/frontier entries whose mode is unset are chat models in practice. Out: audio_speech (ext-tts, tts-1, tts-1-hd, gpt-4o-mini-tts), embedding (qwen3-embedding) and rerank (reranker, reranker-a3-bge-v2-m3) — seven models that cannot answer a chat request at all, so listing them would recreate the visible-but-dead entry this list just got fixed for. Every candidate was probed live from inside the container before being pinned, which corrected an earlier claim in this file. char-rp and char-rp-reasoning are NOT retired: they are configured and their upstream seat (ana-ml2:8016) is refusing connections while under maintenance, which is a 500, not a 400. Both are retained deliberately — dropping a name because its seat is bouncing makes the list rot in the other direction. Only char-rp-fable is genuinely gone (400, absent from model_info). The comment now spells out the 400-vs-500 distinction so the next editor does not repeat the mistake. Also corrects the credential-posture header, which still described the key as scoped to free local models and cited the glm/kimi rejection as current verification. The key now carries the all-proxy-models access group with no budget ceiling, so that block had become a false security claim in the most load-bearing comment in the file. |
||
|
vh
|
303fb7a5aa |
feat(lobe-chat): add the sec seats to the picker, drop two retired ones
Two independent gates kept the new `sec` family out of Lobe, and only one of
them was visible from the symptom.
The picker never auto-discovers. `OPENAI_MODEL_LIST=-all,+<names>` clears
Lobe's built-in OpenAI catalogue and re-adds one model per `+name`, so anything
added to LiteLLM stays invisible until this list is edited and the container
bounced. That pin is deliberate — an unpinned picker offers models that fail on
click — but it means the list rots in both directions, and it had:
- `sec` / `sec-reasoning` missing (hosted_vllm/mog-sec-27b{,-thinking} on
ana-ml2:8019, added to config.yaml earlier today), and
- `char-rp-reasoning` / `char-rp-fable` still listed after being retired
upstream, i.e. two picker entries that 400 on click. Verified: a call to
char-rp-fable now returns 400 Bad Request.
The list is now curated to live, chat-capable, free-local seats — eleven, each
round-tripped through the container after the bounce. The paid family stays out
deliberately; that is now a picker decision rather than a key one.
Which is the other half of this commit: the `lobe-chat-esh` key is no longer
scoped to free local models. On the operator's instruction infra-ops swapped its
explicit array for the `all-proxy-models` access group, so it now reaches the
paid passthroughs with `max_budget: None`. The README documented the old posture
as current, which made it a security claim that was no longer true; it now
carries the change, what it costs, and the fact that the picker is the only
remaining gate.
|
||
|
vh
|
564f5ae4f6 |
chore(litellm): rename mog-sec aliases to sec / sec-reasoning
Operator's call: the pen-test seat is reached as `sec` (non-thinking) and `sec-reasoning` (thinking). Backend and served-model-names (mog-sec-27b[-thinking] on :8019) are unchanged; only the gateway alias name changes. Old mog-sec / mog-sec-reasoning now 404. |
||
|
vh
|
36c173c6a1 |
feat(mog-sec): quant + serve M.O.G.-SEC pen-test seat; PPL on gen; retire fable
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
|
||
|
vh
|
e4576f0989 |
test(gen-seat): PPL on orcarouter — mean 7.07 / median 5.76, within noise of heresy
Measured against a spec-decode-free probe on GPU1 (fable downed to free the VRAM, probe torn down after). eval_quality.py aborts PPL under --speculative-config, so a nospec probe is the only way to read it. orcarouter mean 7.0655 median 5.758 heresy mean 6.910 median 5.625 (+2.2% mean) our recipe's usual mixed-quant PPL: 7.059 -- orcarouter is identical to it So orcarouter matches heresy on fidelity and wins on every other axis: MTP acceptance 58.4% vs 47.2%, zero think-leak, vision 7/8. The gen-seat search that ran through Cold-Fusion, heresy, and preetpatel is settled on orcarouter. |
||
|
vh
|
ce09ac4fa6 |
test(gen-seat): add a real vision battery — orcarouter scores 7/8
surface_test.py's vision check is one image and one word. It proves the tower loads; it does not prove the tower works. This battery uses generated images with known ground truth so every answer is objectively gradeable. Against orcarouter NVFP4-mixed on the `gen` alias: T1 OCR, 5 lines incl. one at 18px PASS all 5 exact T2 counting + attribute binding PASS 7 circles / 3 triangles / 1 square T3 bar chart, 6 values + max/min PASS 6/6 exact T4b occlusion, star behind rectangle PASS T4c aspect ratio of a 160x140 rectangle FAIL called it taller than wide T5 two images, which has text PASS T6 four images, the seat's cap PASS all four named T7 five images, one over the cap PASS rejected with HTTP 400 No <think> leak on any vision call. The single miss is fine-grained relative-dimension estimation on a near-square shape, and it reproduced across two runs (the longer T4 called the same rectangle "equal width and height"). Counting, OCR, chart values and occlusion ordering are all solid, so this is a precise-geometry weakness, not a broken tower. Recorded so nobody builds a feature on this model judging relative sizes. T7 earns its place separately: it confirms the per-prompt image cap fails loudly with a 400 rather than silently dropping the extra image. |
||
|
vh
|
f85d102813 |
test(gen-seat): orcarouter passes every gate — in-band MTP head delivers +11 points
Gates run against the live seat while the operator tested in parallel. <think> leak (n=30, 4 prompt types + multi-turn) 0/30, 0 empty MTP acceptance 58.4% @ 117.11 tok/s median surface 6/6 abliteration survival 4/4 compliance deterministic quality gens coherent and correct PPL still blocked For scale on the leak gate, the abandoned h300 build scored 8/30 on this exact instrument, and its abliteration-survival samples had 2 of 4 open with "<think>Ok, let's figure this out:". Orcarouter has none. The headline is MTP acceptance. 58.4% against heresy's byte-identical base head at 47.2% is +11 points, and it sits level with our own in-band L35 at 59.1%. That is the additive in-band-vs-graft delta the entire Cold-Fusion experiment was built to measure and never cleanly delivered -- orcarouter handed it over for free because the author had already done the Robinson edit on the head. Surface 6/6 covers plain chat, vision, tool calling, the thinking split, a 36,042-token long-context retrieval, and streaming. PPL remains blocked on a spec-decode-free probe seat: it needs ~22 GB and GPU1 has ~16 GB free. Comparison target is heresy at 6.910. |
||
|
vh
|
ba53c30192 |
feat(gen-seat): cut over to orcarouter — live, 7/7 aliases, vision intact, no think-leak
Operator directive was seat-first so he can test while the gates run. GEN_MODEL -> /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed. Healthy in ~4 min. KV pool 401,550 tok / 1.53x. MTP drafter detected and wired, sharing embedding and lm_head with the target. 7/7 gateway aliases 200. Vision correct on the shape probe. Live decode observed at 102-133 tok/s under load. Critically, <think> does not appear in the top-20 first tokens on the live seat. That is the Cold-Fusion failure mode measured absent in production, matching the pre-quant screen on the bf16 (1.23e-06, rank 52). Rollback is one line to .env.bak-heresy-restored-20260821. Full gates were still running when this landed; PPL stays blocked on a spec-decode-free probe seat, which needs ~22 GB against GPU1's ~16 GB free. |
||
|
vh
|
c8f128bdff |
feat(gen-seat): quant orcarouter — its MTP head is already Robinson-abliterated in-band
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over. The operator asked whether we could apply the Robinson path to the MTP head. We cannot, because the author already did. compare_mtp_head.py against the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ -- mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered the edit from the weights alone (mtp_delta.py, added here): sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection |cos| between the two recovered dirs = 1.0000 ONE shared direction ||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18% That is the Robinson in-band MTP abliteration, already applied, with a direction that passes our sink screen outright. Nothing to do but preserve it, and the quant carries it byte-identically. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered. The new format screen paid for itself on its first real use: think_prior.py on the bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner. Two durable findings about the pipeline itself: The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was never necessary, it simply had a free card by coincidence. The first attempt OOM'd by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it. post_quant.py now builds a missing output index from the safetensors headers. A sub-23 GB quant saves one bare shard with no index, and post_quant needs one; this has broken three separate rounds and been hand-fixed every time. The header is read by struct-unpacking the u64 length and parsing the JSON -- never safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS. Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the ignore list (llm-compressor pruned it as always), preproc restored. Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so this build stays apples-to-apples with heresy's PPL 6.910. |
||
|
vh
|
bf65d0254d |
docs(pfi): evaluate the two gen-seat replacement candidates
preetpatel/Qwen3.8-27B-Uncensored-NVFP4 is disqualified on two independent hard
failures, both read directly off the artifacts via HTTP Range requests against the
safetensors header (about a megabyte, not a 20 GB download):
- ZERO mtp tensors. The author's recipe.yaml asks to ignore re:.*mtp.*, but the
written config.json has no mtp ignore entry while re:.*visual.* expanded to 110
explicit ones. That asymmetry is llm-compressor pruning a pattern that matched
nothing, i.e. the MTP head was never loaded. Costs roughly half our decode.
- NVFP4 W4A4, 4-bit activations. Precisely the AEON failure mode: the fidelity
gradient is W4A4 < W4+FP8 < W4+bf16, W4A4 drove ~15-20% stochastic degeneration,
and it collapses past ~30k context. The gen seat serves 262K.
orcarouter/Qwen3.8-27B-Uncensored checks out as a quant source: stock-Qwen base
rather than a reasoning-compression finetune, Arditi-style single-direction
abliteration, 15 mtp and 333 visual tensors verified present, chat template
byte-identical to the heresy build we are serving, and the gate is already accepted
on our token.
Also records the author's FP8 release as a noted-but-not-recommended third option:
far more traction, but 30.9 GB against NVFP4's 22 GB, and on a zero-sum GPU0 that
+9 GB comes out of the KV pool and breaks 262K context.
And states the imatrix constraint plainly. Our recipe has always requested
imatrix_mse and always silently fallen back to uniform MSE; playbook 3.13 warns
against assuming an imatrix would help before verifying llm-compressor can consume
external importance data at all. The W4A16 portions are data-free by construction
and cannot use it regardless.
|
||
|
vh
|
48410a6a90 |
chore(coldfusion-abliteration): delete the Cold-Fusion bf16 weights — ~154 GB reclaimed
Operator directive following the decision to abandon the Cold-Fusion base.
Removed with explicit literal paths, one at a time:
qwen38-27b-coldfusion-bf16 stock DavidAU base
qwen38-27b-coldfusion-abliterated-L35-bf16 Robinson L35
qwen38-27b-coldfusion-h300-mtp-bf16 Heretic-300 + MTP graft
qwen38-27b-coldfusion-heretic300-bf16 raw Heretic export
Verified against ZFS used, not df: 4.48T -> 4.33T, ~154 GB. No snapshots were
holding the blocks, all four paths confirmed gone, gen seat unaffected.
The last two were hardlink twins -- same inode, links=2, because the MTP graft
hardlinked every unchanged shard -- so deleting only one would have freed
nothing. `du` across several paths in a single invocation dedupes hardlinks and
reported heretic300-bf16 as 2.5K, which would have made a size estimate wrong in
both directions. Check `stat -c %h` before sizing a delete.
Kept deliberately, so the research record outlives the weights:
qwen38-27b-coldfusion-bf16.PROVENANCE.txt pinned HF revision 9c44193f
coldfusion-abliteration/ harness, 300-trial Optuna
journal, catatonia-T260.json
With those two, every deleted build is reproducible: re-pull stock at the pin and
replay the winning config.
Held back pending an explicit call: the two NVFP4 quants, h300-nvfp4-mixed (the
only remaining servable copy of the Heretic-300 result) and L35-nvfp4-mixed. The
directive named bf16 weights; these are quants, and there is no storage pressure
arguing for haste at 4.26T free.
|
||
|
vh
|
37e9e1ca7f |
revert(gen-seat): abandon Cold-Fusion, roll back to heresy — the leak is in the base
Operator directive, given before the result was in: if it's the base, abandon h300 and the base too. The dose-response said base (18.5% of 22.2%), so it fired. Live gen seat is /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed again, restored from .env.bak-coldfusion-L35-20260820. The h300 env is preserved at .env.bak-h300-abandoned-20260821. The clincher, same probe pointed at heresy: Cold-Fusion stock P(<think>) 0.1850 Cold-Fusion L35 0.2048 Cold-Fusion h300 0.2216 heresy (restored) not in the top 20, <0.002 A >100x gap between the families, which is why no rollback inside Cold-Fusion would have helped -- stock and L35 leak at nearly the h300 rate. Verified after rollback: 0/30 leaks and 0 empty on the same instrument that scored h300 at 8/30, with the EXISTING enable_thinking:false config; KV pool 403,065 tok / 1.54x, heresy's exact documented baseline; 7/7 aliases; vision intact. No LiteLLM change was needed, so the chat_template_kwargs fix is left unapplied -- it worked, but it was a workaround for a base we no longer serve. Cost, stated plainly: 8/100 refusals becomes 29/100, a 3.6x regression on the axis the whole Heretic-300 run existed to move. Accepted deliberately. What carries forward is the methodology, none of which lived in the Cold-Fusion weights: direction_scope=0 beating per-layer on a merged base, aggression not being the lever, PR #317 silently dropping the MTP head on save, the MPOA and sink-screen reasoning, the graft/KL/catatonia/export harnesses, and the finding that a pristine MTP graft accepts as well as an in-band edit. New acceptance gate earned here: run think_prior.py on a candidate's STOCK weights before committing GPU time. It is a ~10s CPU measurement and it would have disqualified Cold-Fusion before the 300-trial study ever started. Heretic's objective has no format-compliance term at all -- the same blindness that removed the self-harm guardrail. Nothing deleted. Every Cold-Fusion artifact, the 300-trial Optuna journal and catatonia-T260.json remain on disk. Abandon means stop serving, not rm. |
||
|
vh
|
5ee2325820 |
feat(coldfusion-abliteration): dose-response settles the <think> leak — base 83%, our abliteration 17%
Answers "how likely is it that our abliteration caused this?" with a measurement instead of a prior. P(<think>) at the first generated token, template rendered enable_thinking=false so the prompt already carries a CLOSED think pair -- the exact event behind the leak. Raw softmax, bf16, CPU-only, one process per model. Deterministic: stock reproduced to 17 significant figures across two runs. coldfusion-bf16 none (stock) 0.1850 rank 3 coldfusion-abliterated-L35-bf16 Robinson L35, mild 0.2048 rank 2 coldfusion-h300-mtp-bf16 Heretic-300, heavy 0.2216 rank 2 The stock, untouched base already puts 18.5% of first-token mass on opening a think block the template had closed. Abliteration adds a real, monotonic, dose-dependent +3.7 points -- a nudge on a pre-existing base, not the cause. Cold-Fusion is a reasoning-token-compression finetune, i.e. a model trained to think briefly, and the leak's text shape agrees: a compact correct trace with a trained transition marker, which is trained behavior rather than damage. This changes the options. Rolling back to L35 or stock does NOT fix the leak -- at 18.5% under temp 0.7 / top_p 0.8 they leak at nearly the h300 rate. Only leaving the Cold-Fusion family escapes it, at the cost of the 8/100 refusal result. The chat_template_kwargs fix is the correct lever. Durable methodology point: a forward-KL budget cannot catch this. Heretic minimizes forward KL(stock||abliterated), which is near-blind to the model putting new mass on tokens stock barely used -- that is reverse KL's job, and we measured exactly that asymmetry on L35 (reverse 1.43 vs forward 0.70). h300's KL of 0.0136 is not evidence of innocence. For any "did the abliteration break behavior X" question, measure P(token) directly. Ran CPU-only deliberately: 96 EPYC cores and 265 GB of RAM make a 27B forward pass cheap, so this cost no GPU window and no seat downtime, where the obvious route was stopping both GPU0 seats. Also normalizes two more abliteration output dirs from root-owned 0600 to llmuser 0664. The unreadable-model failure surfaces as FileNotFoundError rather than a permission error, which is worth knowing before it wastes a run. |
||
|
vh
|
91f4cf22e1 |
fix(gen-seat): diagnose the unterminated-<think> leak — model defect, temp-triggered
Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).
The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.
The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):
temp 0.7, pp 1.5 (current gen) 4/12
temp 0.7, pp 0.0 4/12
temp 0.7, pp 0.5 3/12
temp 0, pp 1.5 0/12
That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.
It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.
Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.
The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.
Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.
|
||
|
vh
|
1d3b80169a |
fix(nevermore): repoint onto live aliases — its LLM pass had been dead 8 days
nevermore pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired with the
granite seat on 2026-08-12. Every summarization call since then failed: 67
consecutive status=failure rows, 0 tokens, twice daily, entirely silently. The
briefing had been rendering with no LLM pass at all. Nothing alerts on
status=failure in the spend logs, so it took an unrelated question about
reranker VRAM to surface it.
It was also pinned to NEVERMORE_RERANK_MODEL=qwen3-reranker -- the incumbent
Brokkr R43 measured harming 80/90 fleet queries -- and was its ONLY caller,
while the production `reranker` alias sat at 0 calls for 4 days. The R43
cutover repointed the alias but never moved the consumer.
nevermore/.env LLAMA_SWAP_MODEL granite-4.1-8b -> summarizer
NEVERMORE_RERANK_MODEL qwen3-reranker -> reranker
(server-only; .env is excluded from the mirror both ways)
Verified against nevermore's exact call shape: summarizer returns clean content
with 0 reasoning chars at temperature 0.2 / max_tokens 4000; reranker scores
0.95 on-topic vs ~1e-5 off-topic; embedding returns dim-1024.
Retired alongside it:
vllm-rerank :8002 Qwen3-Reranker-0.6B + the qwen3-reranker alias
vllm-rerank-a4 :8014 gte-reranker-modernbert + its alias
vllm-granite :8004 Exited 8 days, dead service block
and vllm-rerank-a3 was promoted from a throwaway `docker run` into this stack
(the selection ledger's own open follow-up). Healthy in 55s. It keeps the
bake-off arm name so the ledger, memory and R43 record stay valid.
VLLM_VERSION is pinned latest -> v0.24.0. Every service in the stack shares that
one variable, so a bare `compose up -d` could have silently upgraded all of
them at once; both tags resolved to the same local image (4091d5593f77), so the
pin changed nothing at runtime.
GPU1 is down to 81,448 of 97,887 MiB -- 13.9 GB reclaimed tonight.
Correction: an earlier claim that A4 had no gateway alias was wrong. It did.
LiteLLM serves both config-defined and DB-defined models -- live showed 32
against config.yaml's 26 -- and grepping the file cannot see the difference.
/v1/models and /model/info (which flags db_model) are the ground truth. DB
models delete hot via POST /model/delete with no restart.
Left alone: reranker-a3-bge-v2-m3, a zero-call duplicate of `reranker` on the
same backend. It is Brokkr's cutover-verification handle -- redundant rather
than broken, and another agent's tooling is not mine to delete unilaterally.
|
||
|
vh
|
b990951d80 |
chore(vllm): retire LFM2.5-2.6B permanently; audit finds nevermore on the harmful reranker
Operator directive: lfm2.5-2.6b goes down permanently.
- stacks/vllm/compose.yaml vllm-lfm25 service removed (replaced by a
tombstone comment), pushed live to ana-ml2
- ana-ml2 container docker rm -f'd, 8,721 MiB freed on GPU1
(95,388 -> 86,667 of 97,887)
- litellm config lfm2.5-2.6b alias deleted, live + canonical,
28 -> 27 models
It was an EVAL-ONLY bake-off seat against granite-4.1-8b that never received
the operator ruling it was pending; the comparator was retired from the roster
on 2026-08-15; it was deliberately never wired into any default or fallback
routing chain; and spend logs show 0 calls in the 4-day window to 2026-08-21.
Weights stay in the shared HF cache -- nothing deleted from disk.
The gateway restart that makes the alias deletion take effect is HELD so it can
batch with a pending reranker change. Until then the name is still routable
in-memory and will error against a dead backend.
Auditing the three reranker seats while answering "why do we have three" turned
up a real problem. The design is one production, one rollback, one fallback --
but the traffic is backwards:
:8013 A3 bge-v2-m3 PRODUCTION, backs `reranker` 0 calls / 4 days
:8002 Qwen3-Reranker RETIRED incumbent, rollback only 7 calls, 12-hourly
:8014 A4 gte-modernbert "fallback" no alias at all
nevermore is hard-wired to the incumbent by name (NEVERMORE_RERANK_MODEL=
qwen3-reranker), so the R43 cutover never moved it -- the cutover repointed the
`reranker` alias and correctly left `qwen3-reranker` naming the Qwen model.
Brokkr R43 measured that model harming 80/90 fleet queries, so nevermore's
twice-daily rerank pass is likely degrading its own briefing.
Fix is one line in nevermore's .env plus a nevermore restart, and it must land
before :8002 is retired. Recorded in persistent-memory with the A4 alias also
noted as absent (global CLAUDE.md names reranker-a4-gte-modernbert; it does not
exist).
|
||
|
vh
|
e3ce713f7f |
feat(gen-seat): cut over to Heretic-300 — 7/7 aliases, vision intact, MTP 59.7%
Live GEN_MODEL is now qwen38-27b-coldfusion-h300-nvfp4-mixed (ana-ml2 GPU0 :8015). Served-name left unchanged so all 7 LiteLLM aliases route without a gateway edit. Verification: KV pool 401,550 tok / 1.53x (baseline 403k / 1.54x) LiteLLM aliases 7/7 green vision 3/3 shapes, colour+form+position correct MTP acceptance 59.7% median @ 118.37 tok/s quality gens 4/4 correct abliteration 4/4 compliance PPL NOT measured (see below) The roadmap predicted ~47% acceptance for a pristine MTP graft versus L35's 59.1% in-band edit. Measured 59.7% on the same harness: there is no acceptance penalty, which removes the throughput argument for reimplementing MPOA. A single long-prose generation read 47.5% off the same counters -- below the 8-run minimum of 49.0% -- and would have "confirmed" the prediction by coincidence. Acceptance must be read from quickbench.py, never one sample. PPL is blocked on VRAM, not on the model: eval_quality.py aborts with "prompt_logprobs look uniform" under --speculative-config, and the probe-seat workaround needs ~22 GB while both cards sit at ~96% committed. Also normalizes the quant dir from root:0600 to llmuser:llmuser 0664 to match every other model dir, and records that config.json sha256 is byte-identical across the h300 and L35 quants and is therefore useless for confirming which weights are mounted (mtime and a head-hash are the discriminating views). Rollback is one line to .env.bak-pre-h300-20260820. |
||
|
vh
|
407ca017ae |
memory: snapshot — Heretic-300 built, quantized and verified; gen-seat cutover is the next step
8/100 refusals at KL 0.0136, hand-verified coherent, beating the absolute-heresy bar 3.6x. NVFP4 quant complete: 21 GB, 1968 tensors, MTP head grafted back after PR #317 dropped it, and re:^mtp.* re-injected into quantization_config.ignore after llm-compressor pruned it. Self-harm guardrail is gone on this build and is the operator's own next work item; the four-dwarf panel is stood down. |
||
|
vh
|
f90a5025de |
feat(coldfusion-abliteration): Heretic-300 — 8/100 refusals at KL 0.0136, beats the heresy bar 3.6x
Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion. Best trial scores 8/100 refusals at KL 0.0136 against a 98/100 base, versus absolute-heresy at 29/100 and our hand-tuned Robinson L35 at 72/100 / KL 0.0116 — i.e. 64 fewer refusals for the same damage. Hand-verified coherent: correct arithmetic with shown working, clean code, 66-167 word prose across nine probes. Durable findings: - direction_scope=0 (single shared direction) is decisive on this merged base: n=129, best 8/100. Per-layer directions n=131 never beat 52/100 despite a better median. Points against the multi-direction intuition for a diffuse direction (our two-template |cos| is 0.62 vs Robinson's 0.99 on stock). - Aggression is not the lever. r(KL, refusals) = -0.561 over 261 trials; the KL<0.02 band contains both the worst results (median 87/100) and the single best. A KL 0.3554 trial scored worse than one at 0.0193. - PR #317 confirmed: Heretic silently drops the MTP head on save. Source 1199 tensors -> export 1184, all 15 mtp.* gone, vision 333/333 intact, exit 0, no warning. This is also why absolute-heresy ships a byte-identical MTP head — a bug, not a design choice. Always diff tensor keys after a Heretic export. - Heretic's recovered direction carries 6.18% of its energy in sink dim 3994, versus 0.094% for our L35 and 1.97% for the L39 we rejected as brick-inducing. It survives that only because of magnitude-preserving ablation (row_normalization=FULL); our plain projection has no such protection, so the sink screen correctly refused the in-band MTP graft. Same direction, different operation. MPOA is the prerequisite for in-band MTP on a Heretic trunk. - Heretic's edit is recoverable from weights: delta is rank-1 (s2/s1 ~ 0.010), SVD gives the direction, norms give per-layer weights (1.08 -> 1.34, i.e. over-projection). Cross-layer |cos| agreement 0.9903 independently confirms the single-direction result. New tooling in services/coldfusion-abliteration/: kl_divergence.py first-token KL, class-split, zero noise floor catatonia_gate.py 12 probes x 220 tokens, prints every completion heretic_export.py PTY driver; selects by measured value, never by menu position — Heretic's resume prompt puts "delete the checkpoint and all results" one arrow-key from the target graft_mtp.py recovers the trunk direction by SVD; --pristine for the safe path when the sink screen refuses Also adds quant playbook 3.13: the NVFP4 recipe sets observer="imatrix_mse" but llm-compressor has always silently fallen back to uniform MSE for want of importance data — on this build and on the incumbent. Existing A/B comparisons stay valid since every build shares the fallback. Parked as id 42. Guardrail note: this build has lost the self-harm guardrail that the Robinson L35 build retained. Restoration is the operator's own work item. |
||
|
vh
|
78484ac87d |
memory: GPU0 seat boot order is part of the state — restore rule + KV-pool baselines
vLLM sizes --gpu-memory-utilization against total VRAM but gates startup on free VRAM, so the GPU0 pair coexists only in its original boot order. Records the restore sequence (meromero to healthy first, then gen), the observed-not-slept rule, and the KV-pool baselines to verify a restore against — nvidia-smi used-MiB is the wrong check, it swings ~7 GB on allocator slack at identical capacity. |
||
|
vh
|
a9d73dad41 |
fix(coldfusion-abliteration): GPU0 seat restore order is load-bearing — correct the claim and the runbook
Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put
meromero into a 7-restart crash-loop:
ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less
than desired GPU memory utilization (0.52, 49.38 GiB).
The previous commit's README claimed restore order "is not actually load-bearing"
on the grounds that both seats pass --gpu-memory-utilization as a fraction of
total VRAM. That is half right and the wrong half mattered: the fraction sets the
target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole
target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so
the seats coexist only in the order they were originally brought up, and meromero
is the one that does not fit in the remainder. The pre-existing auto-memory note
("gen takes a fraction of free VRAM at startup and will starve meromero") was
pointing at the real effect.
Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a
two-to-three minute weight load is simultaneity, not ordering — gate on observed
state.
Recovery applied: stop gen, wait for meromero healthy, start gen. Verified
against the pre-window baseline rather than against "both green":
gen KV 14.36 GiB / 403,065 tok / 1.54x -> 14.34 GiB / 401,550 tok / 1.53x
meromero KV 542,202 tok -> 542,202 tok
RestartCount 0 on both; summarizer smoke-tested through LiteLLM
Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It
reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is
allocator slack — serving capacity is unchanged. The anomalous boots were the
high ones (34.95 GiB KV), where gen came up on an empty card mid-window.
|
||
|
vh
|
1b3fb270e7 |
feat(coldfusion-abliteration): first-token KL measured — 28.4x selectivity, harmless median 0.0211
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full 248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless and reserved-harmful prompts. Result (L35, 256 harmless / 104 harmful, answer mode): harmless median 0.0211 mean 0.0364 top-1 agreement 89.8% harmful median 0.5996 mean 0.6992 top-1 agreement 55.8% selectivity 28.4x (72.8x in think mode) Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are bit-identical between a single-process and a two-process run, so the figures are signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the live absolute-heresy seat) this is materially gentler, but those are the other tool's optimizer output on a different base with its own harmless set and template — order-of-magnitude, not head-to-head. KL remains a fidelity number; the viability gate is still MTP acceptance (59.1%). Method notes: - Prompt classes are reported separately by design. A single averaged KL over a mixed corpus is close to meaningless, since the metric is meant to be large on harmful prompts and small on benign ones; the ratio carries the information. - The harmless evaluation set is drawn from the alpaca pool minus calibration's own draw, reconstructed by replaying that draw rather than remembered, and asserted disjoint on text. The harmful set is the reserved test split. - `render` is imported from abliterate.py rather than copied, so the measurement cannot drift from the rendering the direction was captured against. - Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits. Three corrections to the runbook, each of which cost time: - "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must stop. VRAM is now sized from the safetensors headers at run time. - A 27B model cannot be released in-process: `del` + gc + empty_cache left free VRAM at 45,287 MiB, and so did confining the model to an inner frame that exits. Only process exit returned the card (96,689 MiB). The first run completed only because the allocator hit OOM, collected, and retried. Each model now gets its own process, handing log-probs to disk between stages. - The residency gate read hf_device_map, which transformers leaves empty when the model fits on one device — it reported "(unsharded)" whether or not anything was wrong, so it could never fail. It now reads parameter devices directly. Model-agnostic lessons promoted to the quant playbook (new 3.12). |
||
|
vh
|
8c354a0e79 |
memory: snapshot — Cold-Fusion thesis PROVEN (MTP 59.1% > incumbent 47%)
Flip the Cold-Fusion in-flight line to thesis-proven: L35 quantized to mixed NVFP4, MTP acceptance 59.1% median beats the incumbent Heretic graft's ~47%, abliteration survives quant. Not cut over — cutover is a separate operator decision. Records the two env foot-guns hardened (quant venv config-delegation drift; single-file no-index quant needs a header-built index). |
||
|
vh
|
725c8fdf9e |
feat(coldfusion-abliteration): THESIS PROVEN — in-band-abliterated MTP head accepts 59.1% (beats incumbent ~47%)
Quantized the L35 abliterated model to mixed NVFP4 and measured MTP acceptance
end to end. The experiment's whole premise: Heretic (the incumbent gen seat)
leaves the MTP head a byte-identical base graft its wrapper never loads, whereas
Robinson abliterates the MTP head in-band — the question was whether that in-band
edit survives well enough to spec-decode. It does, better than the graft:
MTP acceptance 59.1% median (51-65%, 8 cache-busted topics) vs incumbent ~47%
decode 118.7 tok/s median (faster; image-confounded, read as not-worse)
abliteration survives quant (creative refusals drop, self-harm guardrail
intact, coherent)
Output at /tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed (22.5 GB). Result
JSON in bench/. NOT cut over — the incumbent seat is untouched; making L35 the gen
seat is a separate decision needing the full Stage-3 gate + real multi-turn hold.
Two env foot-guns hardened along the way:
- quant_mixed_nvfp4.py now promotes text_config attention fields
(num_attention_heads etc.) to the top-level config for the oneshot, then
restores. transformers 5.10 / llmcompressor 0.12 (this venv moved under us
since the Aug-15 heresy quant) no longer delegate the top-level lookup, so
oneshot raised "Cannot determine num_attention_heads". Same "the fight is the
environment" pattern as the abliteration capture.
- a sub-~23GB quant saves as a single model.safetensors with no index, so the
post_quant MTP graft needed an index built first — from the safetensors header,
not safe_open (which mmaps the whole shard and ENOMEMs on ZFS).
post_quant grafted the abliterated MTP (15 tensors, 849 MB) and re-injected
re:^mtp.* into quantization_config.ignore (llm-compressor pruned it again — the
two-rounds-lost 0%-MTP bug, fired and repaired as designed). Probe served on the
pinned nightly (#51113 qwen3_5_mtp fix) to match the live seat's vLLM.
|
||
|
vh
|
c55b1390b7 |
memory: snapshot — Cold-Fusion abliteration LANDED at layer 35
Flip the in-flight status from 'capture done, calibration expansion next' to 'landed, works'. New detail file captures the three corrected diagnoses (layer- selection metric, sharding/allocator misdiagnosis, corpus-size falsified) and the verify/quant work still owed. Supersedes the -capture.md detail file's framing. |