vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: vh/esh-pfi-infrastructure:0ad332bb4a9bd00600816e424f18b907eb3c39d4
Branches Tags
vh/esh-pfi-infrastructure:main vh/esh-pfi-infrastructure:graphify-tooling
vh/esh-pfi-infrastructure:booth-v0.1.8 vh/esh-pfi-infrastructure:booth-v0.1.7 vh/esh-pfi-infrastructure:booth-v0.1.6 vh/esh-pfi-infrastructure:booth-v0.1.5 vh/esh-pfi-infrastructure:booth-v0.1.4
...
compare: vh/esh-pfi-infrastructure:7b5fd91d3cef754dd2f8ffa11772986f183bee55
Branches Tags
vh/esh-pfi-infrastructure:main vh/esh-pfi-infrastructure:graphify-tooling
vh/esh-pfi-infrastructure:booth-v0.1.8 vh/esh-pfi-infrastructure:booth-v0.1.7 vh/esh-pfi-infrastructure:booth-v0.1.6 vh/esh-pfi-infrastructure:booth-v0.1.5 vh/esh-pfi-infrastructure:booth-v0.1.4
44 Commits
0ad332bb4a ... 7b5fd91d3c
Author SHA1 Message Date
vh 7b5fd91d3c docs(gemma4-erp-tune): root-cause the 8.6% MFU — attention on Ampere kernels, 29.9% padding 
Run-01 was killed at step 19 by operator instruction to root-cause before
spending a ~13.9h window. Two independent methods now agree on where the step
time went, and neither was the hypothesis the consult panel converged on.

Scaling fit (3 points, 2 params, residuals <3ms over an 8x range):
  A = 6.87e-4 s/token, B = 8.85e-8 s/token^2
  quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384
  No fixed term was needed, which refutes launch-bound outright.

Profiler kernel table (device rows only):
  attention   22,835.8 ms  65.2%   fmha_cutlass*_sm80
  dense GEMM   2,774.0 ms   7.9%
  other        5,739.0 ms  16.4%

The attention kernels are sm80 — Ampere-generation CUTLASS running on an
sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the
mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS.

Correctness cleared separately: the sliding mask asserts at max 1024
allowed/row, so the 25 windowed layers were genuinely windowed. The same probe
found that right-padding is what pins the 5 global layers to an explicit 4D
mask and off the is_causal fast path — measured at 9.4% slower for 24% less
loss work at fixed width.

The largest available win is not the attention kernel. The corpus is 29.9%
padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall
clock, no new dependency, unchanged peak memory. Bucket size turned out not to
be a diversity knob — roots per accumulation window are flat across a 256x
range, so the global micro-batch shuffle does that work alone and the bucket
should be tight.

Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic
home (sibling to the quantization playbook), the four probes under
scripts/training-probes/ with raw output kept for re-derivation, and a §6 to
the sizing doc carrying the Gemma-4-specific numbers and round-2 restart
parameters.

Measured negatives recorded so they are not re-chased: grouped_mm (0.9%
slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over
the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever),
FA4 on sm_120.

Round-1 state preserved: 609MB encode cache, order manifest, truncation
report, resume script. No checkpoints — it died at step 19 and the first was
due at 100, so the lora_B inert-adapter gate never ran and moves to the
restart.
 2026-08-24 22:10:51 -07:00
vh 872c2c562f memory: the MFU hunt — two hypotheses measured and killed, consult dispatched 
Records what has actually been ruled out rather than what is suspected.

The hardware is fine: a plain dense GEMM at the same shape reaches
97.1% of the benchmarked 313.8 TFLOPS peak.

The Python expert loop is not the cause, which was my hypothesis and I
was confident in it. transformers' grouped_mm experts backend runs
0.9% SLOWER than eager with bit-identical output and identical peak
memory, and torch 2.13 has the kernel available, so it is not falling
back for lack of one.

MoE is not the bottleneck at all. Isolated at real shapes the block
runs at 26.5% of peak with 36% of its time in pure gather/scatter, and
a dispatch-free bmm version would reach 80.9% — but the whole MoE
contribution is only about 10% of a step. Making it free buys 7%.

So roughly 90% of the time is unaccounted for. The leading untested
hypothesis is that the five full_attention layers use global_head_dim
512, above FlashAttention-2's 256 cap, which would push SDPA onto a
slow backend for O(n^2) attention at sequence 16384.

Also records that the earlier 5% MFU figure was wrong in two ways —
unpadded tokens and a guessed peak — and that the operator caught it.
Padding is real but secondary at 29.9%.

Consult dispatched to brokkr-smithy-dev for the frontier-dwarf panel.
 2026-08-24 21:38:15 -07:00
vh 07743c6aff memory: snapshot — tune training unattended, MFU root-caused to a Python expert loop 
The in-flight section is rewritten around the run itself rather than
the decisions that led to it. The sizing and seat-call bullet collapses
to a pointer now that both are executed; its detail lives in
docs/pfi/gemma4-erp-tune-sizing.md.

Adds the measured MFU finding: 27.1 TFLOPS against a benchmarked 313.8
TFLOPS peak, root-caused by reading the source rather than inferring —
transformers runs the Gemma-4 experts in a Python loop, 128 experts
across 30 layers, roughly 11,500 iterations per optimizer step under
gradient checkpointing. Padding is a secondary 29.9% tax.

Records that my first estimate of 5% MFU was wrong in two compounding
ways: divided by unpadded tokens, and compared against a guessed peak
rather than a measured one. The operator pushed back on the number and
was right to.

The fused MoE kernel is deferred work with a tracking surface — park
id 47 — per the snapshot rule that deferred decisions go in Recent
decisions with a pointer, never into the volatile in-flight section.

Also records the resume trap: the original launch command begins with
rm -rf on the output directory, which would destroy both the encode
cache and every checkpoint. resume-run-01.sh exists so that cannot
happen.
 2026-08-24 21:20:58 -07:00
vh d6dfd61c91 memory: the ERP tune is running — override granted, 12 defects fixed first 
Operator overrode the corpus gate for one run on 2026-08-25, with the
grant staged beside the recipe rather than asserted in chat. It
deliberately does not flip any root's training_eligible flag, so the
signal that made the run stop in the first place survives intact.

Records where the run lives, what it is configured with, how to restore
the fleet, and the two lessons that generalise past this project.

The first is inert gates. Two turned up in one evening — auditcore,
whose CSAM hard-drop never fired across 42,662 records, and
validate_vision_keys, which compared model.state_dict() against itself
and could not fail on any input. Both read as guards. The question that
catches them is not whether the check passes but whether it can fail.

The second is an invariant enforced on one code path and not its
sibling. That was my own bug: INV-T9 requires a window to hold at least
one complete assistant turn, and I enforced it where the window is cut
but not where it fits, so a trailing user-only remainder became a
zero-loss window and killed the first launch. Same shape as the inert
gates, in code I wrote an hour earlier.

Two further foot-guns worth the space: enable_input_require_grads is
mandatory beside gradient checkpointing on a frozen base, or every
adapter stays at its initialisation and the run completes successfully
having learned nothing; and the upstream Gemma-4 template forward-scans
to suppress a closing turn marker before another assistant message, so
incremental rendering cannot tile against it and assistant runs must be
merged first.
 2026-08-24 21:00:17 -07:00
vh 33433e0d1e docs(gemma4-erp-tune): replace the estimates with measurements — they were 3x optimistic 
Ran the loss path on the real checkpoint on GPU0 with synthetic tokens.
The arithmetic held for parameter counts and was badly wrong for
activation memory.

    naive CE   bsz1 seq 8192    81.93 GiB
    naive CE   bsz1 seq16384    OOM
    chunked CE bsz1 seq16384    65.66 GiB
    chunked CE bsz2 seq16384    79.71 GiB   <- the run config
    chunked CE bsz4 seq16384    OOM

The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the
~5 GiB estimated: the estimate modelled gradient checkpointing as
storing layer inputs plus a modest recompute peak, and the real MoE
recompute peak with top-8-of-128 routing and its scatter/gather buffers
is far heavier. Dense-model intuition does not size an MoE run.

Two predictions landed exactly — 205 target modules and 74,342,400
trainable params at r64 — which is why the rest of the model of the
thing is still worth trusting.

The headline is that chunked CE at seq 16384 costs 16 GiB less than
naive CE at seq 8192, so chunking is what makes brokkr's 16384
recommendation reachable rather than an optimisation on top of it.
max_seq_len moves 8192 -> 16384 on his truncation finding: the cap
drops 6.2% of samples but 22.4% of tokens, concentrated entirely in
dialogue, which is 60% of the mix.

Also records the four harness changes this required (eitri-smithy
62b556b), including the inert-adapter trap: without
enable_input_require_grads() alongside gradient checkpointing on a
frozen base, no gradient reaches the adapters, every one stays at its
initialisation, and the run completes successfully having learned
nothing.
 2026-08-24 19:01:49 -07:00
vh 47ec3d1a97 memory: the ERP tune is blocked on a corpus gate only the operator can clear 
Every clean-v1 CLEANROOT carries training_eligible: false with two named
blockers, and the recipe states plainly that nothing in it is Charter §3
training-eligible.

I initially read scoped_grant: operator-2026-08-22 as authorization and
told brokkr-smithy-dev I was proceeding. That was wrong, and the person
who wrote the field corrected it: the grant governs INV-4 one-way tier
inheritance — the adapter is permanently internal-erp-rnd and never
distributable — not training clearance.

The stage-2 detector is measured-inert rather than merely unvalidated.
auditcore v3.7.2 returned its hard-drop exit code zero times across
42,662 raw RP records, its printed verdict ignores its own printed
threshold, and it passed a record a blind audit had already identified
as sexual content involving a participant the text marks as a child.

Verified the one thing that decides whether that specific record reaches
training: pippa-5083 is present in kept-manifest.jsonl (4,551 rows) and
absent from recipe-dedup-kept.jsonl (20,473 rows), which is the survivor
list the harness gates on. The substitute lexical screen caught it. That
is one known instance caught by a stopgap and says nothing about what the
screen misses.

Both brokkr and I recommend stopping. Neither blocker is hours of work.
 2026-08-24 18:51:52 -07:00
vh c9943b1507 docs(gemma4-erp-tune): whole-card placement — gen moves to GPU1, sec stands down 
Operator chose a third placement over the two the sizing offered: rather
than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to
GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no
co-tenant and gen never goes dark beyond its own restart.

Revised run parameters, since a whole card changes them:

- micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving
  888 optimizer steps instead of 444. At one epoch the step count is
  worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert
  per step against ~512 at micro-batch 1 — a far healthier GEMM on
  704-wide experts.
- Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock
  but pushes activations to ~24 GiB per sequence, which forces
  micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow.
- Scriberr stays on GPU1. The previous revision suggested moving it to
  GPU0, which was correct only while training was going to live on GPU1.

Records the ordering constraint in both directions, the elway identity
requirement, and that sec's aliases should be allowed to fail at the
gateway rather than be substituted with another model.
 2026-08-24 18:40:42 -07:00
vh 9d70100867 feat(ana-ml2): elway playbooks to open and close the ERP/RP tune window 
Operator call: rather than train beside gen on GPU0, move gen to GPU1 and
stand sec down for the night, so the tune gets a whole 95.60 GiB card and
the fleet's general seat never goes dark beyond its own restart.

Order is load-bearing in both directions and the playbooks enforce it.
gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction
of TOTAL card memory: 42,091 MiB must be FREE at startup or the engine
refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up, so
recreating gen onto GPU1 first would take the main seat down and leave it
down. mog-sec stops first and a hard gate checks the freed memory before
gen is touched. The close playbook mirrors it: gen must vacate GPU1
before mog-sec starts, since mog-sec needs 50,901 MiB of its own.

Close opens with a gate that refuses to run while a process is still
resident on GPU0, so it cannot evict a training run mid-flight.
Override with --var allow_busy_gpu0=true.

Three defects found and fixed while landing this, all worth keeping:

- Verifying GPU residency via `docker inspect --format {{.State.Pid}}`
  never matches. vLLM V1 runs EngineCore as a child of the container's
  pid 1, and it is the child that holds the memory and that nvidia-smi
  reports. Match by cgroup instead.
- A step's `sudo: true` does not extend to its when/creates/changed_when
  guards, which run as the login user. The root-only .env made an
  unsudo'd grep exit 2, so the GPU-id flip SILENTLY SKIPPED. The
  effective-value assert is what caught it.
- That assert originally grepped the config YAML for -\s*'?1'? and failed
  against compose's double-quoted `- "1"`. Parse the JSON with jq; an
  assert that fails for the wrong reason is worse than no assert.

elway must be invoked as infra-ops@10.250.50.54 rather than the ana-ml2
ssh-target, which resolves to lkraven and has no NOPASSWD sudo.
 2026-08-24 18:39:13 -07:00
vh c507db9ac0 docs(gemma4-erp-tune): size the run against the checkpoint — QLoRA is structurally unavailable 
The proposed shape was QLoRA r64. It cannot be run as specified. The
checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter
tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj
[128,2816,704] — no .weight suffix, so they are parameters, not modules).
bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params /
42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit
saves ~3.1 GiB of 48.07 and does not error while doing it.

Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per
additional 8192-token sequence.

Two sizing items were absent from the brief and both are load-bearing:

- vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping
  30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at
  ~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a
  95.6 GiB card — it starts, then OOMs on the first long sample. Fused or
  chunked linear CE is mandatory and must be smoke-proven before a window
  is booked, since Liger may not carry a Gemma-4 MoE patch.
- v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the
  full-attention layers). A v_proj target silently produces no adapter
  there, and k_proj adapts K and V simultaneously. 45.96M trainable at
  r64 across q/k/v/o.

Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and
gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on
GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr.
Recommend standing down sec (2 aliases, last request ~5h ago) rather than
gen (7 aliases, 765 busy-engine log lines in 24h).

Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours
at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at
effective batch 16, which makes the wall-clock-checkpointing amendment
concrete rather than hypothetical.

Package as a uv venv on /tank: root is 91% full (36 GB) with
/var/lib/docker on it.
 2026-08-24 18:28:30 -07:00
vh 9d0e628643 memory: correct the vLLM version claim — ana-ml2 runs a spread, and 0.27.1 is on disk 
The snapshot recorded "ana-ml2 now runs vLLM 0.26.0". That is true of the
char-rp seat's pin and false of the box, which the operator caught immediately.

Measured per running container: gen is on nightly-311b3513 reporting
0.27.2rc1.dev150, mog-sec on nightly-e9d1398d reporting 0.26.1rc1.dev1102, and
rerank-a3 / coder / reward / embed still on 0.24.0. char-rp and the trainee
bench stack are pinned to v0.26.0. So there is no single "the version" for this
host, and stating one invites exactly the wrong retest.

The correction improves the LoRA question rather than complicating it:
vllm/vllm-openai:v0.27.1 is already on disk and unused — a TAGGED release, not
a nightly, roughly four months past the 0.24.0 where the silent-no-op was
diagnosed. That is the right target for a decision test: no nightly variance,
no pull. The retest instruction in both the decision entry and the handoff now
names it.
 2026-08-24 18:12:35 -07:00
vh 668e590e7d memory: snapshot — char-rp on the Gemma-4 MoE, abliterated trainee staged, QLoRA sizing next 
Captures an evening that ran from an OOM crash-loop to a measured trainee base.

The durable lessons, none of which CLAUDE.md can carry: --gpu-memory-utilization
sizes the KV cache and does not cover CUDA context or graphs, which is half of
why a seat that fit on the 21st stopped fitting on the 24th; the other half is
that gen's footprint GROWS WITH UPTIME (38.5 GiB fresh against 45.6 GiB after
three days, same container, same flag), so headroom arithmetic against a
long-running gen measures a moving number. The stale-chat-template trap turned
out to be endemic across third-party Gemma-4 derivatives rather than a one-off,
and it is silent in both directions — wrong prompt when serving, train/serve
skew when tuning. And a benchmark finding was retracted because 12% on a
five-option task is below the 20% chance floor: a below-chance score indicts the
instrument before the model, and a preflight can be thorough while aimed in the
wrong direction.

Records the serving decision for the tuned model with its history intact:
LoRA-on-NVFP4 is preferred if it works, merged weights the expected fallback,
but the archived root-cause says the objection was never NVFP4-specific — vLLM
0.24.0's LoRA application was a silent no-op proven quant-agnostic, and ana-ml2
now runs 0.26.0. Retest before designing around merge; the answer changes what
Eitri's harness must emit, and he is still early.

Auto-archival moved 5 entries (Recent decisions) to archival-memory.md; the
guards held back the rest of the 78 age-eligible candidates because their bodies
carry open deferred-work language, per the keep-when-unsure rule. Index sits at
286 lines, above the ~250 target and reported rather than forced.
 2026-08-24 15:54:30 -07:00
vh 5415fd4b30 docs(gemma4-charrp): abliteration measured in isolation — close to free, but it MOVES capability 
Second bench window, operator-authorised after an initial decline and reversal.
Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned
upstream template, same 192 items, CoT off. Abliteration was the only axis that
moved, which is what the previous run could not claim.

Net core cost is 0.6 points — but the headline understates what happened.
Capability MOVED rather than degraded: five items lost on contradiction
detection, four gained on spatial composition, nearly cancelling. A gain was not
predicted by anyone, least of all on that axis.

The decision this was authorised to settle: llmfan46 stands as the trainee base.
No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap
between the builds is smaller than the gap this measurement failed to find.

Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are
~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than
the measurement supports and only marginals were run; and this says nothing
about quantization, because the stock-NVFP4 T2 figure came from n=16 against
n=32 here — different item counts mean different item sets, so that comparison
is n-confounded and is not being made.

Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench
stack already existed — itself the residue of debugging a 35-restart crash-loop
caused by the production compose hardcoding --quantization compressed-tensors.
The fix outlasted the incident.

gen restored and verified through the gateway; char-rp remains down deliberately;
bench stack env reset to the heretic base for the post-tune gate.
 2026-08-24 15:39:43 -07:00
vh 019ccff7e8 feat(gemma4-trainee-bench): BF16 bench stack; record that gen's footprint grows with uptime 
Adds an ephemeral stack for serving the BF16 trainee base on :8016 under the
char-rp aliases, so the abliterated base can be measured on the same battery and
the same gateway routes as the served seat with no harness edit.

It is a separate stack rather than another variable on gemma4-charrp because
that compose hardcodes `--quantization compressed-tensors` for the NVFP4 build.
Pointing it at unquantized BF16 weights crash-loops immediately —
`TypeError: CompressedTensorsConfig.__init__() missing 3 required positional
arguments: 'target_scheme_map', 'ignore', 'quant_format'` — vLLM trying to read
a quantization config out of a checkpoint that has none. 35 restarts before it
was caught. `restart: "no"` here so a bench seat cannot resurrect itself and
block gen's restore, and no homepage labels so it leaves no permanently-offline
dashboard card.

It cannot coexist with gen and says so: 48.07 GiB of BF16 weights plus gen's
footprint exceeds the 94.97 GiB card before any KV cache. Running it means gen
is stopped.

THE MORE USEFUL FINDING is in the meromero env note: gen's memory footprint
GROWS WITH UPTIME. Measured today at 46,726 MiB (45.6 GiB) after ~3 days up, and
39,424 MiB (38.5 GiB) immediately after a restart — same container, same
--gpu-memory-utilization 0.43, ~7 GiB apart. That is the missing half of this
afternoon's crash-loop: the char-rp seat "fit on the 21st and stopped fitting on
the 24th" because nothing about char-rp changed and gen crept up underneath it.
Headroom arithmetic done against a long-running gen is measuring a moving
number, so the note now says to measure against a freshly-restarted one.

Operator's requested end state reached and verified through the gateway: gen and
summarizer both 200, char-rp down deliberately to hold GPU0 headroom for the
upcoming trainee run, bench seat stopped.
 2026-08-24 14:19:49 -07:00
vh 14ff4a3f57 docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap 
The operator directed that the ERP/RP trainee base be a low-damage abliterated
instruct build rather than the stock checkpoint. Two are now staged under
/tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB
/ 25.8B shape with only transformers_version differing in config:

  gemma4-26b-a4b-it-heretic-bf16       llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100
  gemma4-26b-a4b-it-abliterated-bf16   TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset

"Low damage" was treated as a measurable claim rather than a description: the
field spreads from KL 0.09 to 0.4118 and the table is in the README so the next
choice is made on numbers. Fleet anchor for reading them — our own abliteration
work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so
both staged builds sit inside an already-validated band rather than past it.
huihui-ai is rejected despite its reputation: no published metrics, its own card
calls the method a crude proof-of-concept, it abliterates both thinking and
non-thinking modes, and its parameter count runs ~738M over upstream. The
operator's independent read matched.

The more durable finding is the chat template. NOT ONE third-party Gemma-4
derivative pulled here ships upstream's — three independent repos carry the
identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third
365-line variant, and only the RedHatAI NVFP4 build matches upstream's
6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager
slipping, and it is now recorded as a class rather than as the single incident
that surfaced it during the A16 control staging.

That matters twice over and silently both times: serving a mismatched template
renders a different prompt, which is why production pins it; and training
through `base/chat_template.jinja` means training on a different prompt format
than production serves — train/serve skew with no error, presenting as a tuning
failure. brokkr-smithy-dev has been warned on the training side while the
harness contract is still early enough to amend.
 2026-08-24 13:52:00 -07:00
vh 8d6a9390de docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed 
Supersedes what commit 3446367 recorded. That message stated the A16 control
"settles a question" and quoted 12% contradiction detection against gen's 81%.
The quantization half of it stands; the deficit it was measuring does not exist.

brokkr-smithy-dev retracted the finding after the operator asked to see the
individual items. The task presented two mutually contradicting statements and
asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC.
Neither statement was more the contradicting one, the model had no way to know
which had been inserted, and it consistently named the absolute claim: a
defensible reading that the labelling scored wrong on every single item.

The tell was there and both of us walked past it: 12% on a five-option task is
BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before
it indicts the model. That should be the first reaction to a below-chance
result, not a late one, and it is now written into the README as such.

Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (which on a sound instrument does not shrink
but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and
gen. A second defect surfaced during the fix — all generators shared one RNG, so
rewriting one task reshuffled every task after it.

What survives is real and worth separating out: the A16 control result holds.
Activation precision is close to free on this workload, every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template — were independent of the item defect
and remain load-bearing. On the corrected instrument Gemma leads the very axis
it was suspected of failing (94% against sec's 81% and gen's 50%); its actual
weak axis with thinking off is spatial composition at 69%.

Recorded as a dated superseded-claims table rather than a silent edit, per the
repo's quant-work convention, so notes elsewhere stop misleading people.
 2026-08-24 13:07:58 -07:00
vh 3446367d5e feat(gemma4-charrp): pin the chat template; A16 control run executed and reverted 
The template is now passed explicitly, defaulting to the A4 build's
chat_template.jinja. That is a no-op for what is served — the A4 build ships
that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once
trailing newlines are normalised — and it permanently closes the class of bug
found while staging the control: the A16 build ships a stale 266-line template
against upstream's 390, with the thinking path built differently and no
`thinking` property in its tokenizer_config response_schema. Serving each build
with its own template would have moved a second axis.

The control ran on the operator's greenlight and has been reverted. Seat is back
on the W4A4 build, healthy, RestartCount 0, both aliases verified through the
gateway — char-rp returns content with reasoning_content empty,
char-rp-reasoning returns both.

Result, since it settles a question this repo's config now encodes: activation
precision does NOT explain the contradiction-detection deficit. Contradiction
detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16
— one item — against gen's 81% on identical items. Every other task is identical
across the two builds and the core difference is 2.6 points carried almost
entirely by two single items. brokkr-smithy-dev pre-registered that a null
result would be the robust branch, because a hidden third axis would tend to
create a delta rather than suppress one, so the conclusion survives the residual
doubt neither side could close without a dequantization pass.

The practical upshot for future scheme choices: W4A4 costs less on this workload
than the caution warranted. The caution was still correct to have.

Displaced production for 3.7 seconds of measurement plus two container
recreates. The A16 build and the BF16 tuning base both stay on disk with the
runbook in the stack README, so re-running is a two-minute flip.
 2026-08-24 13:03:19 -07:00
vh 1bd90eaacc docs(gemma4-charrp): stack README — the three model dirs, the A16 control runbook 
The stack had no README and now carries three model directories that look
interchangeable and are not: the BF16 QLoRA base that cannot be served here, the
W4A4 quant that is served, and the W4A16 build that exists solely as an
activation-axis control. Writing down which is which, and why, before someone
"simplifies" the compose to the BF16 path and rediscovers the OOM.

Also captures the A16 control procedure end to end, including the two confounds
found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template the real one ships — and the fact that
overriding the template is safe because the tokenizers are identical. Both sides
have now cross-checked this: brokkr independently diffed every non-quantization
config field of both builds against the upstream BF16 and found only
transformers_version differing. Residual risk recorded rather than hidden:
config identity is not weight identity and nobody has done a dequantization
pass.

The Gemma-4 flags are documented as architecture-level rather than
checkpoint-level, since that is why they survived the seat swap unchanged, and
the enable_thinking:false pin is called out as mandatory rather than stylistic —
without it every plain prose turn lands in reasoning_content with a null content
and every consumer breaks.

Notes the non-termination defect with thinking on (32 of 96 calls truncating at
12k tokens, all 16 constraint items among them, reasoning sound right up to the
point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not
applied to a seat whose production mode is thinking-off.

No live change: the seat is still serving the W4A4 build. Displacing it for the
control run is an operator decision and is still open.
 2026-08-24 12:34:50 -07:00
vh 24e8826219 docs(gemma4-charrp): the A16 control needs a chat-template override, not just a path swap 
Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's
battery found a second axis hiding inside what was supposed to be a
single-variable control.

The A16 build ships a STALE chat template. Verified by hash against the upstream
weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI
A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is
not cosmetic — upstream and A4 open the thinking path with
`{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off
it, while the A16 template has no such set and guards with
`enable_thinking is defined and enable_thinking` instead. tokenizer_config.json
corroborates: A4's response_schema carries a `thinking` property, A16's has only
role and content. That build was quantized from an older revision of the
checkpoint.

Served with its own template, the A16 arm would render a different prompt for
identical messages, and a contradiction-detection delta could be attributed to
activation precision when it was the template. That is the same failure class as
the misnamed-A16 repos — a field nobody validated, believed because the name
looked right — one layer further down, and it would have produced a result that
looked like a finding.

Overriding is safe because the tokenizers agree: vocab identical at 262,144
entries, added_tokens identical, so the same template over the same vocab
renders the same token ids. Everything else pre-flights clean — both artifacts
complete with no missing shards, generation_config.json byte-identical.

Seat NOT flipped; displacing production for the bench window is the operator's
call and is still open.
 2026-08-24 12:32:03 -07:00
vh f509668e45 docs(gemma4-charrp): record the A16 activation-axis control and how to run it 
brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction
detection with CoT off, against gen's 81%, while state tracking, deterministic
constraint following, long-context state and the confabulation control all sat
at 100%. That is not general degradation — it is the shape 4-bit input
activations produce on the most reasoning-dense task, which is exactly the
confound flagged when the W4A4 quant was chosen. They have recorded the finding
as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an
A16 build to separate the two.

No quant run was needed: a genuine A16 build of the same checkpoint already
exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16
(prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations
null, 17 GB). Same weights, same loader as the live seat, one axis moved.

⚠ Two other repos would have answered the question wrongly and the note in the
env template says so: bg-digitalservices and ManniX-ITA both publish
"Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits
4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's
17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the
model.

Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no
room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on
it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom
that crash-looped the predecessor this morning. Port and both aliases are
unchanged either way, so no consumer config moves.
 2026-08-24 12:28:17 -07:00
vh 27155c0f3b feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port 
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with
google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every
gateway route are unchanged, so no consumer sees a difference in addressing:
`char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` ->
hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now
include chain-of-thought, which makes throughput more critical rather than less
— the user waits through the whole reasoning block before the first visible
token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7.

Both artifacts are on disk and they are NOT interchangeable. The BF16 weights
(/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since
QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of
weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the
engine would die at allocation exactly the way the predecessor did this
afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB),
chosen over the other -it quants because it is compressed-tensors
(nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the
llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than
-it, and the thinking channel lives in the instruction-tuned weights.

Smaller weights at the same 0.47 memory budget buy a much larger KV pool:
27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same
budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4.

The gemma4 tool-call parser, reasoning parser and the enable_thinking:false
default all carry over unchanged — they are architecture-level, not
checkpoint-level. The --chat-template override does NOT carry over: MeroMero
pointed at a jinja hand-patched against that checkpoint, and this model ships
its own. Verified that dropping it did not reintroduce the failure that flag
existed to prevent — non-thinking prose lands in content with reasoning_content
empty, and the thinking alias populates reasoning_content with content
carrying the answer.

⚠ Scheme differs from the incumbent and the bench should say so: this quant
declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16.
Faster, and not like-for-like on the activation axis.

meromero-charrp is retained stopped in `created` state and relabelled to
AI - Dormant, per the house rollback pattern. Both stacks want :8016, so
rolling back means stopping the gemma4 seat first.
 2026-08-24 12:02:59 -07:00
vh 850e0c3351 fix(meromero-charrp): drop GPU0 budget to 0.47 — the seat was OOM crash-looping 
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the
time it was looked at, taking both the `char-rp` and `char-rp-reasoning`
gateway aliases down with it (they resolve to the same seat on :8016 —
hosted_vllm/char-rp and hosted_vllm/char-rp-thinking).

Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the
engine gets through weights, torch.compile and CUDA-graph capture looking
entirely healthy, then dies at KV-cache allocation with
`torch.OutOfMemoryError: ... 195.19 MiB is free`.

GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds
~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV
cache and does not account for CUDA context, graphs and non-torch overhead.
This seat was at 0.51, so the pair was committed to 0.94 of the card with about
0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today.

0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 ->
23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the
pool still holds 1.4x a full-length sequence. What is lost is concurrent long
requests, not context.

Verified through the gateway rather than at the container: char-rp returns 200
with content, char-rp-reasoning returns 200 with both content and
reasoning_content populated. Seat is healthy with RestartCount 0.

The arithmetic and the "check used_memory, not the flag" warning are written
into the env template, because the next person to raise either budget needs to
lower the other in the same change.
 2026-08-24 11:42:36 -07:00
vh 35adc4a043 feat(homepage): rebuild on Australis Skyfall — dual theme, light mode shipped 
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.

The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.

build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.

⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.

So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.

Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.

Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
 2026-08-24 09:44:45 -07:00
vh 39da1d4a97 feat(homepage): recategorise on "do I open this?", collapse the API groups 
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.

Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.

A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.

The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.

28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.

The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.

Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.

Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
 2026-08-24 08:54:06 -07:00
vh f6f2f69649 fix(homepage): uniform 4-column grid, hold the status gutter, unleak AI Systems 
The board's card width changed at every group boundary because `columns:` was
being tuned per group under the 2026-08-18 "columns = member count" rule. That
rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that
group's CARD WIDTH, not its density. Measured on the live board, Notes rendered
a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything
else 360px. All 20 groups are now `columns: 4` and every card renders at 360.

`.service-name` reserved a 78px status gutter with `padding-right` and relied
on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the
gutter was spill room the title printed straight through. Six cards on the AI
tab rendered their name underneath their own status pill, measured by testing
the title text node's box against the status cluster's. The intended ellipsis
never fired either: it is painted by whichever block's own line overflows, and
that is the anonymous box around the bare title text node, which does not carry
`overflow`. The gutter holds by wrapping now, and the description opts back out
of it with a negative margin since the pill only ever covers the first line.

Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so
it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox
in 2026-08-18, arriving this time from a container label. Relabelled to
`AI - Audio Tools`, where the other ASR seats already live. The
`homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of
the name and now carries the constraint, plus the fact that a labelled
container is discovered from any of the five configured engines and must not
also be listed in services.yaml.

Also: descriptions clamped to three lines so a five-line outlier stops dragging
its row 50px taller than its neighbours; icon ramp overridden off slate-400 ->
slate-700, which was sinking the bottom half of every glyph into the card fill;
bookmark groups and Jellyfin's trailing stream rows brought into the card and
eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates
them.

The icon override sits on `html[class]`, not `:root`: Homepage sets the same
variables on `.theme-slate`, which is on the <html> element, and a class beats
`:root` on the same element.

Verified with Playwright against the live board — per-group card width, card
height spread, and a geometric title-vs-status collision check, before and
after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/
 2026-08-24 08:15:02 -07:00
vh 32349b7653 memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed 
Rewrites the in-flight section, which had gone stale in one place
(speaches is stopped, not live) and did not reflect a session in which
everything opened was also closed. Three detail files carry the bulk:
the FortiGate's public surface reduced to nothing including the ACME
listener, Scriberr's deployment and the three upstream defects it
required working around, and the ESH DNS repair that also made the
IPv6 naming scheme real on three hosts.

The tunnel entry in the decisions log described a cipher change as
proposed and pending; it has since been attempted, found impossible in
the form intended, and resolved differently, so the line now points at
the outcome rather than the intention.

New decisions cover the cipher adoption and the finding that the
per-flow ceiling belongs to the UniFi gateway's software cipher rather
than the firewall, the administrative closure, the transcription
deployment, the DNS repair, the retirement of an ASR service whose
only consumer was abandoned, and a database VM that until now had no
fleet identity and no vaulted credential at all.

Three approaches are recorded as abandoned: an AEAD cipher the far end
cannot express, an upstream DNS setting the gateway accepts and
ignores, and a scheme to claim unused delegated prefixes that founders
not on the prefixes but on having to rebuild a firewall policy to use
them.
 2026-08-24 07:42:30 -07:00
vh d419b11d43 docs(ipv6): close the NH3 multi-prefix question by operator ruling 
The gateway has no IP-passthrough mode, confirmed by the operator with
admin access to it, so the inexpensive path of letting the UDM take the
delegation whole and carve it natively is unavailable at this site.

What remains is a separate DHCPv6 client presenting several identities,
which needs recabling to reach the gateway, splits routing so that one
device handles v4 and another v6, and above all moves IPv6 off the
UDM's zone firewall, leaving the entire policy to be rebuilt elsewhere
before any host could safely hold a globally routable address. The
operator has declined it and the LANs stay without v6.

The index entry also still asserted the superseded single-delegation
conclusion as fact; it now carries the corrected reading alongside the
ruling, so the finding and the decision not to act on it arrive
together rather than the first inviting a retry of the second.
 2026-08-24 07:05:49 -07:00
vh bd209951ac docs(ipv6): correct the NH3 delegation finding — eight /64s, not one 
The note concluded AT&T delegates a single /64 and that reading the
address pattern as a /60 was a mistake. A later finding in the same
session overturned that by reading the BGW's own statistics page, and
that correction was never written down; it survived only in the
transcript and surfaced again while assessing whether more prefixes
could be claimed.

The gateway holds the /60 and rations it, keeping half for itself and
delegating the top eight /64s one at a time. Both observations agree:
a prefix-id only carves within a delegation already held, so the
earlier test could not have moved a lone /64 regardless. The limit is
that UniFi solicits once, which makes the requester the ceiling rather
than the carrier.
 2026-08-24 07:02:53 -07:00
vh d127e29fac docs(ipv6): retire the next-candidates line now that all three are done 
Replaces it with why the remaining segments have no eligible hosts:
two are appliance-only and three have no IPv6 enabled yet, pending the
firewall-policy pass that SLAAC on a client segment would require.
 2026-08-23 22:39:51 -07:00
vh 4e83395ddf docs(ipv6): all three esh-server hosts now carry the segment name 
esh-pve-nas and esh-vm-db join esh-docker-vm on 4411:b105, at
:50:55 and :50:60 respectively, each applied by the same prefix-deriving
if-up.d hook so the last two groups read straight off the IPv4 address.

Two obstacles are recorded because both will recur. The Proxmox node had
link-local only despite every relevant sysctl appearing correct, because
its bridge carries per-interface forwarding and the kernel ignores router
advertisements on a forwarding interface unless accept_ra is explicitly
two rather than one. The fix takes the advertised prefix while declining
the default route, so the hypervisor gains an address without any change
to how it routes; this was verified after applying, with the v6 default
route count still at zero.

The database VM refuses key authentication for the privileged accounts
and its unprivileged login cannot escalate without a password, so the
hook went in through the QEMU guest agent from the hypervisor, which
executes as root inside the guest. The document notes the base64
indirection needed to get a multi-line script through intact.
 2026-08-23 22:39:39 -07:00
vh ffb7fba346 docs: give the ESH IPv6 naming scheme a home, and make it real on one host 
The scheme has existed since August as a single line of persistent
memory, which a snapshot then deleted. It is a naming convention
rather than temporal state, so it now lives in docs/pfi as a proper
document, and the memory entry is reduced to a pointer at it. The
document carries the full table, the address structure, the reasoning
about which slots can and cannot hold a name, and the recipe for
applying one to a host.

It also corrects the conclusion the original note ended on. That note
held that these names could never appear on the wire, which is true
of everything UniFi is able to assign but not of what a host can
assign to itself, and the distinction is the whole difference between
a joke and an address.

AdGuard on esh-docker-vm now holds the esh-server name, at
2607:73c0:402:1d02:4411:b105:50:45, where the segment identity and the
IPv4 address are both legible. It is applied by an if-up.d hook that
derives the prefix at runtime rather than hardcoding it, backgrounds
itself with a retry so it cannot stall interface bring-up, and adds
nothing to the existing interface configuration.

This is load-bearing rather than decorative. The gateway advertises an
IPv6 resolver to clients, macOS prefers it over the IPv4 one, and it
previously pointed at an address derived from that host's MAC.
 2026-08-23 22:31:13 -07:00
vh 41091eef8f docs: restore the ESH IPv6 naming scheme, dropped by a snapshot 
The six ESH LAN hexspeak names were recorded in fe3d765, refined in
959bb6e and 8be8a51, and then removed without comment by the memory
snapshot in 837fa36. Nothing referenced them afterwards, so the loss
went unnoticed until the operator asked for them tonight and no file
on the machine contained the string.

Recovering it was harder than it should have been, for a reason worth
recording: the table was written in uppercase and git log -S is
case-sensitive, so a history search that would otherwise have found it
immediately came back empty. The entry was eventually located in a
session transcript, which named the commit.

The restored text carries the original verbatim plus two additions. It
is marked as a naming convention rather than temporal state, since
that is what made it eligible for pruning in the first place. And its
conclusion that the names can never appear on the wire is corrected:
the reasoning holds for what UniFi can assign, but a Linux host can
take such an address itself, which was verified on esh-docker-vm along
with a persistence path that leaves the existing interface config and
SLAAC untouched.

The scheme's first practical use is noted against the resolver address
the ESH gateway now advertises, which currently depends on a MAC.
 2026-08-23 22:26:39 -07:00
vh 6217d3993e fix(scriberr): force uv to copy rather than reflink when building envs 
Scriberr builds each model backend's Python environment with uv at
container start. uv's default link mode reflinks or hardlinks out of
its cache, which fails on this overlayfs over ZFS combination and
reports it as a failure to clone a metadata file with errno 11,
resource temporarily unavailable. The wording points nowhere near the
cause.

The damage was partial and therefore easy to miss: WhisperX and
PyAnnote came up fine and the application served normally, while the
Parakeet and Sortformer backends were quietly absent. Setting
UV_LINK_MODE to copy trades a little disk and build time for
environments that actually materialise, and the occurrence count for
that error is zero on the following start.

The knob is exposed through the environment so it can be moved back to
the default if a future host does support reflinks.
 2026-08-23 19:31:54 -07:00
vh efddb4e511 feat(scriberr): stand up transcription on ana-ml2, pinned to GPU1 
Scriberr transcribes audio and video locally with WhisperX and
speaker diarization, and it lands on ana-ml2 rather than ana-docker
because the work is GPU-shaped: ana-docker offers eight cores already
shared with fifty containers and thirty-seven gigabytes of disk,
against ninety-six cores, terabytes on /tank and idle capacity on
GPU1. The reservation names device 1 explicitly, since GPU0 is fully
committed to the gen seat, and the container is confirmed to see that
card alone.

The image is built from source, which is not a preference. These are
Blackwell cards at sm_120; the published CUDA image covers Pascal
through Ada only, and the blackwell image the upstream README
documents has never been published at all. The path upstream actually
ships for sm_120 is Dockerfile.cuda.12.9, carrying CUDA 12.9 and cu128
torch, so that is what gets built. The compose header says so, because
the obvious cleanup is to swap in the published image and that would
silently drop the deployment to CPU.

Two configuration details are load-bearing and documented where
someone would go to change them. The application runs as uid 10001
rather than the usual 1000: that Dockerfile moves its user aside for
Ubuntu 24.04's own uid-1000 account and chowns /app accordingly, while
the entrypoint's remapping covers only the data directories, so at
1000 the process cannot open its database and restarts forever behind
a SQLite error that reads as though the machine were out of memory.
Secure cookies stay off while the service is reached over plain HTTP,
or sessions are dropped by the browser and login appears to loop for
no visible reason.

Storage is bind-mounted onto /tank because model weights run to
several gigabytes and the root pool on that host is nearly full.

Also adds the scriberr service alias to internal DNS, following the
existing alias convention so consumers name the service rather than
the box.
 2026-08-23 19:28:33 -07:00
vh 22ae9cd480 ops(ana-gw): disable ACME and retract the all-port VIP claim 
The ACME client is unbound from wan1, which removes the HTTP-01
challenge listener that had been holding port 80 open irrespective of
allowaccess. An external sweep of fifty-five ports against the WAN
address now finds nothing open at all, while the internal interface
still serves the GUI and SSH and retains its certificate, which
remains valid until late October and simply stops renewing.

The previous note's claim that four virtual IPs were unrestricted
all-port static NATs is withdrawn. A FortiOS virtual IP can be scoped
either by an explicit forwarded port or by a service binding on the
object, and the earlier parse inspected only the former. All four use
the latter, and the custom services behind them are narrow. None of
the fourteen is unrestricted.

Ground truth from outside is recorded in place of further config
reading, listing what each public address actually exposes. Three
configured mappings answer nothing at all and are noted as tidy-up
candidates for the migration rather than as exposure.
 2026-08-23 16:02:20 -07:00
vh f39b66d2e1 ops(ana-gw): port 80 on the WAN address is the FortiOS ACME listener 
The previous note attributed the open port to an ISP transparent
proxy. That was wrong. It terminates on the FortiGate: system acme is
bound to wan1, and FortiOS opens port 80 there to answer HTTP-01
challenges regardless of what allowaccess permits, which is why the
port stays open with the interface set to ping only. Every
non-challenge request returns a fixed 403 whose body reads ACME Access
Only. No DNAT is involved; of the fourteen virtual IPs only two land
on that address, neither on port 80.

The wrong conclusion came from a sniffer filtered on dst host, which
matches inbound packets alone and so excluded the replies being looked
for. Filtered bidirectionally the box is plainly seen emitting SYN-ACK
on port 80. The note records the rule.

Two consequences follow. The earlier warning that certificate renewal
would fail without http in allowaccess is retracted, since FortiOS
opens the challenge port itself. And the listener is not an
administrative surface, though its value is now marginal with WAN
administration closed, so the note records how to remove it and leaves
that decision open.

Also captured: four virtual IPs are all-port static NAT and map every
port of their external address.
 2026-08-23 15:53:55 -07:00
vh 7bc9754e40 ops: adopt AES-128 on both Anaheim tunnels and close public admin 
Both tunnels now negotiate AES-128 for ESP, applied make-before-break
so neither dropped waiting on its far end: the FortiGate was widened
to accept the new cipher alongside the old one first, then each UniFi
gateway was flipped. Single-stream throughput moves from 245 to 270
on the NH3 tunnel and from 268 to 304 on the ESH tunnel. Both network
objects were diffed against pre-change snapshots and the only field
that moved on either is the ESP cipher.

The proposal lists are left accepting AES-256 as well. The peers offer
only AES-128 so the extra entries are inert, and retaining them means
a gateway reverting cannot strand a tunnel.

With that up, the WAN administrative surfaces are closed. The
interface is back to permitting only ping, and the infra-ops account
is again restricted to RFC1918 space. Ports 443 and 22 were confirmed
closed from two separate sites and management over the tunnel still
works. The close was issued over the tunnel rather than over the WAN,
since withdrawing SSH from the interface while connected through it
would sever the session mid-command. The box now has no out-of-band
path, which the memory records explicitly.

Also captured: the two UniFi vault items have different shapes, one a
bare key and one a documentation note requiring extraction, which
produces an opaque nginx rejection if missed, and the ESH key's first
confirmed write.
 2026-08-23 15:44:32 -07:00
vh 6edebe4864 ops: forwarding through a downstream WireGuard terminator is free 
FortiOS has no WireGuard, so any WireGuard site-to-site has to
terminate behind the edge, and ana-wg already fills that role. The
earlier 767 Mbit/s figure was taken with traffic terminating on that
box, which left the forwarding case unmeasured and overstated what a
real deployment would see.

Measured properly, transit from NH3 through the tunnel and onward to
ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight,
so the forwarding hop costs nothing and the full threefold gain over
IPsec survives. The container uses around a fifth of four cores at
that rate, against the UniFi gateway spending a third of its own four
to move a third as much.

The note records what still needs deciding rather than presenting this
as ready to build: Anaheim hosts have to route to the terminator
directly or pay a hairpin through the edge, which was not measured;
the terminator introduces a failure mode the edge did not have, with a
fallback route as the mitigation; and the NH3 end needs its own
terminator, where a Linux host matches the measured figure while the
existing UniFi WireGuard server would likely land lower.

The test tunnel, its peer and the temporary route were all removed.
 2026-08-23 15:25:05 -07:00
vh 062215e81a ops: the tunnel's per-flow limit is the UDM's software AES-CBC 
Varying only the cipher settles what inference could not. AES-128 has
ten rounds against AES-256's fourteen, so a software-bound path must
speed up when the cipher gets cheaper while a hardware-offloaded one
will not move. Run as A/B/A on a single stream, AES-256 returns 232,
245 and 243 Mbit/s and AES-128 returns 282 and 275, a gain of about a
fifth at identical CPU. The FortiGate's offload engine is therefore
not the constraint, and the operator's reading of the UniFi side as
software crypto without acceleration is correct.

Two earlier claims of mine were wrong and are corrected in place. The
observation that the UDM sat at seven percent CPU was a sampling
artifact, since UniFi refreshes that statistic on the device report
interval and the windows used were four seconds; a sustained run reads
thirty-five percent, around one and a half of four cores. The
per-session offload hypothesis is refuted rather than merely unproven,
because it predicts no response to a cipher change.

The remaining figures follow from this. One stream is one core, eight
streams engage about three, and aggregate stops responding to cipher
choice once several cores are working because the path bounds it. CBC
chaining is what makes this expensive, as it serialises blocks and
prevents the AES instructions from pipelining, which is also why the
same gateway manages far more over WireGuard.

The cipher was restored and the network object verified unchanged
against its pre-test snapshot.
 2026-08-23 15:11:51 -07:00
vh 8ecffa1fab ops: WireGuard over the same path carries one stream at 767 Mbit/s 
The operator asked for a WireGuard test between ana-wg and NH3 across
the public internet, which is the arrangement that separates the path
from the crypto since neither gateway performs encryption in it. A
single stream returns 767 Mbit/s against 245 for IPsec over the same
wire, and eight streams return 763, so one flow already saturates the
path and there is no per-flow penalty whatsoever.

That settles several things at once. The limit is not the circuit,
the NH3 uplink, the ISP or the physical path, all of which carry 767
on one flow. Ranking the implementations, Linux WireGuard shows no
per-flow penalty, UniFi's WireGuard shows roughly 1.4x, and IPsec on
this pair shows 2.8x. Latency under a single bulk stream tells the
same story, rising to 12.7 ms on WireGuard against 102 ms on IPsec.

This overturns the earlier conclusion that changing transport was not
worth pursuing, which compared eight-stream figures and so understated
the gap for single-stream work by a factor of three. A WireGuard
site-to-site terminated on ana-wg now looks worth considering, and the
note records the open questions around failover and policy.

Attribution between the FortiGate and the UniFi gateway remains
unresolved, since both perform IPsec in the slow measurements, but the
remedy does not depend on which one is responsible.

The test tunnel was removed, ana-wg is back to its original three
peers and the generated keys were shredded.
 2026-08-23 14:57:29 -07:00
vh d42e9d8712 ops(ana-gw): establish the per-flow cap is IPsec-specific, not capacity 
The operator asked whether the 80F is simply out of capacity. It is
not. Routing a single flow between two Anaheim VLANs through the same
box, with no tunnel involved, sustains 940 Mbit/s, which saturates the
link, and eight flows over that path return the same figure. There is
therefore no per-session ceiling in the plain forwarding path and the
roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone.

Measuring the second tunnel with the same probe supports this. The ESH
side, terminating on different gateway hardware at nearly half the
round-trip time, returns 268 Mbit/s on one stream against NH3's 245,
and 715 against 692 across eight. A window-limited path would have
been substantially faster at the shorter round-trip, so the agreement
between two dissimilar far ends is further evidence of a rate cap.

This also corrects the previous note, which named the FortiGate on
reasoning that could not separate the box from the protocol, since
every slow path was both. That separation is still not established:
both tunnels terminate on UniFi gateways running common firmware, so
the cap could belong to either side. The note records the tunnel
topology that would decide it and flags the bearing on the pending
replacement.
 2026-08-23 14:48:34 -07:00
vh cf0cb2cbb3 ops(ana-gw): pin down what limits a single stream across the tunnel 
The earlier note described the constraint as per-flow serialisation
without evidence. Pinning SO_SNDBUF and sweeping it shows what it
actually is: throughput holds flat between 224 and 247 Mbit/s across
a thirteenfold range of in-flight data while round-trip time scales
with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions
are absent throughout. That is a fixed service rate with a standing
queue ahead of it, so socket buffer, window scaling and congestion
control are not the lever and should not be pursued.

The same measurements surface something with wider consequences than
throughput: a single bulk stream lifts tunnel latency from 6.9 ms to
102 ms average, 136 ms peak, with no loss. Interactive traffic
sharing the Anaheim link degrades sharply whenever anything moves
bulk data, and parallelising transfers makes that worse rather than
better.

Localisation rests on the FortiGate appearing in both slow paths and
in neither fast one, with aggregate throughput over the same security
association reaching 692 Mbit/s. Per-session NPU offload fits the
shape but was not confirmed on the device; the note records the test
that would settle it and flags the bearing this has on the pending
FortiGate replacement.
 2026-08-23 14:42:17 -07:00
vh e41d19f1cb ops(ana-gw): close out the Anaheim tunnel cipher question 
The AES-GCM cutover was authorised and attempted, NH3 side first. It
cannot be completed: UniFi's manual site-to-site IPsec implements no
AEAD cipher. Eight GCM spellings were rejected with
api.err.InvalidPayload while an otherwise identical body carrying
aes256 returned rc:ok, which isolates the enum rather than the
request as the cause. The accepted set is aes128/aes192/aes256/3des.
Both Anaheim tunnels terminate on UniFi gateways, so this blocks the
ESH tunnel on the same grounds.

Measuring while testing also retires the premise. NH3's uplink is a
1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant
ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather
than the ~550 previously recorded from a 4-stream run. Compared with
WireGuard over the same gateway and uplink, the deficit narrows from
2.3x at one stream to 15% at eight, so moving this link onto a
different transport is not worth doing.

The constraint is per-stream, around 245 Mbit/s, with both endpoints
idle under load. Parallelising bulk transfers remains the mitigation
and is worth 2.8x at no cost; NFS nconnect is the equivalent lever
for single-stream mounts such as /mnt/smithy on ana-ml2.

FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list,
which is inert while the peer offers only CBC and avoids a further
renegotiation. The UDM network object was diffed field-by-field
against its pre-change snapshot and is unchanged.
 2026-08-23 14:29:17 -07:00
vh 5af362e9d0 ops(ana-gw): restore WAN admin access ahead of the FortiGate cutover 
Re-open the ana-gw admin GUI on wan1 so the Anaheim edge can be
managed remotely if the cutover goes wrong, reversing part of the
2026-08-12 lockdown. Two config changes, nothing else (verified by
diffing pre/post `show full-configuration`):

- wan1 `allowaccess ping https` — https only; http, ssh, and fgfm
  stay off, and wan2 is untouched.
- `infra-ops` trusthost widened to all routable IPv4; the `admin`
  account stays locked to 10.0.0.0/8 so the guessable username
  remains unreachable from the internet.

Verified end-to-end from two sites: a real `/logincheck` POST returns
AUTH OK over the public path, on a browser-trusted Let's Encrypt cert
for ana-fw.phasefinal.com valid through 2026-10-27.

Two FortiOS behaviours worth recording, both of which cost time here:
a trusthost whose base address is 0.0.0.0 is silently treated as
unset (so there is no writable "any" — only decomposed ranges), and
trusthost is enforced before the TCP handshake, so a blocked source
sees a filtered port rather than a refused login.

Follow-ons captured in memory, not actioned: ACME renewal for the
admin cert needs port 80 on wan1 (next attempt ~2026-09-27), and a
~5 SYN/s source in 179.51.184.0/21 now draws SYN-ACKs at no
measurable CPU cost.
 2026-08-23 14:03:40 -07:00
vh b6340519bc memory: snapshot — Anaheim tunnel at 25% of circuit; selene retired; hrafn CI fixed 
Session captured for a context reset. Six new detail entries.

THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a
circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not
CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels
negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization;
execution pending, untracked by operator choice.

Also recorded: selene retired after losing a head-to-head on its own job with
chat-judge moved to gen and the model name left to 404; the 7-alias collision
on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and
its CI found to have been reporting green while deploying nothing for its whole
life; all three Worldtree instances de-armed from a 69-day-stale :latest and
the Matrix homeserver re-plumbed to personal; every secret-bearing .env on
ana-docker tightened to 0600; the pfi org closing the repo-creation half of the
credential-migration directive; booth kept-board deletion and link pruning.

Two entries under Tried and abandoned: the CI checkout assertion that broke the
pipeline twice and was removed, and my proposal to alias a retired model name
at a different model, which the operator correctly overruled.

Index 271 -> 282 lines, under the 300 cap, so no archival fired.
 2026-08-23 13:35:48 -07:00
69 changed files with 8026 additions and 1009 deletions
Expand all files Collapse all files
CLAUDE.md
+36 -2
View File
@@ -62,6 +62,23 @@ repeats the playbook, you are re-litigating — record the delta in the playbook
instead. When a playbook claim turns out wrong, don't just fix it: add a dated
row to its superseded-claims table so old docs stop misleading people.
## Training throughput
Same contract as quantization, different subject: **`docs/pfi/training-throughput-playbook.md`
is the durable home** for why a training run is slow — the 10-minute scaling
triage that names the regime before you profile, the padding/masking landmines,
the profiler traps, and its own superseded-claims table. Read it before
hypothesising about kernels.
The instruments are committed at [`scripts/training-probes/`](scripts/training-probes/)
with raw output kept alongside, so the claims can be re-derived rather than
taken on faith.
⚠ **Measure before you argue.** The playbook exists because a four-model
frontier panel produced four self-retractions in ninety minutes on this
question, and every one of them was a derivation while every survivor was a
measurement.
## Purpose
- Inventory of servers and their state
@@ -98,15 +115,32 @@ Observed and standardized across servers:
- **Named volumes** for service state (pattern: `<stack>_<name>`)
- **Bind mounts** only for: model files (`/tank/aimodels/...`), config files (`/opt/docker/conf/...`), docker socket where required
- **Restart policy:** `restart: unless-stopped` for daemons
- **Homepage labels** on user-facing services:
- **Homepage labels** on user-facing services. The dashboard runs on
`esh-docker-vm` and reads the Docker API of **every** host in
`stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker,
irv-ml1, esh-docker-vm), so a labelled container is discovered from
wherever it runs — you do not add it to `services.yaml` as well. Doing both
renders it twice.
```yaml
labels:
- homepage.group=AI Systems
- homepage.group=<ExistingGroup>
- homepage.name=<ServiceName>
- homepage.icon=mdi-<icon>
- homepage.description=<short>
- homepage.href=http://<host-ip>:<port>
```
⚠ **`homepage.group` must name a group that already exists in
`stacks/homepage/conf/settings.yaml`'s `layout:` block.** A group the layout
has never heard of gets no `tab:`, and Homepage renders an untabbed group on
**all four tabs**. Inventing a group name here is how Scriberr's
`AI Systems` ended up repeated at the bottom of every tab from 2026-08-23
(fixed 2026-08-24). If the service genuinely needs a new group, add the group
to `layout:` **with a `tab:`** in the same change.
Check with `curl -s http://10.0.50.45:5100/api/services | jq -r '.[].name'` —
anything in that list that is not a key in `layout:` is leaking onto all
tabs right now.
Labels only apply at container **creation**, so a label edit needs
`docker compose up -d <service>`, not `restart`.
- **Healthchecks** on services that expose HTTP
## Servers
archival-memory.md
+15
View File
@@ -736,6 +736,21 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
_Archived 2026-07-13._
- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
_Archived 2026-08-24._
- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
_Archived 2026-08-24._
- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]]
_Archived 2026-08-24._
- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]]
_Archived 2026-08-24._
- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
_Archived 2026-08-24._
## Tried and abandoned (archived)
- `[2026-04-30]` task-board workflow with
dns/internal.yaml
+1
View File
@@ -108,3 +108,4 @@ aliases:
- {name: gateway, site: ana, target: ana-docker, note: LiteLLM gateway :4000}
- {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090}
- {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100}
- {name: scriberr, site: ana, target: ana-ml2, note: transcription + diarization :8080 (GPU1)}
docs/pfi/gemma4-erp-tune-sizing.md
+537
View File
@@ -0,0 +1,537 @@
# Gemma-4 26B-A4B ERP/RP tune — GPU sizing adjudication
_Measured 2026-08-24 on `ana-ml2` against
`/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated trainee)._
Division of labour for this run: **Eitri writes the harness, brokkr-smithy-dev
audits, infra-ops owns the GPU window and executes.** This document is the
sizing infra-ops owes; it is arithmetic against the real checkpoint and the
real card, not an estimate.
---
## 1. ⚠ QLoRA IS NOT AVAILABLE ON THIS ARCHITECTURE
**The proposed shape was QLoRA r64. It cannot be run as specified**, and the
reason is structural rather than a tuning preference.
The checkpoint stores each layer's 128 experts as **two fused 3-D
`nn.Parameter` tensors**, not as 128 `nn.Linear` modules:
model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816]
model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704]
Note the absence of a `.weight` suffix — compare `mlp.down_proj.weight`
(an `nn.Linear`) against `experts.down_proj` (a bare parameter). That is the
tell, and it is decisive: **`bitsandbytes` 4-bit replacement walks `nn.Linear`
modules.** A fused 3-D parameter is not one, so it is skipped and stays BF16.
What `load_in_4bit=True` would actually buy on this model:
| block | params | BF16 | after bnb NF4 | saved |
|---|---:|---:|---:|---:|
| **MoE experts** (fused 3-D — **NOT quantized**) | 22.84 B | 42.54 GiB | **42.54 GiB** | **0** |
| lm attention (`nn.Linear`) | 1.11 B | 2.07 GiB | 0.52 GiB | 1.55 |
| dense shared MLP (`nn.Linear`) | 0.54 B | 1.00 GiB | 0.25 GiB | 0.75 |
| vision tower (`nn.Linear`) | 0.57 B | 1.06 GiB | 0.27 GiB | 0.79 |
| embed (tied, normally kept BF16) | 0.74 B | 1.38 GiB | 1.38 GiB | 0 |
| router + norms | 0.01 B | 0.02 GiB | 0.02 GiB | 0 |
| **total** | **25.81 B** | **48.07 GiB** | **~44.98 GiB** | **~3.1 GiB** |
**88.5% of the model is in tensors bitsandbytes cannot touch.** "QLoRA" here
means paying the NF4 dequant tax on 6% of the weights to save 6% of the
footprint. The premise does not survive contact with the checkpoint.
> **Eitri: do not hard-code a `BitsAndBytesConfig` / `load_in_4bit` path.**
> It will not error loudly — it will load, report a 4-bit model, and quietly
> leave 42.5 GiB in BF16. Same silent-failure shape as the stale chat template.
**The one thing that could overturn this** is a third-party fork shipping
custom grouped-GEMM 4-bit MoE kernels for this specific architecture (Unsloth
is the candidate). **Not chased, deliberately** — see §4, where the run fits in
BF16 without displacing anything the fleet depends on, which collapses QLoRA's
value to zero. If it is ever revisited, it must be *before* the harness
hard-codes a quantization path, not after.
**Verdict: plain LoRA on BF16 weights.**
---
## 2. What the run actually costs
Adapter targeting `q_proj,k_proj,v_proj,o_proj` at r64, computed from the real
tensor shapes:
| | layers | per layer | total |
|---|---:|---:|---:|
| sliding-attention (q 4096, kv 2048, o 4096) | 25 | 1,507,328 | 37,683,200 |
| full-attention (q 8192, kv 1024, o 8192) | 5 | 1,654,784 | 8,273,920 |
| **trainable** | | | **45,957,120** (0.178% of base) |
⚠ **`v_proj` DOES NOT EXIST ON LAYERS 5, 11, 17, 23, 29.** Those are the
`full_attention` layers, and `attention_k_eq_v: true` means one projection
serves both K and V. Consequences the harness must respect:
- PEFT matches by name suffix, so a `v_proj` target **silently produces no
adapter** on those five layers. Do not assert a fixed adapter count.
- Adapting `k_proj` on a global layer **adapts K and V simultaneously** — a
different intervention than on the sliding layers. If that asymmetry matters
to the recipe, say so explicitly rather than discovering it in the loss curve.
### Memory budget, batch 1, `max_seq_len` 8192
| item | GiB | note |
|---|---:|---|
| base weights BF16 | 48.07 | measured: 25,805,936,206 params × 2 B |
| adapters + grads + AdamW fp32 m/v | 0.75 | 45.96 M trainable — rounding error |
| checkpointed layer inputs | 1.29 | 30 × 8192 × 2816 × 2 B |
| recompute peak, one layer | ~2.5 | 8192 tok × top-8 of 128, `moe_intermediate 704` |
| loss head, **fused/chunked CE** | ~2.0 | see the warning below |
| CUDA context + cuBLAS + fragmentation | ~3.0 | the item `--gpu-memory-utilization` never covered |
| **total** | **~57.6** | |
Marginal cost per extra sequence in the micro-batch: **~2.5 GiB.**
| micro-batch | GiB |
|---:|---:|
| 1 | 54.3 |
| 2 | 56.8 |
| **4** | **61.8** |
| 6 | 66.8 |
| 8 | 71.8 |
### ⚠ The loss head is the whole ballgame, and it is not in the brief
`vocab_size` is **262,144** and `final_logit_softcapping` is **30.0**. One
8192-token sequence produces **2.147 billion logits**. Through a naive HF
`ForCausalLM` loss that is:
BF16 logits 4.0 GiB
fp32 upcast 8.0 GiB
softcap tanh saved 8.0 GiB (autograd keeps the pre-cap tensor)
softmax + grad 8.0 GiB
------------------------------
~28-30 GiB transient, at BATCH 1
Naive CE at batch 1 lands the run at **~85.6 GiB on a 95.6 GiB card** — it will
appear to work and then OOM on the first long sample. At micro-batch 4 it is
~120 GiB and never starts. **Fused/chunked linear cross-entropy is mandatory,
not an optimization.**
⚠ Honest uncertainty: Liger ships per-architecture patches and Gemma-4 MoE with
softcapping may not have one. Three ways out, in order of preference —
(a) generic `LigerFusedLinearCrossEntropyLoss` wired against the lm_head with
softcapping applied inside the chunk; (b) `cut-cross-entropy`; (c) hand-rolled
sequence-chunked CE. **This must be proven on a 10-step smoke run before the
window is booked**, because everything else in this document assumes it works.
### Step count
58.2 M tokens / 20,576 samples = **2,829 tokens/sample average** — well under
8192, so packing matters.
- Packed to 8192: **7,104 sequences.** At micro-batch 4 × grad-accum 4
(effective 16) → **444 optimizer steps for the whole epoch.**
- ⚠ That is a *small* step count. A "checkpoint every 100 steps" default gives
four checkpoints across a multi-hour run. This is exactly why the amendment
asked for **wall-clock-interval checkpointing, not step-count** — the case is
now concrete, not hypothetical.
- ⚠ **Packing must use `position_ids` + varlen/block-diagonal attention.** Naive
concatenation bleeds samples into each other. `sliding_window` is 1024 on 25
of 30 layers so the damage is bounded there — but the 5 `full_attention`
layers see the entire packed sequence.
**Open question for brokkr/Eitri:** what fraction of the 20,576 samples exceed
8192 tokens? Below ~2%, 8192 is right. A long tail means truncation is cutting
the ends off RP scenes, which is where the signal lives.
### Runtime
Active parameters per token ≈ **3.67 B** (2.93 B routed + attention, plus the
0.74 B tied lm_head matmul). Forward + backward + gradient-checkpoint recompute
≈ 6 × active × tokens = **1.28e18 FLOPs** for the epoch.
At 10–25% MFU on a 300 W-capped Max-Q card — HF MoE paths with 704-wide experts
are not efficient — **4 to 10 hours, most likely ~6.** Treat as a band, not a
number; it will be measured on the smoke run.
---
## 3. Where it fits (measured 2026-08-24, 18:20 PDT)
Card total: 97,887 MiB = **95.60 GiB** each.
| | GPU0 | GPU1 |
|---|---|---|
| resident before the window | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
| free | 54,741 MiB = **53.46 GiB** | 19,446 MiB = **18.99 GiB** |
Three placements were on the table:
- **GPU0 beside `gen`: does not fit.** 53.46 GiB free against ~57.6 GiB needed —
short by ~4 GiB. And `gen` is only three hours old: measured footprint runs
38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the
current number is budgeting against a moving one.
- **GPU1 with `mog-sec` stopped: 76,070 MiB free.** Fits, but shares a card with
four small seats and Scriberr.
- **GPU0 with `gen` MOVED OFF: the whole card.** ← what was chosen.
---
## 4. The window, as executed
**Operator call, 2026-08-24: move `gen` to GPU1 and stand `sec` down, so GPU0 is
emptied completely rather than shared.** This is strictly better than training
beside `gen`: the tune gets 95.60 GiB with no co-tenant, and the fleet's general
seat never goes dark beyond its own ~5-minute restart.
before: GPU0 [ gen 42.5 ] GPU1 [ sec 55.3 | small seats 20.7 ]
after: GPU0 [ ---- empty, 95.60 GiB ---- ] GPU1 [ gen ~41 | small seats 20.7 | ~33 free ]
`sec` is genuinely in use and this is not free — but it is the smaller blast
radius by a wide margin:
| | `gen` | `sec` |
|---|---|---|
| aliases | 7 (`gen`, `gen-reasoning`, `chat-judge`, `image-judge`, summarizer/classifier family) | 2 (`sec`, `sec-reasoning`) |
| standing role | the fleet's general seat; a documented always-available dependency in global `CLAUDE.md` | M.O.G.-SEC, niche |
| measured traffic | 765 busy-engine log lines in 24 h — continuously in use | bursty; peak 8 concurrent, **last request ~5 h ago** |
Traffic to `sec` arrives from `10.250.50.70` (the LiteLLM gateway), so the
aliases will fail at the gateway for the duration. Per the standing rule, let
them fail — **do not route `sec` to another model as a stand-in.**
Both directions are playbooks, and **the order in each is load-bearing**:
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-open.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-close.yaml
⚠ `gen` runs at `--gpu-memory-utilization 0.43`, which vLLM reads as a fraction
of **total** card memory: 42,091 MiB must be *free at startup* or the engine
refuses to boot. GPU1 has 19,446 MiB free while `mog-sec` is up. **Recreating
`gen` onto GPU1 before stopping `mog-sec` takes the fleet's main seat down and
leaves it down.** The open playbook stops `mog-sec` first and hard-gates on the
freed memory; the close playbook mirrors it, because `mog-sec` needs 50,901 MiB
of its own and cannot start until `gen` has vacated GPU1.
⚠ Invoke elway as `infra-ops@10.250.50.54`, not the `ana-ml2` ssh-target — that
resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo
probe.
### ⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic
Everything above was arithmetic. This was run on the real checkpoint on GPU0
with synthetic tokens (`/tank/erp-tune/smoke_ce.py`), and it moves the answer:
| config | peak | verdict |
|---|---:|---|
| naive CE, bsz1 seq 8192 | **81.93 GiB** | fits, ~14 GiB spare |
| naive CE, bsz1 seq 16384 | **OOM** | tried to allocate 16.00 GiB |
| chunked CE, bsz1 seq 16384 | **65.66 GiB** | ✅ |
| **chunked CE, bsz2 seq 16384** | **79.71 GiB** | ✅ **the run config** |
| chunked CE, bsz4 seq 16384 | **OOM** | — |
**The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB
estimated.** The estimate modelled gradient checkpointing as storing only layer
inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens ×
top-8 of 128 experts, plus scatter/gather buffers) is far heavier. **Do not size
an MoE run from dense-model intuition — measure it.**
Two predictions did land exactly, which is why the rest of the model of the thing
is trustworthy: **205 target modules** (q30/k30/v25/o30/gate30/up30/down30) and
**74,342,400 trainable params** at r64.
The headline: **chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq
8192.** Chunking is not an optimisation, it is what makes brokkr's 16384
recommendation reachable at all.
Base load peak: **49,221 MiB**, confirming the 48.07 GiB weight figure.
### Revised run parameters, now that it is a whole card
**FINAL, measured: `max_seq_len` 16384, `per_device_batch_size` 2,
`gradient_accumulation_steps` 8** → effective batch 16, **~1,280 optimizer
steps**, 79.71 GiB of 95.60 with ~15.9 GiB clear.
`max_seq_len` went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap
drops **6.2% of samples but 22.4% of TOKENS** (61.2M → 47.5M), concentrated
*entirely* in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon —
which is 60% of the mix and the axis the seat exists for. Prose and fireball
truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only.
⚠ The 79.71 GiB figure is **worst case** — every sample in the micro-batch at the
full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751
the typical step sits far below it.
⚠ **Keep gradient checkpointing ON**, and keep `enable_input_require_grads()`
with it. Dropping checkpointing looks like ~17% off wall-clock and instead
forces micro-batch 1. Worse, the second call is the silent one: **without
`enable_input_require_grads()` the frozen base produces no gradient through the
checkpointed blocks, every adapter stays at its initialisation, and the run
completes successfully with an inert adapter.** `prepare_model_for_kbit_training`
used to do it as a side effect of the 4-bit path — so removing 4-bit removes it
too, and nothing warns you.
### Harness changes this required (eitri-smithy `62b556b`)
`9d64257` as audited would not have run here. Four fixes:
1. `runtime.py` hardcoded `BitsAndBytesConfig(load_in_4bit=True)` — now a config
key, defaulting off, per §1.
2. Sequence-chunked CE replacing the model's own loss (the measured table above).
3. `chat_template_path` — `apply_chat_template` resolved the checkpoint's own
stale 365-line template and there was **no override parameter anywhere**, so
the upstream-template requirement was not expressible in the code.
4. Gradient checkpointing + `enable_input_require_grads()`.
Plus `training_eligibility_override` / `overridden_blockers` /
`substitute_controls` in the provenance manifest, and `device_map` pinned to
device 0 so the run cannot stray onto the card holding the inference seats.
Also fold in:
- **Scriberr STAYS on GPU1.** (An earlier draft of this document suggested moving
it to GPU0; that was written when training was going to live on GPU1, and it is
now exactly backwards. GPU0 is the training card and wants no co-tenant.)
- **Package as a `uv` venv on `/tank`, not a Docker image.** Root is at **91%
(36 GB free)** and `/var/lib/docker` lives on it; a PyTorch training image
would come close to filling it. `/tank` has 4.0 TB.
- **The run is still resumable-by-design** (INV-T7 + wall-clock checkpointing).
Nothing about a dedicated card removes that requirement — a 4–10 hour window
is long enough that an unresumable run is a bad bet regardless of who owns the
GPU.
---
## 5. Standing warnings that apply to this run
- **Never render training examples through the base's own
`chat_template.jinja`.** Every third-party Gemma-4 derivative ships a stale
one; the trainee's is 365 lines against upstream's 390. Use
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`. Training through
the wrong template is train/serve skew with no error — it presents as a
tuning failure.
- **Base path and chat-template path are config keys, not constants.** The
trainee base already moved once (stock BF16 → `-heretic-bf16`).
- **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA
context, graphs, or non-torch overhead — the same misreading that OOM'd the
char-rp seat.
- **Serving the result is not settled.** LoRA-on-NVFP4 hot-swap was a silent
no-op on vLLM 0.24.0 (#47639, proven quant-agnostic). Retest on the tagged
`vllm/vllm-openai:v0.27.1` already on disk. **If it still no-ops, the harness
must emit merged weights** — and Eitri needs that requirement while he is
early, not after the run.
---
## 6. Round-1 aborted; throughput root-caused (measured 2026-08-24 22:00 PDT)
Run-01 launched, reached step 19 of 1,312 at ~35–46 s/it, and was **killed by
operator instruction** — not a crash, not an OOM. ETA was ~13.9 h at 8.6% MFU
and the operator elected to root-cause before spending the window.
Nothing was destroyed: the 609 MB encode cache, `order-manifest.jsonl`,
`truncation-report.json` and `resume-run-01.sh` are all preserved at
`/tank/erp-tune/run-01/`. **There are no checkpoints** — the first was due at
step 100, so brokkr's `lora_B` inert-adapter gate never ran. That question is
open and moves to the restart.
Model-agnostic lessons from this investigation are in
[`training-throughput-playbook.md`](training-throughput-playbook.md); the
probes are at [`scripts/training-probes/`](../../scripts/training-probes/).
What follows is Gemma-4-specific.
### 6.1 Where the step time goes
Real checkpoint, GPU0, `attn_implementation="sdpa"`, PEFT + gradient
checkpointing + the chunked CE, fwd+bwd, best-of-2 after warmup:
| shape | time | peak |
|---|---:|---:|
| 2 × 2,048 | 1.776 s | 53.2 GiB |
| 2 × 8,192 | 11.570 s | 62.3 GiB |
| 2 × 16,384 | **35.017 s** | 76.6 GiB |
Fitting `t(w) = A·w + B·w²` over all three (per-sequence `w`, batch 2):
A = 6.8715e-04 s/token B = 8.8509e-08 s/token²
| w | predicted | measured | linear | quadratic | quad share |
|---:|---:|---:|---:|---:|---:|
| 2,048 | 1.779 | 1.776 | 1.407 | 0.371 | 20.9% |
| 8,192 | 11.569 | 11.570 | 5.629 | 5.940 | 51.3% |
| 16,384 | 35.017 | 35.017 | 11.258 | 23.759 | **67.8%** |
**Two terms, three points, residuals under 3 ms across an 8× range.** No fixed
per-batch term was needed, which refutes the launch-bound hypothesis outright —
~3,840 expert-GEMM launches per forward are not the cost.
Independently, the profiler kernel table (device rows only — see playbook §3.4):
| device kernel | ms | of step |
|---|---:|---:|
| `fmha_cutlassB_bf16_aligned_128x64_k65536_sm80` (attn BWD) | 16,144.6 | 46.1% |
| `fmha_cutlassF_bf16_aligned_32x128_gmem_sm80` (attn FWD) | 6,691.2 | 19.1% |
| `cutlass_80_tensorop_bf16_s16816gemm` ×3 (dense GEMM) | 2,774.0 | 7.9% |
| elementwise / vectorized / unrolled ×9 | 4,787.4 | 13.7% |
| gather / Memcpy DtoD / dropout | 951.6 | 2.7% |
| **attention total** | **22,835.8** | **65.2%** |
**Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two
independent methods, 2.6 points apart.**
### 6.2 ⚠ The attention kernels are Ampere, on a Blackwell card
`fmha_cutlass*_sm80` on sm_120. There is no Blackwell-tuned attention kernel in
this path at all, and the forward is additionally on `gmem` — the
global-memory fallback tier of the memory-efficient backend, selected when the
working set will not fit in shared memory.
This is the mechanism behind the 100%-SM / 27-TFLOPS / 304-TFLOPS-capable
reading: the chip is saturated running a kernel generation behind on the
dominant cost centre.
The candidate fix is a purpose-built kernel for this architecture's mixed
256/512 head-dim split — `zzhhjjj/gemma-triton-flash-attn`
(`register_triton_attention()`, then `_attn_implementation = "triton_gqa"`),
reported 9.23× over SDPA at N=16K D=256 SWA and 2.94× fwd+bwd at D=512.
`flex_attention` + `BlockMask` is the no-new-dependency alternative.
⚠ **Prefer a UNIFORM backend over a per-layer split.** vLLM special-cased this
exact mixed-head-dim architecture and measured mixed backends **8% slower** than
uniform. And `attn_implementation` is all-or-nothing at `from_pretrained` /
`set_attn_implementation` — per-layer routing requires a custom function
registered on `ALL_ATTENTION_FUNCTIONS` branching on `module.head_dim` /
`sliding_window`.
⚠ **FA2 is not available for this model**: it caps head_dim at 256 and the 5
global layers are at 512. FA3 is Hopper-only. Do not bet on FA4 on sm_120.
### 6.3 Masking is CORRECT — and padding is what costs
Band structure asserted directly against the real config at n=16,384:
sliding_attention max 1,024 allowed/row, saturates at row 1,023 PASS
Constraints were **not** silently dropped; the 25 sliding layers were genuinely
windowed. Run-01 was training the model we intended.
The same probe found the mechanism nobody had measured:
| 2D mask supplied | `full_attention` mask returned |
|---|---|
| `None` | **`None`** → `is_causal` fast path AVAILABLE |
| all-ones (no padding) | **`None`** → `is_causal` fast path AVAILABLE |
| right-padded (what `collate_mixed` emits) | 4D `16384²` → **fast path LOST** |
**Padding is what pins the 5 global layers to an explicit mask.** The 25
sliding layers get a 4D tensor either way — `sdpa_attention_forward` sets
`is_causal=True` only when `attention_mask is None`, and a 1024 window cannot
be expressed as `is_causal`.
Isolated, same width, only the mask differing:
2 × 16,384, no padding 35.244 s 26,048 loss targets
2 × 16,384, 50% pad on row 1 38.567 s 19,640 loss targets
**9.4% slower for 24% less work.**
### 6.4 The corpus is 29.9% padding — and bucketing is the biggest win available
Measured off the preserved encode cache in true `SequentialSampler` order:
records 20,982 (3,583 rp-dialogue / 12,003 prose-chunk / 5,396 actual-play)
seq len min/mean/max 142 / 2,752 / 16,384
micro-batches (mb=2) 10,491
real tokens 57,733,156
padded tokens 82,337,318
PADDING WASTE 29.9%
mb width p50/p90/p99 2,092 / 10,634 / 16,341
micro-batches at 16,384 3 of 10,491 (0.0%)
⚠ Note the last line against §6.1: **the 2 × 16,384 benchmark shape occurs in
three micro-batches out of 10,491.** Weighted over the real distribution the
quadratic share is ~51%, not 67.8%.
**Bucket-to-pair, shuffle-to-mix** (brokkr's design, validated on measured
lengths — form micro-batches within length buckets, then shuffle the resulting
*micro-batches* globally):
| bucket | waste | predicted step | zero-pad mb | roots/accum window |
|---:|---:|---:|---:|---:|
| current | 29.9% | 44.3 s → 16.13 h | 0.1% | 3.68 |
| **2** | **0.0%** | **28.6 s → 10.40 h** | **78.3%** | 3.56 |
| 8 | 0.0% | 28.6 s → 10.41 h | 65.3% | 3.54 |
| 32 | 0.1% | 28.6 s → 10.43 h | 41.9% | 3.55 |
| 128 | 0.6% | 28.8 s → 10.51 h | 14.7% | 3.55 |
| 512 | 2.4% | 29.7 s → 10.82 h | 4.1% | 3.61 |
**≥35.5% wall clock, no kernel work, no new dependency, peak memory unchanged.**
Two findings that changed the design:
- **Bucket size is not a diversity knob.** Roots per accumulation window are
flat at 3.54–3.61 across a 256× range. The global micro-batch shuffle does
all the mixing. Use the tightest bucket.
- **35.5% is a floor.** Zero-pad micro-batches go 0.1% → 78.3%, which puts the
5 global layers back on `is_causal` for most of the run (§6.3). The cost
model does not capture that. Direction certain, magnitude not yet measured at
representative shapes.
⚠ **Source-homogeneity is a real hazard here** — length correlates hard with
root (kvasir short, chunked RP windows long), so length-homogeneous batches are
root-homogeneous batches. The global micro-batch shuffle is what prevents an
accumulation window drawing its whole gradient from one source. It is
load-bearing, not decoration.
### 6.5 The chunked CE is fine — do not swap it
2 × 16,384 CE forward 374 ms of 35.329 s = 1.1%
2 × 4,096 CE forward 93 ms of 4.387 s = 2.1%
⚠ **Forward only** — the `torch.utils.checkpoint` recompute runs inside
`.backward()`, outside the timing window. Even at 3× it is ~3%.
`liger-kernel` fused linear CE is a ~1–3% lever on this shape. §2's finding
stands unchanged: chunking is what makes seq 16384 *reachable*, and it is not
what makes it slow.
### 6.6 MoE is ~8% — stop optimising it
Dense GEMM is 7.9% of the step, confirming the earlier decomposition bound of
~10% from the kernel side.
On `grouped_mm`: **the trace does not adjudicate it.** Run-01 was relaunched on
`eager`, so the profile shows the *default* path — 25,463 `aten::mm` dispatches
in one fwd+bwd, far more than the ~90 a grouped path would produce, so the
default is per-expert sequential. Whether the flag changes that when set is a
different measurement and was not run. At 7.9% it is not worth running.
### 6.7 Restart parameters for round 2
**Do not relaunch without the sampler change.** It is the only lever that wins
under every branch of the diagnosis.
1. **Implement bucket-to-pair + shuffle-to-mix** in the harness, tightest
bucket, global micro-batch shuffle. Expected ~16.1 h → ~10.4 h or better.
2. **Re-assert the mask band structure** after the sampler change —
`scripts/training-probes/step0_mask.py`, 30 s, no GPU. The sampler touches
batch composition, which is what drives mask construction.
3. **Resume with `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch
command** — it begins `rm -rf /tank/erp-tune/run-01` and would destroy the
609 MB encode cache (2.5 min to reuse, ~4.3 h to rebuild). ⚠ A sampler change
alters record *order*, not encoding, so the cache stays valid — but bump
`encode_version` if anything upstream of `input_ids` changes.
4. **Run the `lora_B` inert-adapter gate at step 100.** It never ran in round 1.
Norm every `lora_B` tensor in the checkpoint: all-non-zero = real, all-zero =
INERT (kill the run), partial = module-selection problem. This is the one
failure that stays invisible until brokkr's acceptance gate reports
base-identical numbers.
5. **The corpus override is ONE RUN ONLY** (`operator-2026-08-25-rnd-run`). A
second run needs a second operator grant.
6. **Attention backend is round 2's second lever**, gated on an A/B on the
replica — not on argument. It can run while the tuned job trains.
⚠ GPU0 is currently **reserved and idle** by operator instruction; `sec` /
mog-sec remains down. The window is still open, so
`playbooks/ana-ml2-training-window-close.yaml` has NOT been run.
docs/pfi/ipv6-naming-scheme.md
+174
View File
@@ -0,0 +1,174 @@
# ESH IPv6 naming scheme
Every ESH LAN carries an **eight-hex-digit phrase** as the first half of the
interface identifier. Picked 2026-08-18/19. This file is the canonical record.
> **Why this file exists.** The scheme originally lived as a single line in
> `persistent-memory.md` and was silently deleted by a `memory: snapshot`
> commit (`837fa36`). Recovering it took a hunt through session transcripts to
> find the commit that had held it. A naming convention is not temporal state —
> it belongs in a document, so it now is one.
## The names
| network | hex | reads as |
|---|---|---|
| `Default` | **`4BA5:3417`** | A BASE FOR IT |
| `esh-mgmt` | **`15DA:B055`** | IS DA BOSS |
| `esh-server` | **`4411:B105`** | FOR ALL BIOS |
| `esh-userland` | **`CAFE:4411`** | CAFE FOR ALL |
| `esh-iot` | **`4DBA:D107`** | FOR DA BAD IOT |
| `esh-cameras` | **`1533:FACE5`** | I SEE FACES |
| *(reserved)* DMZ | **`4411:DBAD`** | FOR ALL DA BAD |
The DMZ name is **claimed against a network that does not exist yet** — there is
no DMZ on the ESH UDM. Do not reuse it.
Substitutions are the standard hexspeak set: `0`→O, `1`→I/L, `5`→S, plus letters
that are already native hex (`A`–`F`). Anything outside `0-9a-f` is not
expressible — `b0ss` and `c00l` do **not** work, which is why the set above uses
`B055` and avoids `c00l` entirely.
House style, arrived at rather than designed: **eight digits, and a complete
phrase rather than a single word.** Words are allowed to straddle the group
boundary (`4DBA:D107` is `4·D·BAD·107`); the phrase reads through the colon.
## Address structure
```
2607:73c0:402:1d02 : 4411:b105 : 50 : 45
└──── ISP /64 ────┘ └ segment ─┘ └ 10.0.50.45 ┘
```
- **Prefix** — Cityside's, not ours to name. ESH holds a `/56`
(`2607:73c0:402:1d00::/56`, 256 × /64); the subnet id is assigned by UniFi's
`ipv6_pd_prefixid`. `esh-cameras` is `1d00`, `esh-server` is `1d02`.
- **Segment word pair** — 32 bits, from the table above.
- **Host** — the last two IPv4 octets, written as literal digits so they read
straight off the address. `10.0.50.45` → `:50:45`.
The scheme lives entirely in the **interface identifier**, so it is
**delegation-size independent**. It works identically on a `/56`, a `/48`, or
NH3's single `/64`. It never competes with the subnet id, which is far too small
to hold a word (8 bits at ESH — two hex digits).
Note: `4411:b105:50:45` fills all four host groups, so there is **no `::`** in
these addresses. Writing `...::4411:b105:50:45` is malformed and will be
rejected.
## What can and cannot carry a name
| slot | nameable? |
|---|---|
| `/64` subnet id (`ipv6_pd_prefixid`) | **No** — 8 bits at ESH, two hex digits, no room for a word |
| the gateway's own address | **No** — fixed at `::1` by UniFi, no field for it |
| a UniFi *client* reservation | **No** — UniFi has no IPv6 equivalent of `use_fixedip` |
| **a host taking its own address** | **Yes** — this is the one that works |
⚠ **The original note concluded these names could never appear in a `dig` or
`ip -6` output. That is wrong.** The first three rows are correct, but they only
establish that *UniFi* cannot assign the address. A Linux host can simply take
one within its own advertised prefix, and the router gets no vote. Appliances
with no shell — cameras, most IoT — genuinely cannot, so `1533:FACE5` and
`4DBA:D107` are likely to stay documentation-only.
## Applying it to a host
Do **not** use an `iface … inet6 static` stanza: on Debian that sets
`accept_ra=0`, killing SLAAC and the IPv6 default route — a good way to strand a
headless box. Use an `if-up.d` hook that derives the live prefix instead.
Live example, `/etc/network/if-up.d/ipv6-scheme-addr` on `esh-docker-vm`:
```sh
#!/bin/sh
[ "$IFACE" = ens18 ] || exit 0
(
i=0
while [ $i -lt 30 ]; do
PFX=$(ip -6 -o addr show dev "$IFACE" scope global 2>/dev/null \
| awk '{print $4}' | cut -d/ -f1 | head -1 | cut -d: -f1-4)
if [ -n "$PFX" ]; then
ip -6 addr replace "${PFX}:4411:b105:50:45/64" dev "$IFACE" && exit 0
fi
sleep 2
i=$((i + 1))
done
) >/dev/null 2>&1 &
exit 0
```
Three deliberate properties:
- **The prefix is derived, never hardcoded** — self-heals if Cityside
re-delegates.
- **Backgrounded with a retry** — SLAAC may not have landed when `if-up.d` runs,
and a hook that blocks or fails would stall interface bring-up.
- **Additive** — `/etc/network/interfaces` already sources `interfaces.d/`;
nothing existing is edited, and removal is one `rm`.
Remaining gap: a *mid-life* prefix change is only picked up at the next
interface-up. A timer would close it; not worth building until the prefix is
observed to actually move.
## Deployed
All three Linux hosts on `esh-server` now carry the segment name, with the last
two groups reading straight off their IPv4 address:
| host | address | v4 | applied via |
|---|---|---|---|
| `esh-docker-vm` (AdGuard) | `2607:73c0:402:1d02:4411:b105:50:45` | 10.0.50.45 | `if-up.d` on `ens18` |
| `esh-pve-nas` | `2607:73c0:402:1d02:4411:b105:50:55` | 10.0.50.55 | `if-up.d` on `vmbr0` |
| `esh-vm-db` | `2607:73c0:402:1d02:4411:b105:50:60` | 10.0.50.60 | `if-up.d` on `ens18` |
`esh-docker-vm`'s is load-bearing, not decorative: the ESH UDM advertises an
IPv6 resolver to clients via RDNSS, macOS prefers it over the DHCPv4-supplied
one, so whatever sits there is what resolves `*.internal` for every Mac on the
network. It previously pointed at AdGuard's **MAC-derived SLAAC address**, which
would have broken if that VM's NIC ever changed. Both `esh-userland` and
`esh-server` now advertise the scheme address instead
(`dhcpdv6_dns_auto=false` + `dhcpdv6_dns_1=<address>`), verified on the wire by
soliciting an RA and parsing option type 25.
### ⚠ Proxmox bridges need `accept_ra=2` or SLAAC never runs
`esh-pve-nas` had **link-local only** despite `accept_ra=1`, `autoconf=1` and
IPv6 enabled — every sysctl looked correct. The cause: **`vmbr0.forwarding = 1`**
(Proxmox sets per-interface forwarding on bridges), and the kernel ignores RAs on
a forwarding interface unless `accept_ra` is explicitly **`2`**. `accept_ra=1`
means "accept only if not forwarding", so it silently did nothing.
Fixed in `/etc/sysctl.d/60-ipv6-accept-ra.conf` on that host:
```
net.ipv6.conf.vmbr0.accept_ra = 2
net.ipv6.conf.vmbr0.accept_ra_defrtr = 0
```
`accept_ra_defrtr=0` is deliberate — it takes the advertised **prefix** (so
SLAAC configures an address) while **declining the default route**, so a
hypervisor gains an IPv6 identity with no change to its routing behaviour.
Verified after: SLAAC address present, v6 default routes still **0**, v4 intact.
Expect the same on any other Proxmox node when its LAN gets IPv6.
### Getting into a host with no direct root
`esh-vm-db` refuses key auth for `root` and `infra-ops`, and `lkraven`'s sudo
wants a password. It is VMID 101 on `esh-pve`, and the **QEMU guest agent** runs
as uid 0 inside it, so the hook was installed with:
```
qm guest exec 101 -- /bin/sh -c 'echo <base64> | base64 -d > /etc/network/if-up.d/... '
```
base64 because quoting a multi-line script through two SSH layers mangles it.
Worth remembering as the general path for guests whose credentials are not
vaulted.
Every Linux host on `esh-server` now carries its name. The remaining ESH
segments have no eligible hosts: `esh-cameras` and `esh-iot` are appliances
with no shell, and `esh-mgmt`, `esh-userland` and `Default` are still
`ipv6_interface_type: none` pending the firewall-policy pass — enabling SLAAC
there gives every client a globally reachable address.
docs/pfi/training-throughput-playbook.md
+359
View File
@@ -0,0 +1,359 @@
# Training throughput playbook — how to find where the step time went
_Sibling to [`model-quantization-playbook.md`](model-quantization-playbook.md).
That one is for making a model small; this one is for making a training run
fast. Same contract: **model-agnostic lessons live here, model-specific ones
stay in the per-model artifact and link up.**_
First written 2026-08-24 out of the Gemma-4 26B-A4B ERP/RP tune, which ran at
8.6% MFU and cost a four-model frontier panel and most of a night to explain.
The worked example in §7 is that run. The lessons above it are not about
Gemma-4.
> **Read this before hypothesising about kernels.** The single most expensive
> failure in that investigation was not a wrong hypothesis. It was *four
> people, including four frontier models, reasoning confidently from
> arithmetic instead of spending ten minutes on a measurement that settled
> it.* Two of the panel's conclusions were retracted by their own authors
> within the hour. Every retraction was a derivation; every survivor was a
> measurement.
---
## 1. The 10-minute triage — do this FIRST, always
Before you profile, before you read a modelling file, before you ask anyone:
**measure the step's scaling curve.** Three sequence lengths, fixed batch,
fwd+bwd, best-of-2 after a warmup.
t(w) = A·w + B·w² w = per-sequence length
Fit two parameters to three points. The residuals tell you which regime you
are in, and the regime tells you which lever exists:
| observed `t(4w)/t(w)` | regime | the lever |
|---|---|---|
| ~4× | **linear** — per-token work dominates | fewer tokens; fused elementwise |
| ~16× | **quadratic** — attention dominates | attention backend / kernel |
| ~1× | **launch-bound** — fixed per-batch cost | CUDA graphs, `torch.compile`, bigger batch |
**If the two-term fit closes with residuals under ~1%, launch-bound is
refuted.** You did not need a constant term, so there is not a meaningful one.
This is the cheapest possible refutation of the most seductive wrong answer,
and it costs one extra data point.
### ⚠ 1.1 ⭐⭐ Three points minimum. A two-point fit with three plausible terms is UNDETERMINED
This is the lesson that cost the most. A two-point fit over {quadratic,
linear, fixed} has infinitely many solutions, and which one you land on is
decided by whichever per-step number you happened to quote. In the worked
example a peer produced **two confident, opposite conclusions from the same
method inside an hour** — "attention is ~5 s of 35" and then "attention is
21–33 s of 35" — because the inputs drifted between attempts.
Three points, two parameters, and check the residuals. If they do not close,
you have a third term and you need a fourth point.
### ⚠ 1.2 ⭐⭐ Benchmark the shape you RUN, not the worst case you can construct
The quadratic share is **strongly shape-dependent** — in the worked example it
ran 20.9% at w=2,048, 51.3% at w=8,192, 67.8% at w=16,384. A synthetic
`max_seq_len` benchmark therefore measures the shape where attention looks
worst, and generalising from it overstates the attention prize by ~1.3×.
Get the real distribution off the encode cache and weight by it:
E[t] = A·E[w] + B·E[w²]
**`E[w²]` is not `E[w]²`.** For a bimodal length distribution they can differ
by 2× or more, and a quadratic term is dominated by the rare long batches that
an `E[w]²` shortcut averages away. In the worked example `E[n²]/E[n]²` was
**2.08**.
Sanity check the weighted prediction against the observed `s/it` before you
trust any of it.
---
## 2. The reference probe set
Committed at [`scripts/training-probes/`](../../scripts/training-probes/).
Run them in this order; each is minutes and none needs the real checkpoint
except the profiler.
| probe | what it settles | needs GPU? |
|---|---|---|
| `step0_mask.py` | mask band structure + which layers keep the fast path | no |
| `step2_padding.py` | padding waste, length distribution, CE chunk sizing | no |
| `step_bucket.py` | bucketing gain, bucket-size sweep, root diversity | no |
| `step1_profile.py` | scaling fit, padding penalty, CE wall clock, kernel table | yes |
`step1_profile.py` loads the real model but reuses the harness's own
`discover_target_modules` and `compute_loss`, so it measures the thing that
actually runs rather than a re-implementation. **Keep that property when you
adapt it** — a probe that reimplements the training step measures the probe.
---
## 3. The recurring landmines
### 3.1 ⭐⭐ Right-padding is a compute tax AND a backend tax
Everyone knows padding wastes tokens. The second effect is the one that gets
missed: **an explicit padding mask can knock fast-path-eligible layers off
`is_causal`.**
`scaled_dot_product_attention` takes `is_causal=True` **or** an `attn_mask`,
never both usefully. HF sets `is_causal=True` only when `attention_mask is
None`. Right-pad a batch and you hand it a 2D mask, it materialises a 4D
tensor, and every layer that could have taken the clean causal route now takes
a masked dense one.
Measured, same width, same `n`, only the mask differing:
no padding 35.244 s 26,048 loss targets
50% pad on one row 38.567 s 19,640 loss targets
**9.4% slower for 24% less work.** Verify this on your own stack with
`step0_mask.py` — it prints whether `create_causal_mask` returns `None` or a
tensor for each mask case.
### 3.2 ⭐⭐ Length-bucket to PAIR, shuffle micro-batches to MIX — and the bucket should be TIGHT
Naive length-bucketing has a real hazard: length correlates with data source,
so length-homogeneous batches are **source-homogeneous batches**, and an
accumulation window can end up drawing its entire gradient from one root.
The fix costs nothing: **form micro-batches within length buckets, then
shuffle the resulting micro-batches globally.** Padding efficiency is a
property of the pairing alone, so all of the saving survives the shuffle.
**The non-obvious part — bucket size is not a diversity knob.** Measured
across a 256× range of bucket sizes, roots per accumulation window stayed flat
at 3.54–3.61 (against 3.68 for a pure shuffle). The *global micro-batch
shuffle* does all of the mixing; the bucket contributes nothing to diversity
and only costs padding. So use the tightest bucket you can — which in the
limit is a full length sort.
| bucket | padding waste | zero-pad micro-batches | roots/window |
|---|---|---|---|
| current (shuffle) | 29.9% | 0.1% | 3.68 |
| 2 | 0.0% | **78.3%** | 3.56 |
| 32 | 0.1% | 41.9% | 3.55 |
| 512 | 2.4% | 4.1% | 3.61 |
Note the `zero-pad micro-batches` column — that is §3.1 compounding. A tight
bucket does not merely cut tokens, it puts most batches back on the causal
fast path.
**Peak memory does not rise.** `padded = batch × max(len)`, so one long record
forces a full-width batch regardless of its partner. Bucketing pairs long
records *with each other*, which roughly halves the number of worst-case
batches.
### 3.3 ⭐⭐ Check the kernel GENERATION, not just the backend name
The backend name (`EFFICIENT_ATTENTION`, `FLASH_ATTENTION`, …) is not the whole
story. Read the actual kernel symbols out of the profiler:
fmha_cutlassF_bf16_aligned_32x128_gmem_sm80
fmha_cutlassB_bf16_aligned_128x64_k65536_sm80
^^^^
`sm80` is **Ampere**. Those were running on an sm_120 Blackwell card, on the
dominant cost centre of the step. A backend can be "selected correctly" and
still be a generation behind, and nothing in the config surface tells you.
Also read the variant suffix: `gmem` on the forward kernel is the
**global-memory fallback tier** of the memory-efficient path, chosen when the
working set will not fit in shared memory. Wrong backend *and* that backend's
slow path.
### 3.4 ⭐ `key_averages()` double-counts — use device-kernel rows only
`torch.profiler`'s `key_averages()` table lists both the ATen op and the CUDA
kernel it launched, each carrying the same `self_device_time_total`. Summing
the whole table gives you roughly **2× the real step time**.
The tell is exact equality between an `aten::` row and a kernel row:
aten::_efficient_attention_backward 30 16144.6
fmha_cutlassB_bf16_aligned_128x64_k65536 30 16144.6
Filter to device kernels (`void …`, `fmha_…`, `cutlass::…`, `Memcpy…`) and
sanity-check the total against the measured wall clock. In the worked example
the filtered total came to 89.5% of the step, which is the right shape; the
unfiltered total came to 202%.
### 3.5 ⭐ Time the loss forward AND account for its backward recompute separately
If the loss head is gradient-checkpointed, a CUDA-event window around the
forward loop measures **half the story at best** — the recompute happens inside
`.backward()`, outside your window.
State the caveat explicitly when you report the number. In the worked example
the CE forward measured 374 ms of a 35.3 s step (1.1%); even at 3× for
recompute-plus-backward it is ~3%, which was enough to kill a proposed
dependency swap — but "1.1%" alone would have been an unearned claim.
### 3.6 ⭐⭐ Assert mask band structure directly; never infer it from performance
`transformers` can **silently skip mask creation** and pass
`attention_mask=None` when a mask function is not registered. If that fires on
a sliding-window model, the windowed layers do full causal attention — not a
speed bug, **a different model from the one you will serve**.
There is a tempting alibi: "if constraints were dropped we would be on the
fast path and fast; we are slow, therefore correct." It is decent evidence and
it is not an assertion. Materialise the mask once and count allowed positions
per row:
sliding_attention max 1,024 allowed/row, saturates at row 1,023 PASS
Thirty seconds, on CPU, no weights. Do it before every run that changes the
masking path, and before believing any optimisation result.
### 3.7 ⭐ "Bit-identical output from a different backend" — ask *could this have disagreed?*
A backend flag that produces `max_abs_diff == 0.0` against the reference is
either (a) legitimately the same GEMMs behind a different launcher, or (b) a
flag that never took. **Argument cannot separate these** — in the worked
example three frontier models split 2–1 on it and the majority was not
obviously right.
Do not resolve it by vote. **Count kernel launches.** A per-expert loop leaves
`n_experts` dispatches per layer visible; a grouped path leaves one. That is
unambiguous and falls out of a trace you are running anyway.
Related trap: **a trace of the default path does not test the flag.** If the
run was relaunched without the flag set, the profile tells you what the default
does and nothing about the flag. Say so rather than over-claiming.
### 3.8 ⭐ MFU is a denominator argument waiting to happen — report the decomposition instead
MFU invites an unwinnable fight about what counts as a FLOP (active vs dense
params for MoE, whether checkpoint recompute counts, whether frozen-base
skipped GEMMs count). That fight consumed an hour of the worked example and
produced nothing.
Report these **beside** MFU, not instead of it:
- tokens/s, and **real (unpadded) tokens/s** separately
- achieved hardware FLOPs straight from the profiler
- the time decomposition (attention / GEMM / elementwise / other)
Then the denominator stops mattering.
**The reading that actually diagnosed it** was not an MFU number at all:
> 100% SM utilisation at 279–292 W, running 27 TFLOPS, on a card that does
> 304 TFLOPS on a dense GEMM at the same power.
**SM-busy, tensor-core-idle.** The chip is fully occupied doing work that is
not matrix multiplication. No FLOP-counting convention changes that, and it
points straight at the kernel table.
### 3.9 Frozen-base LoRA is ~4ND, not ~6ND — and the arithmetic intensity does NOT drop
A claim that circulated and was wrong: "frozen-base LoRA has structurally lower
arithmetic intensity, so a dense-GEMM ceiling is unreachable in principle."
The correct accounting: forward is 2ND, input-gradient backward through the
frozen weights is 2ND, and only the weight-gradient (~2ND) is skipped. So
**~4ND against ~6ND — two-thirds of the work, at the same arithmetic intensity
per remaining GEMM.** You do fewer GEMMs; the ones you do are exactly as dense.
Gradient checkpointing is a separate, real ~⅓ recompute tax. Account for it
separately rather than folding it into an intensity story.
---
## 4. Panel / consult discipline for perf work
Perf investigations are unusually good at generating confident wrong answers,
because the arithmetic is easy and the ground truth is expensive. Specific
guards, learned the hard way:
- **Every arm's claim gets a measurement or an expiry date.** In the worked
example the panel produced four self-retractions in ninety minutes. The
measurements produced zero.
- **Treat cross-arm agreement as weak evidence.** Ask arms to attack a
hypothesis rather than extend it; agreement among similarly-primed readers of
the same artifact is not independent confirmation.
- **A dispute about what a specific dispatcher does is a question of fact.**
Do not put it to a panel. Instrument it.
- **When an arm says "you missed X," check what they read.** If your settled
artifact was not on their reading list, the "miss" is usually
restatement-of-a-settled-prior, not a genuine gap.
---
## 5. Superseded claims — do not follow these
| claim | status | replaced by |
|---|---|---|
| "Explicit mask → `EFFICIENT_ATTENTION`" is over-specific; Blackwell defaults to `CUDNN_ATTENTION` | **WRONG** (2026-08-24) | Measured: sm_120 selects `fmha_cutlass*_sm80`, i.e. `EFFICIENT_ATTENTION`. The original claim was right. |
| Attention's quadratic share is ~5 s of a 35 s step | **WRONG** (2026-08-24) | Measured 22.8 s / 65.2% at w=16,384; 67.8% by independent scaling fit |
| Frozen-base LoRA has structurally lower arithmetic intensity | **WRONG** (2026-08-24) | ~4ND vs 6ND at unchanged intensity — see §3.9 |
| The chunked CE is a 2–5× under-estimated cost centre | **WRONG** (2026-08-24) | Measured 1.1% of step forward, ≲3% with recompute |
| `attn_implementation="flash_attention_2"` is the per-layer lever | **NOT A FLAG** (2026-08-24) | All-or-nothing at `from_pretrained`; per-layer needs a custom fn on `ALL_ATTENTION_FUNCTIONS`. FA2 also caps head_dim at 256. |
| Bucket size ~256 is needed to preserve source diversity | **UNNECESSARY** (2026-08-24) | Diversity is flat in bucket size; the global micro-batch shuffle does that work — see §3.2 |
## 6. Measured negatives — don't re-chase
- **Fused MoE kernel (`grouped_mm`) as the throughput fix.** Measured 0.9%
*slower* than the Python loop and bit-identical. Independently, dense GEMM is
only 7.9% of the step, so the whole category is capped near 10%.
- **CUDA graphs / `torch.compile` over the expert loop.** The two-term scaling
fit closed without a constant term, so there is no meaningful fixed per-batch
cost to amortise. ~3,840 expert-GEMM launches per forward are not what you
are paying for.
- **`liger-kernel` fused linear CE.** Real and correct, but a ~1–3% lever on
this shape. Not a project.
- **FlashAttention-4 on sm_120.** Public reports are sour — one measurement of
1.07× over FA2, and an sm_120 patch people could not get working that fell
back to torch SDPA. Do not bet a round on it.
---
## 7. Worked example — Gemma-4 26B-A4B ERP/RP tune, 2026-08-24
Model-specific detail lives in
[`gemma4-erp-tune-sizing.md`](gemma4-erp-tune-sizing.md) §6. The short version,
because the *shape* of the investigation is the transferable part:
**Symptom.** 8.6% MFU, ~35–46 s/it, 1,312 steps, ~13.9 h ETA.
**What the panel produced.** Four frontier arms plus an orchestrator, over
ninety minutes: a sliding-window hypothesis, a retraction of it, a retraction
of the retraction, a correctness scare that resolved itself, two mutually
contradictory readings of one dispatcher, and four self-corrections.
**What settled it, in about twenty minutes of GPU time:**
scaling fit (3 points, 2 params, residuals <3 ms over an 8× range)
A = 6.87e-4 s/token B = 8.85e-8 s/token²
quadratic share: 20.9% @ w=2,048 → 67.8% @ w=16,384
kernel table (device rows only)
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
Two independent methods, 2.6 points apart. **Attention was the answer, on
Ampere-generation kernels, with the forward on a global-memory fallback tier.**
**The largest actionable win was not the attention kernel.** It was a sampler
change — bucket-to-pair, shuffle-to-mix — worth 29.9% of tokens and ~35.5% of
wall clock, with no new dependency, no kernel work, and unchanged peak memory.
It also wins under *every* branch of the diagnosis, which is why it was
recommended while the rest was still unresolved.
**The transferable ordering:**
1. Assert correctness (mask band structure). Everything downstream assumes it.
2. Scaling curve. Names the regime in ten minutes.
3. Kernel table. Names the cost centre.
4. Data-side levers first (padding, bucketing) — they need no dependency and
they multiply into every other cost.
5. Kernel/backend levers last, gated on 2 and 3.
persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md
+56
View File
@@ -127,3 +127,59 @@ Cloudflare DNS-edit token, so this is self-serve.
WireGuard holds keys in kernel memory, so no restart was needed. The parent
`/etc/wireguard` was already 700, which capped the real exposure to root-capable
contexts inside the LXC — but the modes were still wrong.
---
## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
## per-REQUEST truth, not the total — eight /64s exist and are unclaimed
The section above concludes AT&T hands out a single `/64` and that the
`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
itself superseded later in the same session, and the correction never made it
into memory** — it survived only in the session transcript, and was recovered
2026-08-24 while assessing a proposal to grab more prefixes.
Reading the **BGW's own LAN statistics page** gave the whole picture:
```
BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix
BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself
Delegated 2600:1700:b25:c11f::/64 <- what the UDM got
```
**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
delegating from the top down — the UDM got `c11f`, the last one. So
`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
sitting unclaimed.
Both observations are compatible, which is why the first one looked conclusive:
the prefix-ID test only carves *within* a delegation already held, so a UDM
holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.
**Consequence — the ceiling is the requester, not the carrier.** More prefixes
need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
looks like BGW rationing rather than a provisioning-profile limit.
Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.
### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise
The seven unclaimed `/64`s stay unclaimed. Two facts close it:
- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
on it — so the cheap path (let the UDM take the `/60` directly and carve it
natively, as it already does at ESH) does not exist here.
- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
(UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
firewall policy in nftables on that VM**, because routing v6 around the UDM
bypasses its zone firewall entirely and would leave every LAN host globally
reachable.
Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
Do not re-propose on the strength of "there are seven free prefixes" — the
prefixes are real, the firewall rebuild is why nobody wants them.
persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md
+34
View File
@@ -0,0 +1,34 @@
# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
four interactive accounts, verified as real exposure by reading one as `nobody`.
Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
stacks already used 0600, so this converged on the existing house pattern rather
than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
by `nobody`.
Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
## The check that matters
Every run asserts `docker compose config` still renders **as the deploy user**
(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
as the deploy user proves the next deploy can still resolve its variables.
## Two gotchas recorded in the playbook
- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
*container's* UID reads it and 0600 could break the password vault. It does not:
that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
item without checking its parent key. The playbook now **refuses** any stack that
genuinely bind-mounts its `.env`, since that case is read by the container UID.
- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
`lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
the infra-ops target on this host.
Commit `a896c0a`.
persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md
+663
View File
@@ -0,0 +1,663 @@
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
> the bottom before acting on anything here.** The headline is wrong (the
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
> original stopped measuring at 4), and the proposed remedy is **impossible**:
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
> per-stream observation and the parallelise-your-transfers mitigation are the
> parts that survive.
The operator noticed site-to-site transfers were slow for a datacenter fiber
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
circuit is fine.
## Measured
```
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
FortiGate's own recorded peak 554 Mbit/s
ANA <-> NH3, single stream 227 Mbit/s
ANA <-> ESH, single stream 249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
```
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
## What it is not
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
## Most likely cause
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
faster on **AES-GCM**, which combines encryption and authentication in one
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
shortfall is consistent with that.
## Not executed
Changing the proposal is a **production-edge change requiring a matching
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
it renegotiates. Left for the operator. See the index entry for authorization
state.
## Immediate mitigation, no config change
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
single stream through IPsec gets 227. **Anything moving bulk data across the
Anaheim link should parallelise** — that alone roughly doubles throughput
today.
## Practical consequence already observed
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
## Access note
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
prints live ESP session keys** — never paste its output into althing, a
booth, or a commit.
---
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
and the measurements taken while trying show there is very little left to win.
### AES-GCM is unavailable on the far end — not a naming problem
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
**The control that makes this conclusive:** the *identical* request body with
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
enum rejecting the value, not a malformed body. Corroborating: **zero
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
UniFi far ends, so this blocks the ESH tunnel too.
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
functionally identical while the peer only offers CBC, and reverting it would
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
IKE protects the control channel only and has no bearing on data throughput.
### The numbers that retire this as a problem
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
the same UDM and the same uplink, with the same stdlib TCP probe:
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|---|---|---|
| 1 | 245 Mbit/s | 557 Mbit/s |
| 4 | 471 Mbit/s | 767 Mbit/s |
| 8 | **692 Mbit/s** | **795 Mbit/s** |
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
### What the constraint actually is
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
a higher floor (557 → 795 is only 1.43× scaling).
### Actionable consequence
Anything moving bulk data across this link should **parallelise** — 245 → 692
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
lever**: it opens N TCP connections per mount, converting a single-stream
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
example — remounting with `nconnect=8` is the obvious test.
### Foot-gun recorded
Probing the enum by PUTting candidate values **applies the accepted ones**. A
probe loop here timed out with `3des` briefly live on the NH3 side, which the
FortiGate would not accept — a short tunnel outage until `aes256` was restored
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
config enum this way, restore the known-good value after **every** 200, not at
the end of the loop. Post-change verification: the UDM object was diffed
field-by-field against its pre-change snapshot and is **byte-identical**.
---
## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
The correction above called the constraint "per-SA/per-flow serialisation".
That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
problem, not loss, not window size.
### The evidence: pin the send buffer and sweep it
Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|---|---|---|---|---|
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
**Throughput is flat across a 13× range of in-flight data while RTT scales with
it.** That is the signature of a fixed service rate with a standing queue: the
window controls only how much queue you build, never how fast you go. Had this
been window-limited, throughput would have risen with the buffer. Had it been
congestion, there would be retransmits — there are essentially none
(`retrans:0`, 0% ping loss).
So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
**red herrings here**. Do not go there.
### Bufferbloat: one bulk stream wrecks latency for everything else
Measured on the same tunnel, ping to ana-docker:
- idle: **6.9 ms** avg
- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
**15× latency inflation from one transfer.** This is the operationally
important finding — any interactive traffic sharing the Anaheim link (ssh,
RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
takes only one stream to do it. Parallelising transfers makes throughput
better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
tunnel (or rate-limiting bulk jobs), not more buffer.
### Where the cap lives — strong inference, not proof
Three paths, and the FortiGate is the only variable:
| path | single-stream |
|---|---|
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
Present in both slow paths, absent from the fast one. Aggregate over the same
SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
many flows spread out fine, one flow does not.
The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
each firewall session is bound to one crypto engine, so a single TCP flow is
capped at one engine's rate while many sessions spread across engines. **This
is inference from the throughput shape, not something confirmed on the box** —
`diagnose sys session list` was not captured for a TCP flow (the filter caught
only traceroute UDP probes). A single-stream control through ana-gw *without*
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
window-limited and does not discriminate. **If this matters, the clean test is
a non-IPsec single stream between two Anaheim VLANs at low RTT.**
**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
throughput toward the WireGuard figure. That is a point in favour of the
cutover, and it is cheap to verify afterwards by re-running the sweep.
---
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
Operator asked directly whether the 80F "can't handle the traffic". It can.
Two new measurements settle the shape of this, and correct an overstatement in
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
be separated by that argument).
### The 80F routes a single flow at line rate when IPsec is not involved
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
`10.250.50.1`), 0.36 ms RTT, no tunnel:
| streams | throughput |
|---|---|
| 1 | **940.2 Mbit/s** |
| 8 | 939.3 Mbit/s |
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
general, and there is no per-session ceiling in its plain forwarding path. The
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
### Both IPsec tunnels converge on the same numbers despite different far ends
Measured today with the same probe:
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|---|---|---|---|---|
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
Different gateway hardware, different sites, different uplinks, and RTT
differing by 1.7× — yet single-stream differs by only 9%. **If this were
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
independent confirmation of a rate cap rather than a BDP effect.
### Capacity summary — the box has headroom it will not give one flow
- plain routing, 1 stream: **940 Mbit/s** (line rate)
- plain routing to internet, 8 streams: **2,153 Mbit/s**
- IPsec, 8 streams: **692–715 Mbit/s**
- IPsec, 1 stream: **245–268 Mbit/s**
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
distributes work **by inner session** — consistent with IPsec offload binding a
session to one crypto engine.
### What is still NOT separated
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
same UniFi firmware, so identical caps are explainable either way. The Pro Max
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
CPU should show more), but that is suggestive, not conclusive.
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
FortiGate cutover** — worth running before that decision, not after.
---
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
internet. It is the test that separates the *path* from the *crypto*, and the
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
transport is not worth it" — that conclusion compared 8-stream numbers and was
wrong for single-stream workloads.**
### Setup (fully torn down afterwards)
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
place** (benign, and wanted if this becomes permanent).
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
only NAT/forward UDP, and Linux does WireGuard at both ends.
### The full comparison
| path | crypto performed by | 1 stream | 8 streams |
|---|---|---|---|
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
no per-flow penalty at all, and a single flow already saturates the path. So
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
uplink and not the physical path — all of which sustain 767 on one flow.
Per-flow penalty ranks by implementation:
- **Linux WireGuard — none** (767 → 763, flat)
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
### Latency under load — the same story
| path | idle | during ONE bulk stream |
|---|---|---|
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
WireGuard carries **3.1× the single-stream throughput with 8× less latency
inflation** on the same wire.
### Attribution — still not fully separated, and it no longer matters much
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
does not isolate which one imposes the 2.8× penalty. Closing that would need
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
decision no longer depends on the answer**, because the fix is the same either
way and it is already demonstrated.
### Recommendation (supersedes FOLLOW-UP 1)
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
worth real consideration: 3.1× single-stream, flat scaling, far better latency
under load, and it reuses infrastructure that already exists and is already
internet-reachable. It is also the architecture already proven for NH3↔ESH.
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
has headroom.
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
approval to apply it once the FortiGate work settles, with no need to ask again.
---
## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
Operator's theory — the UDM does IPsec in software with no crypto offload, so
the cost of the cipher itself is the limit — is **correct**, and it is now
demonstrated rather than inferred. He also correctly pointed out that
UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
about UniFi's IPsec performance. It didn't, and I had leaned on it.
### The controlled experiment: vary cipher cost, hold everything else
AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
constraint, throughput must rise when the cipher gets cheaper. If the limit
were the FortiGate's NPU, it would not move at all — hardware crypto is not
cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
| condition | ESP cipher | single-stream | UDM CPU |
|---|---|---|---|
| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
CPU work, more payload through it. That is the signature of CPU-bound software
crypto, and it rules out the FortiGate's NPU as the limiter.
### Correcting two of my own earlier claims
1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
artifact.** UniFi's `system-stats.cpu` refreshes on the device report
interval; 4-second sample windows were reading stale values. Under a
sustained 60 s single-stream load it reads **35.4%**, with load1 rising
0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
on crypto plus overhead. **Always drive load for ≥60 s before trusting a
UniFi CPU figure.**
2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
unproven. It predicts no change from a cipher swap; a 20% change was measured.
### Why the numbers all line up now
- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
and is *not* cipher-sensitive, consistent with it being bounded by the path/
uplink rather than crypto once several cores are engaged.
- **AES-CBC is the specific villain: it is serial.** Each block depends on the
previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
(CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
- **This retroactively vindicates the GCM cutover as the right idea aimed at the
right box** — GCM would have removed the serial dependency on the constrained
end. UniFi simply does not offer it, which is what made it impossible.
### Options this opens
- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
call per end. 128-bit is not the weak link here (SHA1 integrity is more
dated, and unchanged either way). Operator's call — **not adopted**, restored
to aes256.
- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
- Replacing the FortiGate **will not help this** — it was never the constraint.
Worth knowing before the cutover.
### State left behind
UDM network object verified **byte-identical** to its pre-test snapshot
(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
offers only aes256 so the extra entries are inert, but **narrowing it back to
`aes256-sha1` is one line** if the looser list is unwanted.
---
## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
pfi-pve, 10.250.50.252) already is that box.
**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
Anaheim hosts, which was never measured. Now it is:
| topology | 1 stream | 8 streams |
|---|---|---|
| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
architecture delivers the full 3.1× single-stream for real transit traffic, not
just for traffic landing on the tunnel box.
ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
load, not the container's — do not read it as ana-wg's own. For contrast the
UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
### Design consequences of terminating downstream — the parts that need decisions
1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
transit VLAN for ana-wg, or accept the hairpin.
2. **New single point of failure.** Today site-to-site dies only when the edge
dies, which is total anyway. A downstream terminator fails independently.
Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
route so it takes over when ana-wg is down.
3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
hypervisor shared with the rest of the Anaheim VMs.
4. **The NH3 end needs a terminator too**, and there are two shapes:
- **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
was measured: **764 Mbit/s**.
- **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
Site Magic), so expect ~557 rather than 764. Still 2.3× today.
### Standing recommendation
Worth doing, but it is **a project, not a config tweak** — routing, failover and
policy all need deciding. The cheap wins remain available meanwhile and are
independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
the FortiGate work settling) and AES-128 for ~20%.
---
## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
is that trigger.
### What changed
Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
a far end:
| phase2 | proposal now |
|---|---|
| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
Then each UDM flipped to `ipsec_esp_encryption: aes128`:
| tunnel | UDM object | before | after |
|---|---|---|---|
| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
the difference is live-link variance, not a different result. Both UDM objects
were diffed field-by-field against pre-change snapshots: **the only field that
moved on either is `ipsec_esp_encryption`.**
The FortiGate proposal lists were deliberately **left permissive** (still
accepting aes256). The peers offer only aes128 so the extra entries are inert,
and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
`aes128-sha1` alone is a one-liner if the looser list is unwanted.
### Admin surfaces closed
`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
**Sequencing that matters: the close was executed over the TUNNEL path, not over
WAN** — removing `ssh` from allowaccess while connected over WAN kills the
session mid-command.
**Consequence to hold in mind: ana-gw now has no out-of-band management path.**
If both tunnels drop it is console-only until someone is on site.
### Gotcha: the two UDM vault items have DIFFERENT shapes
- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
`key:` line. `secret get` piped straight into a header yields a 1396-byte
value and the UDM answers **`400 Bad Request` from nginx**. Extract with
`grep '^key:' | awk '{print $2}'`.
**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
it as read-verified only): a control PUT of the unchanged object returned
`rc:ok`, then the real change did too. That key has a full read+write admin role.
---
## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
The claim in the previous section that `.42:80` was an **ISP transparent proxy**
was **WRONG**, and so was the earlier warning that ACME renewal would fail with
port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
map terminated. It terminates **on the FortiGate itself**.
**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
allowaccess` reads `ping` only and the port is still open. Every non-challenge
request returns a fixed 403 whose body is literally:
```
<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
```
**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
WRONG, see the correction below.** All fourteen VIPs are scoped.
### The methodology error that produced the wrong answer — worth not repeating
The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
matches only inbound packets**, so outbound SYN-ACKs were excluded *by
construction*; concluding "the box sends no SYN-ACK" from that capture was
unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
**Rule: when testing whether a box *answers*, the sniffer filter must be
bidirectional. `dst host` silently answers a different question.**
### Consequences
- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
back or the cert expires" warning is retracted — FortiOS opens the challenge
port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
- **It is not an admin surface** — static 403, no auth, no GUI.
- Its practical value is now low: WAN admin is closed, so the cert only serves
the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
it (`config system acme` → unset interface) would close the last WAN listener
at the cost of cert renewal. Operator's call; **not done**.
---
## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
### ACME disabled — the WAN IP now exposes nothing
`config system acme / unset interface` (the account object is left in place;
with no interface bound there is no listener). Verified:
- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
- Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
Reverse with `config system acme / set interface "wan1"`.
### RETRACTION: the four VIPs are NOT all-port
A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
A FortiOS VIP can be scoped **two different ways** and the parser used only
checked one:
1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
All four use form 2. The custom services are narrow: `Rustdesk` = TCP
21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
check `service` too.** Better still, do what settled it here: scan from outside
rather than reading config.
### Ground-truth public exposure (external TCP scan, post-change)
| IP | open | maps to |
|---|---|---|
| 38.120.12.41 | *nothing* | — |
| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
the OPNsense translation but not exposure.
persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md
+63
View File
@@ -0,0 +1,63 @@
# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
`http://hrafn:8080`. Canonical at `stacks/hrafn/`.
## Intake found a live credential exposure
`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
— verified as real exposure by reading it as `nobody` on a box with four
interactive accounts. Tightened to 0600. That triggered the wider sweep (see
[[2026-08-23-ana-docker-env-perms-sweep]]).
## The CI defect — the one worth remembering
I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
tagging the image with the commit SHA for provenance. nevermore-claude later found
v1.0.0 deploying "green" while the host still served 0.1.0.
**Root cause was mine and nastier than either hypothesis.** The staging dir was
`$compose_dir/.stage` — **inside** the rsync target. So
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
destination (absent from the source listing) **during** the transfer, destroying
its own source mid-copy. Reproduced exactly:
```
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
after: app.py="OLD" leftover.txt GONE, .stage GONE
```
Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
*looked* converged while host source stayed frozen at the first manual rsync —
and because the build's `COPY` inputs never changed, Docker full-cache-hit and
every SHA tag aliased one image. **The provenance the tagging existed to provide
was false for the pipeline's entire life.**
**The real failure is the verification.** The verify steps asserted the marker,
container health, and a 200 from `/readyz` — all of which pass against a
completely frozen host. None measured *content*. A deploy that reports success
without asserting the bytes changed is verifying an **uptime**, not a deploy.
## Fixes
- stage at `/tmp/hrafn-deploy-stage`, outside the target
- CI computes `context_sha256` over the shipped file list; the playbook recomputes
it **on the host after the converge** and fails on mismatch
- compare the running container's `src/**/*.py` against the host's, so a SHA tag
cannot name layers the image lacks
- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
false-fails on every healthy deploy. Verified against a known-good container
before shipping (12 host files, 18 in container, 0 content differences).
- declined `--no-cache`: a cache hit is *correct* when the context is genuinely
unchanged; assert the property rather than brute-force it.
## Access
Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
canonical; `stacks/hrafn/ci/` is a verified mirror.
Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
persistent-memory.d/2026-08-23-selene-retired-alias-collision.md
+81
View File
@@ -0,0 +1,81 @@
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
## Why selene went
Benchmarked against `gen` on selene's own job — 24 designed judge items with
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
```
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
```
gen won on both templates and **selene's best sat below gen's worst**. Selene was
given its own fine-tuned template as a fairness check before any recommendation;
it gained one point, not three.
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters.
brokkr-smithy-dev independently corroborated from the other end with a **null
control** (an excerpt compared against ITSELF, where tie is definitional):
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
designed items is an easier task than ranking real text — the harness is a
**screen, not a verdict**, and its README says so.
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
## The naming rule, restated the hard way
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
> never repoint a named model at a different model's endpoint — that is
> intentionally misleading
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
moved to gen with a deterministic judge profile copied from `image-judge`.
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
promise a capability, or an identity?*
## The 7-way alias collision — the finding with the longest reach
```
chat-judge classifier gen image-judge
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
```
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
**Cross-checking a result against another alias measures nothing when they are the
same weights — agreement is an echo, not corroboration.** Documented at the head of
`model_list` in the live gateway config, because it belongs where people read it.
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
independent second opinion. They pinned the backing model in the preregistration
and dropped the second-alias idea instead.
## Provenance seam (brokkr's pushback, adopted)
The gateway returns the **alias** in the response `model` field, not the backing
model — so a per-call guard catches a swap *during* a run and is blind to one
*between* runs. **Role alias for routing, concrete model for provenance.**
`GET :4000/model/info` with the shared key already exposes backing model +
api_base; resolve at run start AND end and void on mismatch.
## Artifacts
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
would silently benchmark a retired seat).
- `stacks/selene/` keeps compose + a README explaining the retirement.
- Technique worth stealing, from brokkr: **a control constructed so the correct
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
Item vs itself; response vs its own truncation; text vs its own clauses
permuted. Add those before adding more judged items.
Commits `ca3c984`, `b8a5355`.
persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md
+50
View File
@@ -0,0 +1,50 @@
# [2026-08-23] `/mnt/smithy` mounted on ana-ml2 — read-only and SOFT, deliberately not matching nh3-dev
brokkr-smithy-dev asked for `10.100.50.50:/volume1/smithy` on ana-ml2 to run R47's
CPU-bound corpus pipeline on 96 idle EPYC cores instead of one nh3-dev vCPU. Granted,
with two deliberate deviations from what was requested.
```
sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
10.100.50.50:/volume1/smithy /mnt/smithy
```
The export already permitted ana-ml2 — no DSM change needed. Write is genuinely
refused.
## Why soft, not hard
They asked to match nh3-dev's mount, which is `hard`. **nh3-dev is same-site as the
NAS; ana-ml2 is not** — this is cross-site NFS on the box running the fleet's
inference seats. A hard mount turns a link blip into unkillable D-state, and this
fleet has already lost a host that way (esh-docker-vm; only fix was a reboot). Soft
returns EIO, the batch job fails, you rerun it. The soft-mount corruption caveat is a
**write** hazard and this is read-only. Mirrors the existing ESH books mount.
## Why not in fstab
Manual only, matching irv-ml1's `/mnt/smithy` precedent. A cross-site NFS entry in
fstab can hang boot on a GPU host with 71 days uptime. **Needs remounting after a
reboot.**
## The performance reality, measured on the same file through the same mount
```
ana-ml2 (cross-site) nh3-dev (same-site)
sequential read 24.7 MB/s 98.3 MB/s
small-file rate 45.3 files/s 34.6 files/s
```
Two different stories, and file layout decides which you get:
- **Many small records -> ana-ml2 wins on BOTH axes.** That path is bound by per-file
round-trips and NAS overhead, not bandwidth, and ana-ml2 is an idle 96-core box
while nh3-dev is a loaded 16-vCPU VM.
- **Bulk sequential streaming -> the link eats the win.** 4x read penalty against a
6x CPU gain. `datasets/raw` is 126 GB, `datasets/derived` is 1.7 GB — which one the
pipeline traverses changes the answer by two orders of magnitude. Staging a subset
to ana-ml2 local disk (195 GB free) beats pulling it over the wire repeatedly.
The 24.7 MB/s is the Anaheim tunnel, not NFS and not the NAS — see
[[2026-08-23-anaheim-ipsec-tunnel-ceiling]]. No mount tuning will move it;
parallelism will.
persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md
+70
View File
@@ -0,0 +1,70 @@
# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
## b187 pre-stage (#405 phases 1+2)
The matrix bridge stopped embedding the engine and became an HTTP client of the
Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
container env, the bridge exits by design. Demo's compose never passed it; the next
recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
up, verified with `docker compose config`, nothing restarted).
**Key decision, and I got its scope wrong first.** I argued demo should stay keyless
(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
recorded authorization, vaulted, wired, three-hop hash-verified.
## The 69-day-stale `:latest` landmine
All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
compose config rather than reading `.env`, containers untouched. Playbook at
`playbooks/repin-worldtree-image.yaml`.
**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
frozen was running a **dangling image with no repo tags**, kept alive only by the
running container. One `docker rm` from garbage collection. Tagged
`:446e5807bf43` first, then pinned.
The guard I wrote had two bugs the pinned case exposed: it compared the container's
`.Config.Image` **string** (only the tag it was CREATED from — pinned was created
from `:latest` back when that meant 446e5807), and it reported CHANGED
unconditionally. Now compares **image IDs** and skips when already correct.
## Matrix homeserver ownership
Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
instances had them at length **zero**. Someone wired four of six Matrix vars and
stopped. Wired them into personal, three-hop verified.
**The trap worth remembering:** Synapse's registration pointed at
`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
and looked correct while routing the operator's live rooms to the demo instance.
**Personal's bridge is :8010.** `docker port` is ground truth.
Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
smoke passed first try: room created, mimir accepted the invite, a real engine turn
ran, mimir replied in persona voice. #408 closed.
## Open on worldtree-dev's side
- **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
/app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
litter that compounds silently. Needs a which-container-writes-what check on the
sessions volume before anyone chowns it.
- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
listed by the engine on either instance.
- Historical Domari pairwise verdicts from the selene era are coin-flip-grade
(see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
no future arc leans on them without re-judging.
Commits `064181a`, `bb19a96`.
persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md
+50
View File
@@ -0,0 +1,50 @@
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
WAN admin was opened at the start of the session as a cutover contingency
("so I don't have to drive down there"), then closed again on operator
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
address now exposes no TCP port at all.**
## Final state
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
## Port 80 was the FortiOS ACME listener, and I got it wrong first
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
was **wrong**, and the reason is worth keeping:
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
*answers*, the filter must be bidirectional.**
The listener is opened by `config system acme / set interface "wan1"` and
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
because WAN admin is closed and the box is being replaced.
## Retracted in the same pass: the "four all-port VIPs" alarm
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
Ground-truth external scan of all six public IPs is recorded in
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
candidates for the OPNsense translation, not exposure.
**Lesson, twice in one session: measure from outside instead of parsing config.**
Both wrong answers came from a filter that answered a different question.
persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md
+167
View File
@@ -0,0 +1,167 @@
# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base
One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`,
`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`,
`019ccff`, `5415fd4`.
## 1. The seat was crash-looping, and the cause was NOT its config
`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the
time it was examined, not the 4 first reported). Startup logs looked clean all
the way through weights, `torch.compile` and CUDA-graph capture, then:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB.
GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free.
**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA
CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually
held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card
with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the
24th.
Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 →
371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the
pool still holds 1.4x a full-length sequence; what is lost is concurrent long
requests, not context.
**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH
UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB
freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the
21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done
against a long-running gen is measuring a moving number.** Measure against a
freshly-restarted one.
## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two
Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and
`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure
rates as two failing services; it was one outage sampled twice. This also
*improved* a result of theirs: their CoT on/off battery had assumed both aliases
were the same weights under two chat templates, and the routing detail turned an
assumption into a verified fact.
(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been
stopped for 12+ days and is unrelated — it is not what that alias resolves to.)
## 3. The seat swapped to the Gemma-4 26B-A4B MoE
Operator-directed straight-across replacement: same port, same
served-model-names, so no gateway route or consumer config moved. Rationale is
throughput under CoT — the user waits through the whole reasoning block before
the first visible token, and the MoE measures ~114 tok/s @32K against the dense
31B's ~40.7.
Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the
other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) —
the same loader path the outgoing seat used. Smaller weights at the same 0.47
budget bought **1,724,110 KV tokens against the predecessor's 371,023**.
`meromero-charrp` is retained stopped in `created` state, labelled
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**.
## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF
Verified by hash across every third-party Gemma-4 derivative pulled:
| build | lines | sha256 (normalised) |
|---|---|---|
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match |
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
Three independent repos carrying the *identical* stale file means it propagated
through the ecosystem. Consequences differ by use and **both are silent**:
serving a mismatched template renders a different prompt; **training through
`base/chat_template.jinja` means training on a different prompt format than
production serves** — train/serve skew, no error, presents as a tuning failure.
The production compose now pins the template explicitly. It is a **no-op for the
served weights** (the A4 build ships that exact file) and permanently closes the
class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template
default must move with it.
## 5. A benchmark result was RETRACTED — below chance indicts the instrument
A battery appeared to show Gemma at **12% contradiction detection with CoT off
against gen's 81%**. An A16 activation-precision control was staged to test
whether the quant scheme owned it. Then the operator asked to see the individual
items, and the item was **ill-posed**: it presented two mutually contradicting
statements and asked for "*the* contradicting statement", but **contradiction is
symmetric**. The model consistently named the absolute claim — a defensible
reading the labelling scored wrong every time.
**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20%
CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts
the model, and neither side reacted to it. I spent the afternoon verifying repo
names, config fields, template hashes and tokenizer vocabs — every layer of
plumbing — and never asked whether the number itself was *possible*. **A
preflight can be thorough and still be aimed in the wrong direction.**
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (on a sound instrument it **reverses**); all
pre-fix T2 numbers. Recorded as a dated superseded-claims table in
`stacks/gemma4-charrp/README.md` rather than a silent edit.
**What survived:** the A16 control result — activation precision is close to free
on this battery, every other task identical across W4A4 and W4A16 builds.
## 6. The abliterated trainee base — measured, not assumed
Operator directed a low-damage abliterated instruct build. "Low damage" was
treated as a measurable claim; the field spreads from KL 0.09 to 0.4118:
| build | method | KL | refusals |
|---|---|---|---|
| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset |
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
| huihui-ai | remove-refusals | none published | none published |
Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at
83.7% acceptance**, so both staged builds sit inside an already-measured band.
huihui-ai rejected — no metrics, its card calls the method "a crude,
proof-of-concept implementation", it abliterates both thinking and non-thinking
modes, and its parameter count runs ~738M over upstream. Operator's independent
read matched ("huihui produces garbage").
**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision,
same pinned template, same 192 items):
T2 contradiction 75% → 59% (−5 items)
T6 spatial 75% → 88% (+4 items)
core 90.0% → 89.4% (−0.6 pts)
**It MOVED capability rather than removing it** — five lost on contradiction,
four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46
stands**; no case for re-staging on TrevorJS over 0.6 points.
⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing
about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32,
different item sets, n-confounded.
## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors`
Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately:
TypeError: CompressedTensorsConfig.__init__() missing 3 required
positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
vLLM trying to read a quantization config out of a checkpoint that has none. 35
restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate
ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot
resurrect itself and block gen's restore, and no homepage labels so it leaves no
permanently-offline card. That detour is why a base swap is now ~5 minutes
instead of ~15.
## 8. BF16 cannot coexist with gen
48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before
a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such
windows were run and gen restored both times; the restore was triggered by
observing the seat's own throughput logs (a large prefill burst then zero
running/zero waiting) rather than waiting on a courtesy message.
Cross-links: [[2026-08-24-homepage-uniform-grid]]
persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md
+77
View File
@@ -0,0 +1,77 @@
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
wrong turns worth recording.
## Root cause
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
entirely**, so every host there got the UDM and could never resolve `.internal`
— esh-docker-vm was living proof.
## The three wrong turns
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
explicit server is supplied**. Setting it bare looks like a no-op and invites
the conclusion that the field is inert — which I drew, wrongly. Despite the
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
AdGuard persists, reads back, and changes nothing. Proven with **fresh
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
real IPs. Reverted.
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
known `cmd/devmgr` behaviour.
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
probed with a stdlib raw-socket Router Solicitation parsing option type 25
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
## What landed
RDNSS **redirected** rather than disabled — better than switching it off:
| VLAN | v4 | v6 RDNSS |
|---|---|---|
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
accepts it and never retries. A secondary that doesn't know your private zone is
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
camera DNS through AdGuard's filtering risks their cloud features.
## The naming scheme became real
The resolver address is the scheme's first live use, replacing a MAC-derived
SLAAC address that would break on a NIC change. All three `esh-server` Linux
hosts now carry `4411:B105` ("FOR ALL BIOS"):
```
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
esh-vm-db …:4411:b105:50:60
```
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
a blocking hook would stall bring-up on a headless box), and adds nothing to
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
`accept_ra=0` and would strand the host.
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
on every Proxmox node when its LAN gets v6.
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
auto-memory `reference_unifi_dns_rdnss_limits`.
persistent-memory.d/2026-08-24-homepage-uniform-grid.md
+308
View File
@@ -0,0 +1,308 @@
# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid
Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and
can absolutely consume their labels, please verify again. I am still
unsatisfied with the layout and aesthetics."*
## The verification — the operator was right, and the record now says so
**Homepage on `esh-docker-vm` discovers services by container label from all
five Docker engines in `conf/docker.yaml`, not just its own.** This is not an
inference; `GET /api/services` returns every card's `server` field, and the
2026-08-24 snapshot resolves to:
| `server` | host | label-discovered services |
|---|---|---|
| `ana-pfi-docker` | 10.250.50.70 | 30 |
| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 |
| `ana-ml2-docker` | 10.250.50.54 | 14 |
| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 |
| `nh3-pfi-docker` | 10.100.50.40 | 2 |
**74 of 107 cards are label-discovered, and only 13 of those come from the
dashboard's own engine** — the other 61 are read off four remote hosts,
including irv-ml1 across the WireGuard tunnel. The remaining 33 carry
`server: null`: those are the manual `services.yaml` entries — hardware, BMCs,
hypervisors, printers, and user-level systemd services that have no container
to label in the first place. **That null is the only thing "not label-driven"
about this dashboard**, and it is a property of the entry, not of the host it
points at.
⚠ If a future session doubts this again, the check is one command and takes two
seconds — do not reason about it from the docs:
```bash
curl -s http://10.0.50.45:5100/api/services \
| jq -r '.[].services[] | .server' | sort | uniq -c
```
## What was actually wrong with the layout
Measured with Playwright against the live board (per-group `card=` width, card
height spread, and a geometric title-vs-status overlap test), not judged by
eye:
- **Card width changed at every group boundary.** `columns:` is not a density
dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card
width. Notes rendered a single **1464px** card; News and Media **728px**;
Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the
grid resized five times.
- **Long names printed underneath their own status pill.** Measured by
re-injecting the old rule and testing the title text node's box against the
status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in
Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why
it survived earlier passes. Root cause is a genuinely counter-intuitive one:
the rule reserved a
78px gutter with `padding-right` and relied on `overflow: hidden` to hold it,
but **overflow clips at the padding box, not the content box** — so the
reserved gutter was spill room the title rendered straight through. The
intended `text-overflow: ellipsis` never fired either, because the ellipsis
is painted by whichever block's own line overflows, and here that is the
anonymous box around the bare title text node, which does not carry
`overflow`.
- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox
bug recurring, this time arriving from a container label rather than from
`settings.yaml`.
- **Icons were grey smudges.** Homepage masks every glyph over
`--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half
of each icon into the card fill.
- Bookmark groups and Jellyfin's trailing stream rows were the two components
the theme had never reached.
## The fixes
`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.**
`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`:
gutter held by wrapping, description clamped to 3 lines (floor still 2), icon
ramp overridden, bookmark + trailing-widget components themed, group gap
10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio
Tools`, container recreated on ana-ml2.
After: **every group renders at card=360**, and the collision count is zero.
Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/`
(24h TTL; also on the standing link board).
## ⚠ Three traps worth carrying forward
1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was
avoiding dead cells in a short last row and bought a worse defect. A short
last row is what a grid looks like; a card wider than its neighbours is what
a mistake looks like.
2. **A `:root` override of a Homepage theme variable is silently ignored.**
Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the
`<html>` element — the same element `:root` matches. `.theme-slate` (0,1,0)
beats `:root` (0,0,1), so the override does nothing and looks like the
variable is not the one in play. `html[class]` (0,1,1) wins, and does not
hard-code which `theme-*` class is active. Specificity alone is not enough
either: a custom property resolves from the *nearest* ancestor that sets it,
so the override has to land on `<html>`, not on `<body>`.
3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The
2026-08-19 note reads as though every recreate costs up to an hour of broken
render. This recreate came up correct within 10 seconds — fresh payload on
the first poll, all four tabs clickable a minute later. Recreate, *check*,
and only then walk away if it is actually in the broken state.
Also re-confirmed, since the change depended on it: **a `settings.yaml` edit
needs a container recreate, not a restart.** `docker restart homepage` left the
old `"columns":1` payload embedded in the served HTML with the correct file
mounted and readable inside the container; `compose up -d --force-recreate`
cleared it immediately.
## Deliberately not done — operator's call
The Main tab still opens on three sparse bands: **Notes** (1 member) and
**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes +
News, or folding Games into Apps, would tighten the top of the page — but that
is information architecture, not layout, and the group names are the operator's.
Surfaced rather than done.
→ **Resolved in pass 2 below**, where the operator delegated the naming
("re-categorize however you want"). Notes + News became `Daily`, Games folded
into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded
(Scriberr moved on to `AI - Studios`).
---
# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed
Operator, after seeing pass 1: *"You can re-categorize however you want.
service networking tab is uneven, you can split out the adguard cards, etc.
most of the issues are that tools I use and have a UI are interspersed with API
endpoints which are largely informational only. They might even go in their own
cards or start collapsed."*
## The axis
Every group is now either **tools** (expanded, top of tab) or **endpoints** (an
API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A
collapsed group still renders its eyebrow and rule, so presence costs one line
instead of two rows.
Second, quieter rule that fell out of the same pass: **a group's members should
all carry a widget or none should.** A stat strip adds ~50px, so one widget card
in a row of plain ones opens a void under the plain ones — which is most of what
made the 13-member `Service Networking` band look broken.
## Shape
- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the
1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the
1-card `Games` band), `Media`, `UltraSeedbox`.
- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then
`AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4),
`AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed.
- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2),
`Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed).
- **Infrastructure** — unchanged; every card there is already a console.
Measured after: every group `card=360`, and `DNS & Filtering` and
`Reverse Proxies` both `h=134..134` — dead flush.
## ⚠ The move that made it affordable
**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is
read at container **creation**, so renaming `AI - Inference` to something
clearer would have meant recreating six vLLM seats plus four eval seats plus
four TTS engines — multi-minute model reloads on endpoints peers reach through
the gateway. Order plus `initiallyCollapsed` buys the same separation for free,
so the names stay ugly on purpose. **Do not spend that recreate on a label.**
28 containers *were* relabelled — all cheap web services — via five rerunnable
elway playbooks, `playbooks/homepage-regroup-<host>.yaml`. The canonical
`stacks/` tree was synced to match afterwards, so intent and reality agree.
`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in
this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in
`/app/.next/server/pages/index.js`).
## AdGuard (ANA) gained its widget, and the credential is fleet-wide
It was the only AdGuard without a query/blocked/latency strip, so it sat short
beside two tall siblings. **One `infra-ops` AdGuard login authenticates against
all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200
on `POST /control/login`, verified 2026-08-24). Vaulted at
`secret get nh3-dev/adguard-infra-ops-password`; written to
`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon
was also the odd one out (`mdi-dns` against two `si-adguard`).
## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION
The biggest durable finding of the day, and it cost ~25 minutes. Full write-up
in `stacks/homepage/README.md`; the short version:
`initialSettings":{}` in the served HTML is **the catch branch** of the page's
data loader, not a warm-up and not a cache. And the error can vanish without
trace: the logger is assigned as the first statement *inside* the same `try`,
and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is
written anywhere — which is exactly what was observed.
Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`,
`/api/widgets` and `/api/hash` all return **200 with correct content** while the
page serves `{}`; restoring the previous known-good `settings.yaml` reproduces
it identically; `/api/validate` returns `[]`; disk and permissions are fine.
**One-command test:**
`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'`
**What broke the streak:** three consecutive recreates came up empty, then
rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed
it within 15 seconds. That is one observation, not proof — but it is a coherent
mechanism (oversized log → logger init throws → silent catch) and it is the
cheapest thing to try first next time.
---
# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped
Operator supplied the Skyfall design-system README and said "Go full with
skyfall."
## The bundle was already in this repo's git history
**The Skyfall tokens did not need to be hunted down.** A predecessor vendored
them on 2026-08-19 and a later commit deleted them; git kept everything:
```bash
git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes
git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens
git show 45c1995:stacks/homepage/theme/typography.css
git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2
```
`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall
Day) — so the light ramp is canonical, not derived. That killed the entire
objection from the previous answer, which was correct only about the
`australis-design` skill ("Always dark first. No light mode in this system").
**Skyfall is the dual-theme derivative; australis-design is the terminal
theme. They are different systems and only one of them has a light mode.**
## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD
The documented way to get Homepage's own light/dark toggle is to leave `theme:`
unpinned. **Do not.** With the key absent, the page's data loader throws and its
catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n.
Measured, not inferred: six force-recreates over seven minutes all came up
empty with the key removed; restoring `theme: dark` rendered correctly on the
next recreate in **12 seconds**. `/api/services` stays 200 and fully correct
throughout, which is exactly why this reads as a caching or warm-up problem and
is not one.
This is the first *confirmed* trigger for the long-running "tab bar goes
missing" mystery. It does not explain every occurrence (the symptom has
appeared with `theme:` present), but it means **the first diagnostic step is
now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container
archaeology. Also retires an earlier lead from this same session: rolling the
8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing
during the `theme:`-key episode — coincidence, not cause.
## So the toggle is ours
`conf/custom.js` renders it (was an empty placeholder). Precedence:
1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle;
2. OS preference — `@media (prefers-color-scheme: light)`;
3. dark — Skyfall's default.
`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as
`[data-theme="light"], html.light`, and inside the media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair
is what lets a stored *dark* choice survive a light-mode OS.** Verified across
both OS preferences: load, click, click again, reload — all four correct.
⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `<html>`
regardless, because `theme:` is pinned. That is fine and was checked
explicitly: with the dark class present AND `data-theme="light"`, every themed
surface resolves to Skyfall Day, because our rules carry `!important` on the
surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is
the control surface; the class is not.**
## The anti-fork guard is now mechanical
`build.py` records the SHA-256 of each vendored file and **fails the build** on
a mismatch, rather than warning. A vendored file is either byte-identical to
the bundle or it is a fork wearing the bundle's name. Overrides go in
`skyfall.css.in`, which is written entirely against the semantic layer
(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no
raw family tokens, no colour literals.
The one place a literal is unavoidable: Homepage consumes
`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`.
Those four values are exact sRGB conversions of real tokens (`--sea-80`,
`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather
than eyeballed, with the conversion recorded in the file.
## Deviations, all deliberate and all written down
- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two
accent expressions — the active rail and hero-only glows — and a decorative
gradient across the chrome is neither. The colour moved to a 2px accent bar
plus `--accent-soft` fill on the active tab, which *is* the rail.
- **Widget stat values moved from the display face to mono**, per Skyfall's
"numbers and telemetry are always `--font-mono`".
- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono
for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call
Victor Mono "user-supplied". Two-line swap when the real faces arrive.
Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/`
persistent-memory.d/2026-08-24-scriberr-ana-ml2.md
+46
View File
@@ -0,0 +1,46 @@
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
Self-hosted audio/video transcription + diarization. Operator chose GPU
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
canary voxtral openai`. ~30 GB of weights on `/tank`.
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
`reference_scriberr_ana_ml2`.
## Three upstream bugs, none of them ours
**1. The Blackwell image does not exist.** Upstream's README documents
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
back to the published image.
**2. It must run as uid 10001, not 1000** — and the error lies:
`unable to open database file: out of memory (14)`. Error 14 is
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
it, while the entrypoint's PUID remap covers only the data dirs.
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
mount; fails on a plain named volume too → not the storage; the **published CPU
image works at PUID=1000** because there `appuser` *is* 1000.
Generalisable: *when a container "permission" bug appears, compare the uid the
image was BUILT for against the uid you are RUNNING as.*
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
## Related
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
~5.9 GB quoted — that figure is the loaded-model working set.
persistent-memory.md
+51 -31
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-08-22_
_Last updated: 2026-08-25_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,24 +108,53 @@ no longer deployed sidecars here. See Recent decisions.)
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
## Current state / in-flight
_As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that moved this session._
_As of 2026-08-25 ~04:20Z — the ERP/RP tune is TRAINING on ana-ml2 GPU0, ~17h, unattended. The homepage and char-rp arcs closed earlier. **The live thread is the run itself plus a parallel question: whether a fused MoE kernel lands fast enough to justify restarting it.**_
- **SEAT MAP.** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, 7 aliases, **still on the OLD nightly `311b3513` with MTP k=3**. **`char-rp`** = MeroMero-v2 dual-mode (prose + streaming CoT, one weight set, two aliases), GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019 — **rebuilt this session, see below**.
- **🟢 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h → ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2×accum8 → 1,312 steps, r64/α128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 — ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case — adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** — that has mattered five times. **RESUME: use `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch command** — that one starts `rm -rf /tank/erp-tune/run-01`, which destroys the 609 MB encode cache AND every checkpoint. First checkpoint at step 100; `save_steps=100` at ~46.5 s/it = **~73 min of crash exposure** per interval. → `docs/pfi/gemma4-erp-tune-sizing.md`
- **⚠ MFU IS 8.6% AND I HAVE DISPROVEN MY OWN HYPOTHESIS TWICE — CONSULT OUT TO THE FRONTIER DWARVES.** 27.1 TFLOPS against a **benchmarked 313.8 TFLOPS** peak; one fwd+bwd at the real shape is **34.85s** (4 passes within 1%). **RULED OUT, with numbers, not argument:** (1) **hardware** — a plain dense GEMM hits **97.1% of peak** (304.6 TFLOPS), card draws 279-292W of 300W; (2) **the Python expert loop** — swapping to transformers' `grouped_mm` experts backend gave **35.149s vs eager's 34.847s, bit-identical output (max_abs_diff EXACTLY 0.0), same 75.8 GiB**, and torch 2.13 HAS both `F.grouped_mm` and `torch._grouped_mm`, so it is not a missing kernel; `batched_mm` both OOMs and MISMATCHES (rel 0.79 — it computes all 128 experts per token); (3) **MoE being the bottleneck at all** — isolated at real shapes the MoE block is **37.54 ms at 26.5% of peak**, of which **13.39 ms is pure gather/scatter dispatch** and a dispatch-free `bmm` version would be **12.28 ms at 80.9% of peak** — but **30 layers × 37.54 ms × 3 (fwd+recompute+bwd) ≈ 3.4s of a 34.85s step, only ~10%.** Making MoE free buys ~7%. **~90% of the time is somewhere I have not looked.** ⚠ **LEADING UNTESTED HYPOTHESIS: the 5 `full_attention` layers use `global_head_dim: 512`, and FlashAttention-2 caps head_dim at 256** — if that pushes torch SDPA onto the mem-efficient or math backend, 5 layers are doing O(n²) attention at seq 16384 on a slow path. Other un-excluded candidates: the chunked CE (vocab 262,144 + softcap, 1024-tok chunks re-materialised under `checkpoint`), the `attention_k_eq_v` K=V path, grad-ckpt × MoE dispatch interaction, PEFT's wrapper on 205 modules. ⚠ **My earlier "5% MFU" was ALSO wrong** (divided by UNPADDED tokens, compared against a GUESSED peak) — operator caught it. Padding is a real but secondary **29.9%** tax (82,337,318 padded vs 57,733,156 real). Artifacts: `/tank/erp-tune/{micro_moe,bench_moe,bench_bf16}.py`. → park id 47, althing thread `01M0VKBPZD71Q302NH84BXHTWS`
- **🟢 `sec` NOW RUNS DFLASH2 ON A NEWER vLLM — promoted to its compose stack after real-use testing.** `nightly-e9d1398d` (+259 commits over production, `behind_by=0`), `dflash` k=7 with the 3.85 GB drafter, **util 0.52 / max-model-len 420,000 / KV ~453k**, 2048² vision. `restart: unless-stopped`, survives reboot. Canonical in `stacks/mog-sec/` with a fully-commented `.env.example`. **ROLLBACK:** `.env.bak-pre-dflash2-20260822` on the host, or swap `MOG_SPEC_CONFIG` + `MOG_IMAGE`. ⚠ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` was **deliberately dropped** — the validated container never had it.
- **⚠️ THE `sec` DEGENERATION QUESTION IS OPEN AND CONFOUNDED.** It no longer degenerates, but **engine and drafter changed together**. **The isolating experiment is MTP k=3 on `e9d1398d`** — not yet run. Operator ruling: the degeneration lives in the **un-fixed vLLM**, not the weights; my MTP-head hypothesis is **retracted**. ⚠⚠ **Both the "degenerates at 2k" and "coherent to 10k" sightings are n=1 from uncontrolled sessions and are NOT evidence.** Production is **172 commits behind GDN spec-decode fix #53077**; `#51113` is present in both builds and is **necessary but insufficient**.
- **⏳ `gen` IS UNTOUCHED and still on the old build.** If DFlash2 + the newer engine are the answer, gen is the obvious next beneficiary — but that decision is gated on the isolating experiment above, not on sec's n=1 result.
- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` (SSID `PVC`) and `esh-server` from a delegated `2607:73c0:402:1d00::/56`; hosts egress over v6 as themselves, un-NATted. **v4 remains CGNAT (`100.104.3.250`) and a full gateway reboot proved the purchased static is NOT provisioned** — carrier ticket, nothing left to try locally. v6 firewall audited: default-deny inbound both versions, correct. NH3 stays v6-off deliberately (single /64 reserved for meshing). Flat-zone lateral-movement finding **parked, id 44**.
- **🟢 OTHER SERVICES.** speaches ASR live irv-ml1:8204 (Eyra; loop closed). Open WebUI on esh-docker-vm:3211 (admin creds + admin-scoped API key vaulted; **Lobe retirement still the operator's call**). Waterland, Homepage/Skyfall, fleet `.internal` DNS all landed earlier and are stable.
- **⏳ OPEN:** the MTP-k3-on-new-build isolating experiment; file the drafted upstream vLLM issue (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift (`:main` vs pinned v0.11.0); `/tank` DEGRADED **70+ days**; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; **MANY commits unpushed** — push is the operator's call.
- **🛑 THE CORPUS GATE — OVERRIDDEN FOR THIS ONE RUN ONLY (operator, 2026-08-25).** Grant staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/TRAINING-ELIGIBILITY-OVERRIDE.md`. ⚠ It does NOT flip any root's `training_eligible` flag — they still read `false` and name both blockers, deliberately, so the signal survives. **A second run needs a second grant.** Provenance records `training_eligibility_override: operator-2026-08-25-rnd-run` + both blockers + both substitute controls; those keys are in `REQUIRED_PROVENANCE` as present-with-explicit-null so a future run cannot silently omit them. Background: Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter §3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** — it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev — who WROTE those fields — corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). → `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** — the survivor list the harness gates on — so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.**
- **🟢 SIZING + SEAT CALL — DONE AND EXECUTED, full detail in the doc.** QLoRA structurally unavailable (fused 3-D experts vs bitsandbytes' nn.Linear walk); plain BF16 LoRA; chunked CE mandatory (naive CE OOMs at seq16384, 81.93 GiB at seq8192); `v_proj` exists on only 25 of 30 layers (`attention_k_eq_v`, K=V sharing — real, not a miss). `gen` moved to GPU1, `sec` down, GPU0 dedicated. → `docs/pfi/gemma4-erp-tune-sizing.md`, `playbooks/ana-ml2-training-window-{open,close}.yaml`
- **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.
- **🟢 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
- **🟢 THREE trainee-relevant model dirs on `/tank/aimodels/`, NOT interchangeable:** `gemma4-26b-a4b-it-bf16` (stock, 49 GB — its chat_template is the canonical upstream one), `gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated, the trainee), `gemma4-26b-a4b-it-abliterated-bf16` (TrevorJS, KL 0.09, alternate). Plus `-nvfp4` (served) and `-nvfp4a16` (activation control). ⚠ **BF16 cannot coexist with `gen`** — 48.07 GiB of weights on a 94.97 GiB card. Every BF16 window means gen stops.
- **🟢 `stacks/gemma4-trainee-bench/`** is the ephemeral BF16 bench stack — no `--quantization` flag (the production compose hardcodes `compressed-tensors` and crash-loops on BF16), `restart: "no"`, no homepage labels. Base swap is ~5 minutes because it exists.
- **🎨 Homepage runs AUSTRALIS SKYFALL with a working light/dark toggle**, recategorised on "do I open this?" (TOOLS expanded / ENDPOINTS collapsed). ⚠ **`theme:` MUST stay pinned in settings.yaml** — removing it makes the page loader throw and serve `initialSettings: {}`, the first *confirmed* trigger for the "tab bar goes missing" mystery. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- **🔒 ana-gw's public admin surface is ZERO open TCP ports**; box scheduled for replacement by **OPNsense on a Dell R420** (brings WireGuard onto the edge — the downstream-WireGuard-VM design is moot, do not scope it). **No out-of-band path remains** — if both tunnels drop it is console-only. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
- **🟢 Both Anaheim IPsec tunnels run AES-128.** NH3 245→**270 Mbit/s**, ESH 268→**304**. Ceiling is **the UDM's software AES-CBC, not the FortiGate**. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
- **🟢 Scriberr LIVE** — ana-ml2 **GPU1** :8080, built locally, uid **10001**, needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
- **🟢 ESH DNS fixed at the IPv6 layer**; RDNSS **redirected** to AdGuard. ⚠ Proxmox bridges need `accept_ra=2`. Naming scheme lives in `docs/pfi/ipv6-naming-scheme.md` — **a convention, not memory state; never let a memory line be the only copy again.** → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
- **🟢 SEAT MAP.** ⚠ **ana-ml2 runs a vLLM VERSION SPREAD, not one version** — do not say "ana-ml2 runs X". Measured 2026-08-24: `gen` **0.27.2rc1.dev150** (`nightly-311b3513`), `mog-sec` **0.26.1rc1.dev1102** (`nightly-e9d1398d`), `rerank-a3`/`coder`/`reward`/`embed` **0.24.0**, char-rp + trainee-bench pinned **v0.26.0**. `v0.27.1` (tagged) and three nightlies sit on disk unused. **`gen`** = Qwen3.8-27B-Uncensored NVFP4-mixed, GPU0 :8015, 7 aliases, UP. **`char-rp`** = Gemma-4 MoE NVFP4, GPU0 :8016, DOWN deliberately. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019, sharing GPU1 with Scriberr.
- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Isolating experiment is **MTP k=3 on `e9d1398d`** — still not run. Operator ruling: degeneration lives in the **un-fixed vLLM**, not the weights; MTP-head hypothesis **retracted**. Both sightings n=1.
- **🟢 ana-ml2 mounts `/mnt/smithy`** ro + soft, **NOT in fstab** — manual remount after reboot. `nconnect=8` approved but deliberately not applied. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
- **🟢 ESH IS DUAL-STACK**; v4 static is an unprovisioned Cityside ticket. **NH3 stays v6-off by explicit ruling.**
- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; Lobe retirement is the operator's call; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** — push is the operator's call.
- **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24). Direct task correspondence with a counterparty is fine; unsolicited broadcasts are not.
## Recent decisions
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation.
- `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `<html>` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md`
- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.)
- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink <id|index>` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`)
- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`)
- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
@@ -157,7 +186,7 @@ _As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that move
- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates exactly ONE /64** (`2600:1700:b25:c11f::/64`) — proven by forcing prefix-ID auto→`0` and watching the subnet NOT move, because the `c110`/`c11f` pattern otherwise reads convincingly as a /60. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates one /64 PER REQUEST** (`2600:1700:b25:c11f::/64`) — and the BGW holds the whole `/60`, rationing `c118`–`c11f` one at a time while keeping `c110`–`c117`. So eight /64s exist; UniFi just solicits once. ⛔ **CLOSED 2026-08-24 — operator ruling, do not re-raise:** the BGW has **no IP-passthrough** (operator confirmed, and we have admin on it), so the only route to the other seven is a multi-DUID DHCPv6 client on a VM — which means split-stack routing and rebuilding the entire v6 firewall policy off the UDM. Juice not worth the squeeze. NH3 LANs stay v6-off. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
@@ -225,21 +254,6 @@ _As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that move
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]]
- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]]
- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
@@ -253,9 +267,15 @@ _As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that move
_209 older entries archived to archival-memory.md._
_214 older entries archived to archival-memory.md._
## Tried and abandoned
- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
- `[2026-08-15]` **Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless `re:^mtp.*` is in the quant-config `ignore`; and W4A16=Marlin (not native FP4) costs ~20% even on decode.** Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
playbooks/ana-ml2-training-window-close.yaml
+175
View File
@@ -0,0 +1,175 @@
# ana-ml2 — CLOSE the ERP/RP tune window: put the fleet back the way it was.
#
# gen GPU1 -> GPU0 -> start mog-sec back onto GPU1
#
# The exact inverse of playbooks/ana-ml2-training-window-open.yaml.
#
# ⚠⚠ ORDER IS LOAD-BEARING, AND IT IS THE MIRROR OF THE OPEN ORDER.
# `gen` must vacate GPU1 BEFORE mog-sec is started. mog-sec runs at
# --gpu-memory-utilization 0.52 = 50,901 MiB that must be free at startup. With
# gen still resident on GPU1 only ~30,000 MiB is free, so mog-sec would fail to
# boot. gen moves back to the (empty) GPU0 first; step 4 waits for GPU1 to
# actually release before mog-sec is started at all.
#
# ⚠ FIRST STEP IS A GATE, NOT A COURTESY. If a training process is still
# resident on GPU0 this playbook REFUSES to run — moving gen back would either
# OOM the run or OOM gen. Override only when you have confirmed the run is
# finished or deliberately abandoned:
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-close.yaml \
# --var allow_busy_gpu0=true
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-close.yaml
vars:
gen_dir: /opt/docker/compose/gen-seat
mog_dir: /opt/docker/compose/mog-sec
gen_port: "8015"
mog_port: "8019"
# mog-sec's startup requirement: 0.52 x 97,887 MiB, rounded up.
mog_required_free_mib: "50950"
# Set to "true" to close the window even with a process still on GPU0.
allow_busy_gpu0: "false"
steps:
- name: "GATE — GPU0 is idle (refuses to evict a training run mid-flight)"
sudo: true
shell: |
set -e
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
n=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
echo "GPU0 compute procs: $n"
if [ "$n" -eq 0 ]; then exit 0; fi
if [ "{{ allow_busy_gpu0 }}" = "true" ]; then
echo "GPU0 still busy but allow_busy_gpu0=true — proceeding under override"
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv | grep "$gpu0_uuid" || true
exit 0
fi
echo "REFUSING: a process is still resident on GPU0. Confirm the tune has"
echo "finished, then rerun with --var allow_busy_gpu0=true"
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv | grep "$gpu0_uuid" || true
exit 1
changed_when: "false"
- name: "PREFLIGHT — record gen's current container id (proves the recreate later)"
sudo: true
shell: docker inspect vllm-gen --format '{{.Id}}' | tee /tmp/gen-container-id-before.txt
changed_when: "false"
- name: "Point gen back at GPU0 in its .env (GEN_GPU_ID 1 -> 0)"
sudo: true
# ⚠ `sudo` INSIDE the when: expression — a step's `sudo: true` does NOT
# cover its guards, and the root-only .env makes an unsudo'd grep exit 2,
# which silently skips the step. See the open playbook for the incident.
shell: sed -i 's/^GEN_GPU_ID=1$/GEN_GPU_ID=0/' {{ gen_dir }}/.env
when: "sudo grep -qx 'GEN_GPU_ID=1' {{ gen_dir }}/.env"
- name: "Assert the EFFECTIVE device id, not the .env line"
sudo: true
# ⚠ Parse the JSON, do not regex the YAML — compose emits `- "0"` with
# DOUBLE quotes. See the open playbook for the incident.
shell: |
docker compose --project-directory {{ gen_dir }} config --format json \
| jq -e '.services["vllm-gen"].deploy.resources.reservations.devices[0].device_ids == ["0"]'
changed_when: "false"
- name: "Recreate gen onto GPU0"
sudo: true
shell: docker compose --project-directory {{ gen_dir }} up -d vllm-gen
- name: "Wait for gen to serve /health on GPU0"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ gen_port }}/health; then
echo "gen healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: gen did not become healthy in 900s"; exit 1
changed_when: "false"
- name: "Wait for GPU1 to release gen's memory before mog-sec is started"
sudo: true
shell: |
for i in $(seq 1 60); do
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
if [ "$free" -ge {{ mog_required_free_mib }} ]; then
echo "GPU1 free: ${free} MiB"; exit 0
fi
sleep 2
done
echo "TIMEOUT: GPU1 free is ${free} MiB, need >= {{ mog_required_free_mib }}"; exit 1
changed_when: "false"
- name: "Start mog-sec back up on GPU1 (`sec` / `sec-reasoning`)"
sudo: true
shell: docker compose --project-directory {{ mog_dir }} start vllm-mog-sec
when: "! docker inspect -f '{{.State.Running}}' vllm-mog-sec 2>/dev/null | grep -q true"
- name: "Wait for mog-sec to serve /health"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ mog_port }}/health; then
echo "mog-sec healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: mog-sec did not become healthy in 900s"; exit 1
changed_when: "false"
verify:
- name: "gen was genuinely RECREATED (container id changed)"
sudo: true
shell: |
before=$(cat /tmp/gen-container-id-before.txt)
after=$(docker inspect vllm-gen --format '{{.Id}}')
echo "before=${before:0:12} after=${after:0:12}"
test "$before" != "$after"
changed_when: "false"
- name: "gen's process is resident on GPU0 again"
sudo: true
# ⚠ Match by CGROUP, not by `.State.Pid` — vLLM V1's EngineCore is a CHILD
# of the container's pid 1, and it is the child nvidia-smi reports.
shell: |
cid=$(docker inspect vllm-gen --format '{{.Id}}')
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
found=0
for p in $(nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv,noheader \
| grep "$gpu0_uuid" | cut -d, -f2 | tr -d ' '); do
if grep -q "$cid" /proc/$p/cgroup 2>/dev/null; then
echo "gen pid $p resident on GPU0: $(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | grep "^$p,")"
found=1
fi
done
test "$found" -eq 1
changed_when: "false"
- name: "gen answers a real completion"
sudo: true
shell: |
. {{ gen_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ gen_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${GEN_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "sec answers a real completion"
sudo: true
shell: |
. {{ mog_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ mog_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${MOG_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "Both cards are back to their normal tenancy"
sudo: true
shell: |
nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv
changed_when: "false"
playbooks/ana-ml2-training-window-open.yaml
+182
View File
@@ -0,0 +1,182 @@
# ana-ml2 — OPEN the ERP/RP tune window: clear GPU0 completely.
#
# stop mog-sec (GPU1) -> move gen GPU0 -> GPU1 -> GPU0 empty for training
#
# Operator call 2026-08-24: rather than train beside `gen`, move `gen` off GPU0
# entirely and stand `sec` down for the night. Training then gets a whole card
# (95.60 GiB) instead of a shared one, and the fleet's general seat never goes
# dark beyond its own restart.
#
# ⚠⚠ ORDER IS LOAD-BEARING — DO NOT REORDER THE STEPS.
# `gen` runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction of
# TOTAL card memory: 0.43 x 97,887 MiB = 42,091 MiB that must be FREE at startup
# or the engine refuses to boot. GPU1 has only 19,446 MiB free while mog-sec is
# up. Recreating `gen` onto GPU1 first would take the fleet's main seat down and
# leave it down. mog-sec stops FIRST, and step 3 hard-gates on the freed memory
# before `gen` is touched at all.
#
# ⚠ `stop`, never `down`. `down` removes the container; `stop` leaves it in
# place so the close playbook can `start` it. Both seats are `restart:
# unless-stopped`, which does NOT resurrect a deliberately-stopped container.
#
# ⚠ device_ids vs nvidia-smi ordering was VERIFIED on this host, not assumed:
# gen (GEN_GPU_ID=0) reports under the GPU nvidia-smi indexes 0, mog-sec
# (MOG_GPU_ID=1) under index 1. They agree here. (They do NOT on irv-ml1 —
# never carry that assumption between boxes.)
#
# Restore with: playbooks/ana-ml2-training-window-close.yaml
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-open.yaml
vars:
gen_dir: /opt/docker/compose/gen-seat
mog_dir: /opt/docker/compose/mog-sec
gen_port: "8015"
# gen's startup requirement: 0.43 x 97,887 MiB, rounded up. If GPU1 has less
# than this free, gen will not boot and the window must not proceed.
gen_required_free_mib: "42100"
steps:
- name: "PREFLIGHT — GPU0 holds vllm-gen and nothing else unexpected"
sudo: true
shell: |
set -e
procs=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | wc -l)
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
gpu0_procs=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
echo "GPU0 compute procs: $gpu0_procs (total on box: $procs)"
test "$gpu0_procs" -le 1
changed_when: "false"
- name: "PREFLIGHT — record gen's current container id (proves the recreate later)"
sudo: true
shell: docker inspect vllm-gen --format '{{.Id}}' | tee /tmp/gen-container-id-before.txt
changed_when: "false"
- name: "Stop mog-sec (the `sec` / `sec-reasoning` seat) — frees ~55.3 GiB on GPU1"
sudo: true
shell: docker compose --project-directory {{ mog_dir }} stop vllm-mog-sec
# Skip if already stopped, so the playbook is rerunnable.
when: "docker inspect -f '{{.State.Running}}' vllm-mog-sec 2>/dev/null | grep -q true"
- name: "Wait for GPU1 memory to actually release (teardown is not instant)"
sudo: true
shell: |
for i in $(seq 1 60); do
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
if [ "$free" -ge {{ gen_required_free_mib }} ]; then
echo "GPU1 free: ${free} MiB"; exit 0
fi
sleep 2
done
echo "TIMEOUT: GPU1 free is ${free} MiB, need >= {{ gen_required_free_mib }}"; exit 1
changed_when: "false"
- name: "HARD GATE — GPU1 has room for gen's 0.43 budget before we touch gen"
sudo: true
shell: |
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
echo "GPU1 free ${free} MiB vs required {{ gen_required_free_mib }} MiB"
test "$free" -ge {{ gen_required_free_mib }}
changed_when: "false"
- name: "Point gen at GPU1 in its .env (GEN_GPU_ID 0 -> 1)"
sudo: true
# ⚠ `sudo` INSIDE the when: expression. A step's `sudo: true` covers the
# shell, NOT its when/creates/changed_when guards — those run as the login
# user. The .env is root-only 0600, so an unsudo'd grep exits 2
# (permission denied), which is not 0, so the step SILENTLY SKIPS and the
# flip never happens. Caught 2026-08-24 by the effective-value assert below.
shell: sed -i 's/^GEN_GPU_ID=0$/GEN_GPU_ID=1/' {{ gen_dir }}/.env
when: "sudo grep -qx 'GEN_GPU_ID=0' {{ gen_dir }}/.env"
- name: "Assert the EFFECTIVE device id, not the .env line"
sudo: true
# grep on the .env proves a substring is present; only `compose config`
# proves what the container will actually be created with.
# ⚠ Parse the JSON, do not regex the YAML. The first version of this grepped
# for -\s*'?1'? and failed against compose's DOUBLE-quoted `- "1"` — an
# assert that fails for the wrong reason is worse than no assert.
shell: |
docker compose --project-directory {{ gen_dir }} config --format json \
| jq -e '.services["vllm-gen"].deploy.resources.reservations.devices[0].device_ids == ["1"]'
changed_when: "false"
- name: "Recreate gen onto GPU1 (a device change needs up -d, not restart)"
sudo: true
shell: docker compose --project-directory {{ gen_dir }} up -d vllm-gen
- name: "Wait for gen to serve /health (cold start: weights + CUDA graphs + MTP)"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ gen_port }}/health; then
echo "gen healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: gen did not become healthy in 900s"; exit 1
changed_when: "false"
verify:
- name: "gen was genuinely RECREATED (container id changed)"
sudo: true
shell: |
before=$(cat /tmp/gen-container-id-before.txt)
after=$(docker inspect vllm-gen --format '{{.Id}}')
echo "before=${before:0:12} after=${after:0:12}"
test "$before" != "$after"
changed_when: "false"
- name: "gen's process is resident on GPU1"
sudo: true
# ⚠ Match by CGROUP, not by `.State.Pid`. vLLM V1 runs EngineCore as a CHILD
# of the container's pid 1, and it is the child that holds the GPU memory —
# nvidia-smi never reports `.State.Pid`, so comparing against it always fails.
shell: |
cid=$(docker inspect vllm-gen --format '{{.Id}}')
gpu1_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 1)
found=0
for p in $(nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv,noheader \
| grep "$gpu1_uuid" | cut -d, -f2 | tr -d ' '); do
if grep -q "$cid" /proc/$p/cgroup 2>/dev/null; then
echo "gen pid $p resident on GPU1: $(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | grep "^$p,")"
found=1
fi
done
test "$found" -eq 1
changed_when: "false"
- name: "gen answers a real completion, not just /health"
sudo: true
shell: |
. {{ gen_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ gen_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${GEN_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "GPU0 IS EMPTY — zero compute processes"
sudo: true
shell: |
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
n=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 0)
echo "GPU0 compute procs=${n} free=${free} MiB"
test "$n" -eq 0 && test "$free" -ge 95000
changed_when: "false"
- name: "mog-sec is stopped (not removed — close depends on `start` working)"
sudo: true
shell: |
docker inspect -f '{{.State.Status}}' vllm-mog-sec | tee /dev/stderr | grep -qx exited
changed_when: "false"
- name: "GPU1 still has headroom for Scriberr's on-demand load"
sudo: true
shell: |
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
echo "GPU1 free after gen landed: ${free} MiB"
test "$free" -ge 12000
changed_when: "false"
playbooks/homepage-regroup-ana-docker.yaml
+226
View File
@@ -0,0 +1,226 @@
# Homepage recategorisation — ana-docker (10.250.50.70), 13 containers.
#
# Splits the dashboard on ONE axis: do you open this thing, or is it an
# endpoint you only want to know is alive? See the layout: block in
# stacks/homepage/conf/settings.yaml for the target shape.
#
# memos, miniflux, nevermore, searxng -> Daily (was Notes / News / Apps)
# zed-fim-proxy -> AI - Inference (no UI; href is /ping)
# adguardhome -> DNS & Filtering
# traefik -> Reverse Proxies
# dockge -> Compose Consoles
# crowdsec, mailrise, rest-server,
# gitea-runner, hbbr (RustDesk relay) -> Agents (no UI)
#
# `homepage.group` is read at container CREATION, so each edit is followed by
# `compose up -d <service>` — a restart would leave the old label in place.
# Both halves are idempotent: the sed is gated on the old value still being
# present, and `up -d` is a no-op when the container already matches its spec.
#
# Run: scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml
steps:
# ---- label edits -------------------------------------------------------
- name: memos -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=Notes$|homepage.group=Daily|'
/opt/docker/compose/memos/compose.yaml
when: grep -q 'homepage.group=Notes$' /opt/docker/compose/memos/compose.yaml
- name: miniflux -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=News$|homepage.group=Daily|'
/opt/docker/compose/miniflux/compose.yaml
when: grep -q 'homepage.group=News$' /opt/docker/compose/miniflux/compose.yaml
- name: nevermore -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=News$|homepage.group=Daily|'
/opt/docker/compose/nevermore/compose.yaml
when: grep -q 'homepage.group=News$' /opt/docker/compose/nevermore/compose.yaml
- name: searxng -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=Daily|'
/opt/docker/compose/searxng/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/searxng/compose.yaml
- name: zed-fim-proxy -> AI - Inference
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Gateways . Chat$|homepage.group=AI - Inference|'
/opt/docker/compose/zed-fim-proxy/compose.yaml
when: grep -q 'homepage.group=AI - Gateways . Chat$' /opt/docker/compose/zed-fim-proxy/compose.yaml
- name: adguardhome -> DNS & Filtering
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|'
/opt/docker/compose/adguard-ana/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard-ana/compose.yaml
- name: traefik -> Reverse Proxies
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|'
/opt/docker/compose/traefik/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/compose.yaml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
- name: crowdsec -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|'
/opt/docker/compose/crowdsec/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/crowdsec/compose.yaml
- name: mailrise -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|'
/opt/docker/compose/mailrise/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/mailrise/compose.yaml
- name: rest-server -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|'
/opt/docker/compose/rest-server-ana/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/rest-server-ana/compose.yaml
- name: gitea-runner -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Toolchain$|homepage.group=Agents (no UI)|'
/opt/docker/compose/gitea-runner/compose.yaml
when: grep -q 'homepage.group=Toolchain$' /opt/docker/compose/gitea-runner/compose.yaml
- name: rustdesk (hbbr) -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|'
/opt/docker/compose/rustdesk/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/rustdesk/compose.yaml
# Two containers were both named plain "Open WebUI" and, once the ESH one
# joined this group, they landed side by side — same name, same icon family,
# only the description telling them apart. Site suffix, like Traefik/Dockge/
# AdGuard already carry.
- name: openwebui (ana) -> "Open WebUI (ana)"
sudo: true
shell: >-
sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (ana)|'
/opt/docker/compose/openwebui/compose.yaml
when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/openwebui/compose.yaml
# ---- recreates ---------------------------------------------------------
# traefik goes LAST: crowdsec is its bouncer, so bounce the bouncer first
# and let traefik come up against a settled agent.
- name: recreate memos
sudo: true
shell: cd /opt/docker/compose/memos && docker compose up -d memos
- name: recreate miniflux
sudo: true
shell: cd /opt/docker/compose/miniflux && docker compose up -d miniflux
- name: recreate nevermore-web
sudo: true
shell: cd /opt/docker/compose/nevermore && docker compose up -d nevermore-web
- name: recreate searxng
sudo: true
shell: cd /opt/docker/compose/searxng && docker compose up -d searxng
- name: recreate zed-fim-proxy
sudo: true
shell: cd /opt/docker/compose/zed-fim-proxy && docker compose up -d zed-fim-proxy
- name: recreate mailrise
sudo: true
shell: cd /opt/docker/compose/mailrise && docker compose up -d mailrise
- name: recreate rest-server
sudo: true
shell: cd /opt/docker/compose/rest-server-ana && docker compose up -d rest-server
- name: recreate gitea-runner
sudo: true
shell: cd /opt/docker/compose/gitea-runner && docker compose up -d runner
- name: recreate rustdesk relay
sudo: true
shell: cd /opt/docker/compose/rustdesk && docker compose up -d hbbr
- name: recreate openwebui (ana)
sudo: true
shell: cd /opt/docker/compose/openwebui && docker compose up -d open-webui
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate adguardhome
sudo: true
shell: cd /opt/docker/compose/adguard-ana && docker compose up -d adguardhome
- name: recreate crowdsec
sudo: true
shell: cd /opt/docker/compose/crowdsec && docker compose up -d crowdsec
- name: recreate traefik
sudo: true
shell: cd /opt/docker/compose/traefik && docker compose up -d traefik
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
memos miniflux nevermore-web searxng zed-fim-proxy adguardhome traefik
dockge crowdsec mailrise rest-server gitea-runner hbbr
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
# dig is not installed everywhere in the fleet, so fall back to the AdGuard
# UI — a resolver that serves its own dashboard on :8053 has come back up.
- name: adguard is back (DNS answer, or its UI if dig is absent)
changed_when: "false"
shell: >-
if command -v dig >/dev/null 2>&1;
then dig +short +time=3 +tries=2 @10.250.50.70 gitea.phasefinal.com | grep -q .;
else curl -sf -o /dev/null -m 8 http://10.250.50.70:8053/; fi
# Retried, not one-shot: the first run of this playbook checked 0.12s after
# `Started` and got rc=7 while traefik was still binding. The container was
# fine — `:8380/` 301s to /dashboard/ and both public hostnames answered 200
# seconds later. A recreate needs a moment; assert the settled state.
- name: traefik still routes
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0;
sleep 3; done; exit 1
- name: everything is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' memos miniflux nevermore-web
searxng zed-fim-proxy adguardhome traefik dockge crowdsec mailrise
rest-server gitea-runner hbbr | sort -u)" = "true"
playbooks/homepage-regroup-ana-ml2.yaml
+74
View File
@@ -0,0 +1,74 @@
# Homepage recategorisation — ana-ml2 (10.250.50.54), 2 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there.
#
# scriberr -> AI - Studios (a transcription UI you open, not an API seat)
# dockge -> Compose Consoles
#
# ⚠ THE vLLM SEATS ON THIS HOST ARE DELIBERATELY NOT TOUCHED. Every one of them
# would need a recreate to change its `homepage.group`, and a recreate means a
# multi-minute model reload on a seat that peers reach through the gateway. The
# separation the operator asked for — UI up top, API endpoints out of the way —
# is achieved for those groups by ORDER and `initiallyCollapsed` in
# stacks/homepage/conf/settings.yaml, which costs nothing. Keep it that way: if
# a future pass wants to rename `AI - Inference`, weigh it against bouncing six
# model seats.
#
# Run: scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
steps:
- name: scriberr -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|'
/opt/docker/compose/scriberr/compose.yaml
when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/scriberr/compose.yaml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate scriberr
sudo: true
shell: cd /opt/docker/compose/scriberr && docker compose up -d scriberr
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
scriberr dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
# Container age, not liveness — a seat may be legitimately stopped, so
# "is it running" cannot answer "did I bounce it". See the same step in
# playbooks/homepage-regroup-irv-ml1.yaml for how that distinction was found.
- name: the vLLM seats were NOT recreated by this run
sudo: true
changed_when: "false"
shell: >-
for c in $(docker ps -a --filter 'name=vllm-' --filter 'name=llama-'
--format '{{.Names}}'); do
created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue;
age=$(( $(date +%s) - $(date -d "$created" +%s) ));
if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi;
done
- name: scriberr answers
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10 11 12; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0;
sleep 5; done; exit 1
playbooks/homepage-regroup-esh-docker-vm.yaml
+135
View File
@@ -0,0 +1,135 @@
# Homepage recategorisation — esh-docker-vm (10.0.50.45), 6 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; the rationale, the
# label-at-creation constraint and the idempotency scheme are documented there.
#
# lobe-chat, open-webui -> AI - Gateways & Chat (chat frontends belong
# with the other chat frontends, not in Apps)
# adguardhome -> DNS & Filtering
# traefik -> Reverse Proxies
# dockge -> Compose Consoles
# mosquitto -> Agents (no UI) (an MQTT broker has no page)
#
# ⚠ adguard and traefik here use `docker-compose.yml`, not `compose.yaml`.
#
# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
steps:
- name: lobe-chat -> AI - Gateways & Chat
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|'
/opt/docker/compose/lobe-chat/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/lobe-chat/compose.yaml
- name: open-webui -> AI - Gateways & Chat
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|'
/opt/docker/compose/open-webui/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/open-webui/compose.yaml
# Site suffix — the ana instance is also called "Open WebUI" and the two now
# sit side by side in the same group. See the sibling step in
# playbooks/homepage-regroup-ana-docker.yaml.
- name: open-webui -> "Open WebUI (esh)"
sudo: true
shell: >-
sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (esh)|'
/opt/docker/compose/open-webui/compose.yaml
when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/open-webui/compose.yaml
- name: mosquitto -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|'
/opt/docker/compose/mosquitto/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/mosquitto/compose.yaml
- name: adguardhome -> DNS & Filtering
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|'
/opt/docker/compose/adguard/docker-compose.yml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml
- name: traefik -> Reverse Proxies
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|'
/opt/docker/compose/traefik/docker-compose.yml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/docker-compose.yml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
# ---- recreates ---------------------------------------------------------
- name: recreate lobe-chat
sudo: true
shell: cd /opt/docker/compose/lobe-chat && docker compose up -d lobe-chat
- name: recreate open-webui
sudo: true
shell: cd /opt/docker/compose/open-webui && docker compose up -d open-webui
- name: recreate mosquitto
sudo: true
shell: cd /opt/docker/compose/mosquitto && docker compose up -d mosquitto
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate adguardhome
sudo: true
shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome
- name: recreate traefik
sudo: true
shell: cd /opt/docker/compose/traefik && docker compose up -d traefik
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
lobe-chat open-webui mosquitto adguardhome traefik dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
- name: adguard is back
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0;
sleep 3; done; exit 1
- name: traefik still routes
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0;
sleep 3; done; exit 1
# ⚠ Must be 10.0.50.45, NOT 127.0.0.1. `HOMEPAGE_ALLOWED_HOSTS` matches
# host AND port, and `127.0.0.1:5100` is not in the list — it answers 400
# while the dashboard is perfectly healthy. The first run of this playbook
# failed here on rc=22 for exactly that reason.
- name: the dashboard itself is still served
changed_when: "false"
shell: curl -sf -o /dev/null -m 10 http://10.0.50.45:5100/api/services
- name: everything is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' lobe-chat open-webui
mosquitto adguardhome traefik dockge | sort -u)" = "true"
playbooks/homepage-regroup-irv-ml1.yaml
+115
View File
@@ -0,0 +1,115 @@
# Homepage recategorisation — irv-ml1 (10.100.79.3), 5 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there.
#
# arbo, comfyui, waterland-studio -> AI - Studios (was AI - Image & Media)
# yt-voice-clipper -> AI - Studios (was AI - Audio Tools)
# dockge -> Compose Consoles
#
# `AI - Studios` is the "you open this and do work in it" group; the ASR and
# TTS API seats stay where they are and get collapsed by settings.yaml instead,
# which is what keeps the GPU seats out of this playbook entirely.
#
# ⚠ yt-voice-clipper carries its homepage labels in `docker-compose.override.yml`,
# not in `docker-compose.yml`, and its compose dir is a git checkout of the
# project — the override is the deploy-local layer, which is the right place
# for it.
#
# ⚠ This host is reached over the WireGuard tunnel. If the run cannot connect,
# check the tunnel before assuming the host is down.
#
# Run: scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
steps:
- name: arbo -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|'
/opt/docker/compose/arbo/compose.yaml
when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/arbo/compose.yaml
- name: comfyui -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|'
/opt/docker/compose/comfyui/compose.yaml
when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/comfyui/compose.yaml
- name: waterland-studio -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|'
/opt/docker/compose/waterland-studio/compose.yaml
when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/waterland-studio/compose.yaml
- name: yt-voice-clipper -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|'
/opt/docker/compose/yt-voice-clipper/docker-compose.override.yml
when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/yt-voice-clipper/docker-compose.override.yml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
# ---- recreates ---------------------------------------------------------
- name: recreate arbo
sudo: true
shell: cd /opt/docker/compose/arbo && docker compose up -d engine
- name: recreate comfyui
sudo: true
shell: cd /opt/docker/compose/comfyui && docker compose up -d comfyui
- name: recreate waterland-studio
sudo: true
shell: cd /opt/docker/compose/waterland-studio && docker compose up -d waterland-studio
- name: recreate yt-voice-clipper
sudo: true
shell: cd /opt/docker/compose/yt-voice-clipper && docker compose up -d api
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
arbo comfyui waterland-studio yt-voice-clipper-api-1 dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
# Assert NOT-RECREATED, not RUNNING. The first version of this checked that
# all five seats were up and failed — because chatterbox-fast has been down
# since 2026-08-10 and speaches since earlier the same morning, both long
# before this playbook existed. "Is it running" is the wrong question: a seat
# can be legitimately stopped. The question this step is actually asking is
# "did I bounce a model seat to relabel a dashboard", and container age
# answers it directly.
- name: the TTS and ASR seats were NOT recreated by this run
sudo: true
changed_when: "false"
shell: >-
for c in kokoro dots-tts chatterbox-fast tts-gateway parakeet speaches; do
created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue;
age=$(( $(date +%s) - $(date -d "$created" +%s) ));
if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi;
done
- name: everything relabelled is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' arbo comfyui
waterland-studio yt-voice-clipper-api-1 dockge | sort -u)" = "true"
playbooks/homepage-regroup-nh3-docker.yaml
+58
View File
@@ -0,0 +1,58 @@
# Homepage recategorisation — nh3-docker (10.100.50.40), 2 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there.
#
# adguardhome -> DNS & Filtering (⚠ uses docker-compose.yml)
# dockge -> Compose Consoles
#
# Run: scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
steps:
- name: adguardhome -> DNS & Filtering
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|'
/opt/docker/compose/adguard/docker-compose.yml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate adguardhome
sudo: true
shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
adguardhome dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
- name: adguard is back
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0;
sleep 3; done; exit 1
- name: everything is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' adguardhome dockge
| sort -u)" = "true"
scripts/training-probes/README.md
+53
View File
@@ -0,0 +1,53 @@
# Training throughput probes
Instruments for finding where a training step's time actually went. Written
2026-08-24 during the Gemma-4 26B-A4B ERP/RP tune investigation; the lessons
they produced live in
[`docs/pfi/training-throughput-playbook.md`](../../docs/pfi/training-throughput-playbook.md).
**These are diagnostic instruments, not production code.** They hard-code paths
for that run. Adapt the constants at the top; keep the measurement design.
## The probes
| script | settles | GPU | runtime |
|---|---|---|---|
| `step0_mask.py` | mask band structure; which layers keep the `is_causal` fast path | no | ~30 s |
| `step2_padding.py` | padding waste, length distribution, CE chunk sizing | no | ~2 min |
| `step_bucket.py` | bucketing gain, bucket-size sweep, source diversity | no | ~3 min |
| `step1_profile.py` | scaling fit, padding penalty, CE wall clock, kernel table | **yes** | ~15 min |
Run in that order. Only the last needs the real checkpoint, and it wants an
idle card — it loads ~48 GiB and peaks near 77 GiB at `2 × 16,384`.
## Design rules worth preserving when you adapt these
**`step1_profile.py` reuses the harness's own `discover_target_modules` and
replicates its `compute_loss` byte-for-byte** rather than re-implementing the
step. A probe that reimplements the training step measures the probe. If you
port this, keep the import from the real harness.
**`step0_mask.py` needs no weights and no GPU** — SDPA backend selection and
mask construction depend on shapes, dtype and mask presence, not on weight
values. That is what makes the correctness assertion cheap enough to run before
every job.
**The scaling test takes three points, not two.** Two points over three
plausible terms (quadratic, linear, fixed-per-batch) is underdetermined; see
playbook §1.1 for the hour that cost.
**`step_bucket.py` sweeps bucket size deliberately.** The first version
re-sorted within each bucket, which silently collapsed every bucket size to a
full global sort and made the sweep a no-op. If you change the pairing logic,
check that the sweep still varies something.
## Raw evidence
`step1-profile-output-2026-08-24.txt` is the unedited output of the run the
playbook's numbers come from — scaling points, padding penalty, CE timing, and
the full `key_averages()` kernel table. Kept so the claims can be re-derived
rather than taken on faith.
⚠ That table **double-counts**: `key_averages()` lists both the ATen op and the
CUDA kernel it launched, each carrying the same self device time. Sum device
kernel rows only. See playbook §3.4.
scripts/training-probes/step0_mask.py
+75
View File
@@ -0,0 +1,75 @@
"""Step 0 - assert the sliding mask band structure, and record which path
mask creation actually takes under the run config (attn_implementation=sdpa).
Correctness gate: transformers can SILENTLY skip mask creation and pass
attention_mask=None, which would make the 25 sliding layers do full causal
attention - a different model from the one vLLM serves. This converts
"probably fine because we are slow" into a measurement.
CPU only. No weights. No GPU.
"""
import torch
from transformers import AutoConfig
from transformers.masking_utils import (
create_causal_mask, create_sliding_window_causal_mask,
)
MODEL = "/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16"
N = 16384
W = 1024
PAD = " " + " " * 20
cfg = AutoConfig.from_pretrained(MODEL)
text = cfg.get_text_config()
text._attn_implementation = "sdpa"
print("sliding_window %s" % text.sliding_window)
print("layers %d (%d sliding / %d full)" % (
len(text.layer_types),
text.layer_types.count("sliding_attention"),
text.layer_types.count("full_attention")))
print("_attn_implementation %s" % text._attn_implementation)
print()
def build(attn_2d, label):
batch = attn_2d.shape[0] if attn_2d is not None else 1
embeds = torch.zeros(batch, N, 8, dtype=torch.bfloat16)
pos = torch.arange(N).unsqueeze(0)
kw = dict(config=text, inputs_embeds=embeds, attention_mask=attn_2d,
past_key_values=None, position_ids=pos)
full = create_causal_mask(**kw)
slide = create_sliding_window_causal_mask(**kw)
print("--- %s ---" % label)
for name, m in (("full_attention", full), ("sliding_attention", slide)):
if m is None:
print(" %-20s None -> flash / is_causal path AVAILABLE" % name)
continue
print(" %-20s tensor shape=%s dtype=%s" % (name, tuple(m.shape), m.dtype))
allowed = m if m.dtype == torch.bool else (m == 0)
per_row = allowed[0, 0].sum(-1)
print("%sallowed/row min=%d max=%d mean=%.1f" % (
PAD, per_row.min().item(), per_row.max().item(),
per_row.float().mean().item()))
if name == "sliding_attention":
ok = per_row.max().item() <= W
print("%sBAND <= %d ? %s" % (PAD, W, "PASS" if ok else "FAIL"))
sat = (per_row >= W).nonzero()
if sat.numel():
print("%ssaturates at row %d" % (PAD, sat[0].item()))
else:
print("%slast row allows %d of %d (%s)" % (
PAD, per_row[-1].item(), N,
"causal-full OK" if per_row[-1].item() == N else "UNEXPECTED"))
print()
# 1. no 2D mask at all - the "constraints silently dropped" scenario
build(None, "attention_mask=None (no padding info)")
# 2. all-ones 2D mask - equal-length batch, no padding
build(torch.ones(2, N, dtype=torch.long), "all-ones 2D (no padding)")
# 3. REAL right-padded batch - what collate_mixed actually produces
real = torch.ones(2, N, dtype=torch.long)
real[1, 6000:] = 0
build(real, "right-padded 2D (what collate_mixed emits)")
scripts/training-probes/step1-profile-output-2026-08-24.txt
+126
View File
@@ -0,0 +1,126 @@
========================================================================
loading model
========================================================================
Loading weights: 0%| | 0/1013 [00:00<?, ?it/s] Loading weights: 0%| | 2/1013 [00:00<01:19, 12.71it/s] Loading weights: 0%| | 4/1013 [00:00<01:10, 14.40it/s] Loading weights: 2%|▏ | 25/1013 [00:00<00:16, 61.02it/s] Loading weights: 5%|▍ | 48/1013 [00:00<00:12, 79.42it/s] Loading weights: 7%|▋ | 70/1013 [00:00<00:10, 87.32it/s] Loading weights: 9%|▉ | 92/1013 [00:01<00:10, 91.14it/s] Loading weights: 11%|█ | 113/1013 [00:01<00:09, 92.58it/s] Loading weights: 13%|█▎ | 135/1013 [00:01<00:09, 95.13it/s] Loading weights: 15%|█▌ | 157/1013 [00:01<00:08, 97.82it/s] Loading weights: 18%|█▊ | 179/1013 [00:02<00:08, 101.21it/s] Loading weights: 20%|█▉ | 201/1013 [00:02<00:08, 100.37it/s] Loading weights: 22%|██▏ | 223/1013 [00:02<00:07, 102.12it/s] Loading weights: 24%|██▍ | 244/1013 [00:02<00:07, 104.64it/s] Loading weights: 26%|██▋ | 266/1013 [00:02<00:07, 105.68it/s] Loading weights: 28%|██▊ | 288/1013 [00:03<00:06, 106.87it/s] Loading weights: 30%|███ | 308/1013 [00:03<00:05, 122.26it/s] Loading weights: 32%|███▏ | 322/1013 [00:03<00:05, 117.76it/s] Loading weights: 33%|███▎ | 335/1013 [00:03<00:06, 99.26it/s] Loading weights: 35%|███▍ | 354/1013 [00:03<00:06, 100.53it/s] Loading weights: 37%|███▋ | 375/1013 [00:03<00:06, 104.62it/s] Loading weights: 39%|███▉ | 397/1013 [00:04<00:05, 103.34it/s] Loading weights: 41%|████▏ | 419/1013 [00:04<00:05, 106.05it/s] Loading weights: 44%|████▎ | 441/1013 [00:04<00:05, 109.33it/s] Loading weights: 46%|████▌ | 463/1013 [00:04<00:05, 107.04it/s] Loading weights: 48%|████▊ | 485/1013 [00:04<00:04, 107.29it/s] Loading weights: 50%|█████ | 507/1013 [00:05<00:04, 105.14it/s] Loading weights: 52%|█████▏ | 528/1013 [00:05<00:04, 103.18it/s] Loading weights: 54%|█████▍ | 550/1013 [00:05<00:04, 102.05it/s] Loading weights: 56%|█████▋ | 572/1013 [00:05<00:04, 101.82it/s] Loading weights: 59%|█████▊ | 594/1013 [00:05<00:03, 106.80it/s] Loading weights: 61%|██████ | 616/1013 [00:06<00:03, 104.33it/s] Loading weights: 63%|██████▎ | 638/1013 [00:06<00:03, 103.57it/s] Loading weights: 77%|███████▋ | 778/1013 [00:06<00:00, 320.91it/s] Loading weights: 91%|█████████ | 920/1013 [00:06<00:00, 532.70it/s] Loading weights: 100%|██████████| 1013/1013 [00:06<00:00, 152.35it/s]
loaded in 9.8s targets=205
final_logit_softcapping = 30.0
attn_implementation = sdpa
========================================================================
A. SEQUENCE SCALING (no padding - isolates n)
========================================================================
2 x 2,048 1.776 s kept=3227 peak= 53.2 GiB
2 x 8,192 11.570 s kept=13050 peak= 62.3 GiB
2 x 16,384 35.017 s kept=25989 peak= 76.6 GiB
16384 -> 2048 ratio 19.71x (linear ~8x, launch-bound ~1x, quadratic ~64x)
16384 -> 8192 ratio 3.03x (linear ~2x, quadratic ~4x)
========================================================================
B. PADDING PENALTY (same real tokens, with vs without pad)
========================================================================
2 x 16,384 no padding 35.244 s kept=26048 peak= 76.6 GiB
2 x 16,384 50% pad on row 1 38.567 s kept=19640 peak= 77.8 GiB
========================================================================
C. ISOLATED CE WALL CLOCK
========================================================================
2 x 16,384 (CE timed) 35.329 s kept=26210 peak= 76.6 GiB CE=374 ms (1.1%)
2 x 4,096 (CE timed) 4.387 s kept=6512 peak= 55.3 GiB CE=93 ms (2.1%)
========================================================================
D. KERNEL TABLE - one fwd+bwd at 2 x 16,384
========================================================================
USDT:2026-08-24 22:03:51 574811:574811 SyncActivityProfilerHandler.cpp:52] profiler_start
USDT:2026-08-24 22:04:27 574811:574811 SyncActivityProfilerHandler.cpp:59] profiler_stop
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
Name Self CPU % Self CPU CPU total % CPU total CPU time avg Self CUDA Self CUDA % CUDA total CUDA time avg # of Calls
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
aten::_efficient_attention_backward 0.00% 573.671us 0.00% 1.681ms 56.028us 16.145s 45.71% 16.156s 538.546ms 30
fmha_cutlassB_bf16_aligned_128x64_k65536_sm80(PyTorc... 0.00% 0.000us 0.00% 0.000us 0.000us 16.145s 45.71% 16.145s 538.152ms 30
aten::_efficient_attention_forward 0.00% 795.660us 0.01% 1.933ms 32.218us 6.691s 18.94% 6.691s 111.519ms 60
fmha_cutlassF_bf16_aligned_32x128_gmem_sm80(PyTorchM... 0.00% 0.000us 0.00% 0.000us 0.000us 6.691s 18.94% 6.691s 111.519ms 60
aten::mm 0.51% 180.757ms 0.77% 270.268ms 10.614us 3.745s 10.60% 3.745s 147.063us 25463
aten::mul 0.13% 45.428ms 0.18% 64.624ms 12.129us 2.721s 7.70% 2.721s 510.606us 5328
aten::copy_ 0.06% 19.823ms 92.46% 32.574s 6.100ms 1.977s 5.60% 1.977s 370.189us 5340
void cutlass::Kernel2<cutlass_80_tensorop_bf16_s1681... 0.00% 0.000us 0.00% 0.000us 0.000us 1.542s 4.37% 1.542s 656.804us 2348
void at::native::elementwise_kernel<128, 2, at::nati... 0.00% 0.000us 0.00% 0.000us 0.000us 1.033s 2.92% 1.033s 545.219us 1894
void cutlass::Kernel2<cutlass_80_tensorop_bf16_s1681... 0.00% 0.000us 0.00% 0.000us 0.000us 868.642ms 2.46% 868.642ms 583.373us 1489
void at::native::vectorized_elementwise_kernel<4, at... 0.00% 0.000us 0.00% 0.000us 0.000us 787.028ms 2.23% 787.028ms 395.095us 1992
void at::native::unrolled_elementwise_kernel<at::nat... 0.00% 0.000us 0.00% 0.000us 0.000us 697.963ms 1.98% 697.963ms 304.521us 2292
aten::masked_fill_ 0.01% 3.403ms 0.01% 4.927ms 27.373us 576.927ms 1.63% 576.927ms 3.205ms 180
void at::native::vectorized_elementwise_kernel<4, at... 0.00% 0.000us 0.00% 0.000us 0.000us 561.815ms 1.59% 561.815ms 413.403us 1359
void at::native::vectorized_elementwise_kernel<4, at... 0.00% 0.000us 0.00% 0.000us 0.000us 469.818ms 1.33% 469.818ms 459.255us 1023
aten::add_ 0.01% 3.730ms 0.02% 6.640ms 7.209us 448.665ms 1.27% 448.665ms 487.150us 921
void cutlass::Kernel2<cutlass_80_tensorop_bf16_s1681... 0.00% 0.000us 0.00% 0.000us 0.000us 363.206ms 1.03% 363.206ms 394.789us 920
aten::add 0.03% 9.745ms 0.04% 14.287ms 10.205us 356.226ms 1.01% 356.226ms 254.447us 1400
void at::native::elementwise_kernel<128, 2, at::nati... 0.00% 0.000us 0.00% 0.000us 0.000us 352.705ms 1.00% 352.705ms 1.959ms 180
aten::index 0.01% 4.542ms 0.09% 32.242ms 132.682us 352.404ms 1.00% 352.430ms 1.450ms 243
void at::native::vectorized_gather_kernel<16, long>(... 0.00% 0.000us 0.00% 0.000us 0.000us 351.768ms 1.00% 351.768ms 1.933ms 182
Memcpy DtoD (Device -> Device) 0.00% 0.000us 0.00% 0.000us 0.000us 341.588ms 0.97% 341.588ms 634.922us 538
aten::pow 0.06% 21.418ms 0.11% 37.019ms 18.659us 332.072ms 0.94% 493.271ms 248.624us 1984
void at::native::vectorized_elementwise_kernel<4, at... 0.00% 0.000us 0.00% 0.000us 0.000us 330.315ms 0.94% 330.315ms 499.720us 661
void at::native::vectorized_elementwise_kernel<4, at... 0.00% 0.000us 0.00% 0.000us 0.000us 316.108ms 0.89% 316.108ms 383.161us 825
aten::sum 0.01% 5.258ms 0.02% 7.700ms 13.461us 286.461ms 0.81% 286.464ms 500.811us 572
aten::native_dropout 0.02% 6.453ms 0.03% 11.139ms 27.168us 258.244ms 0.73% 258.244ms 629.865us 410
void at::native::(anonymous namespace)::fused_dropou... 0.00% 0.000us 0.00% 0.000us 0.000us 258.244ms 0.73% 258.244ms 629.865us 410
void at::native::unrolled_elementwise_kernel<at::nat... 0.00% 0.000us 0.00% 0.000us 0.000us 239.055ms 0.68% 239.055ms 583.061us 410
aten::_index_put_impl_ 0.01% 4.170ms 3.22% 1.134s 7.508ms 235.816ms 0.67% 236.930ms 1.569ms 151
void at::native::elementwise_kernel<128, 4, at::nati... 0.00% 0.000us 0.00% 0.000us 0.000us 231.434ms 0.66% 231.434ms 385.081us 601
void at::native::vectorized_elementwise_kernel<4, at... 0.00% 0.000us 0.00% 0.000us 0.000us 226.451ms 0.64% 226.451ms 692.511us 327
void at::native::elementwise_kernel<128, 4, at::nati... 0.00% 0.000us 0.00% 0.000us 0.000us 224.222ms 0.63% 224.222ms 2.491ms 90
void cutlass::Kernel2<cutlass_80_simt_sgemm_64x128_8... 0.00% 0.000us 0.00% 0.000us 0.000us 219.030ms 0.62% 219.030ms 534.219us 410
void at::native::elementwise_kernel<128, 4, at::nati... 0.00% 0.000us 0.00% 0.000us 0.000us 209.409ms 0.59% 209.409ms 1.745ms 120
aten::div 0.01% 4.911ms 0.02% 6.652ms 13.278us 192.925ms 0.55% 192.925ms 385.080us 501
void (anonymous namespace)::indexing_backward_kernel... 0.00% 0.000us 0.00% 0.000us 0.000us 183.000ms 0.52% 183.000ms 6.100ms 30
void at::native::unrolled_elementwise_kernel<at::nat... 0.00% 0.000us 0.00% 0.000us 0.000us 148.335ms 0.42% 148.335ms 988.899us 150
aten::mean 0.02% 6.214ms 0.02% 8.406ms 12.717us 144.904ms 0.41% 144.904ms 219.220us 661
void at::native::reduce_kernel<512, 1, at::native::R... 0.00% 0.000us 0.00% 0.000us 0.000us 144.904ms 0.41% 144.904ms 219.220us 661
aten::_log_softmax 0.00% 527.616us 0.00% 716.309us 13.775us 139.294ms 0.39% 139.294ms 2.679ms 52
void at::native::(anonymous namespace)::cunn_SoftMax... 0.00% 0.000us 0.00% 0.000us 0.000us 139.294ms 0.39% 139.294ms 2.679ms 52
void at::native::reduce_kernel<512, 1, at::native::R... 0.00% 0.000us 0.00% 0.000us 0.000us 137.743ms 0.39% 137.743ms 286.368us 481
void at::native::reduce_kernel<128, 4, at::native::R... 0.00% 0.000us 0.00% 0.000us 0.000us 130.526ms 0.37% 130.526ms 1.088ms 120
aten::native_dropout_backward 0.00% 1.378ms 0.01% 3.156ms 15.394us 118.690ms 0.34% 118.690ms 578.975us 205
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
Self CPU time total: 35.229s
Self CUDA time total: 35.322s
========================================================================
E. LAUNCH COUNTS (grouped_mm: 128/layer sequential = no-op, 1 = grouped)
========================================================================
kernel count self ms
aten::_efficient_attention_backward 30 16144.6
fmha_cutlassB_bf16_aligned_128x64_k65536_sm80(PyTorchMemEf 30 16144.6
aten::_efficient_attention_forward 60 6691.2
fmha_cutlassF_bf16_aligned_32x128_gmem_sm80(PyTorchMemEffA 60 6691.2
aten::mm 25463 3744.6
aten::mul 5328 2720.5
aten::copy_ 5340 1976.8
void cutlass::Kernel2<cutlass_80_tensorop_bf16_s16816gemm_ 2348 1542.2
void at::native::elementwise_kernel<128, 2, at::native::gp 1894 1032.6
void cutlass::Kernel2<cutlass_80_tensorop_bf16_s16816gemm_ 1489 868.6
void at::native::vectorized_elementwise_kernel<4, at::nati 1992 787.0
void at::native::unrolled_elementwise_kernel<at::native::d 2292 698.0
aten::masked_fill_ 180 576.9
void at::native::vectorized_elementwise_kernel<4, at::nati 1359 561.8
void at::native::vectorized_elementwise_kernel<4, at::nati 1023 469.8
aten::add_ 921 448.7
void cutlass::Kernel2<cutlass_80_tensorop_bf16_s16816gemm_ 920 363.2
aten::add 1400 356.2
void at::native::elementwise_kernel<128, 2, at::native::gp 180 352.7
aten::index 243 352.4
void at::native::vectorized_gather_kernel<16, long>(char*, 182 351.8
Memcpy DtoD (Device -> Device) 538 341.6
aten::pow 1984 332.1
void at::native::vectorized_elementwise_kernel<4, at::nati 661 330.3
void at::native::vectorized_elementwise_kernel<4, at::nati 825 316.1
aten::sum 572 286.5
aten::native_dropout 410 258.2
void at::native::(anonymous namespace)::fused_dropout_kern 410 258.2
void at::native::unrolled_elementwise_kernel<at::native::C 410 239.1
aten::_index_put_impl_ 151 235.8
total self CUDA time 70644.4 ms
GEMM-ish kernels 31761.7 ms (45.0%)
non-GEMM 38882.7 ms (55.0%)
scripts/training-probes/step1_profile.py
+206
View File
@@ -0,0 +1,206 @@
"""Steps 1/2/4 - profiler kernel table, sequence scaling, isolated CE timing.
Loads the real model exactly as erp_sft_harness.runtime does (same
from_pretrained args, same PEFT config, same gradient checkpointing, same
chunked-CE compute_loss) and measures:
A. sequence scaling 2x2048 / 2x8192 / 2x16384 fwd+bwd
linear-dominated -> time falls ~8x from 16384 to 2048
launch-bound -> time barely falls
quadratic-dominated -> time falls ~64x
B. isolated CE wall clock (CUDA events around the chunked-CE block)
C. torch.profiler kernel table, sorted by self CUDA time
D. expert-GEMM launch counts (settles grouped_mm without kernel-name
archaeology: 128 sequential launches per layer = no-op, 1 = grouped)
Runs on GPU0, which is reserved and idle. Nothing else touches it.
"""
import json
import sys
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
sys.path.insert(0, "/tank/erp-tune/eitri-smithy")
from erp_sft_harness.core import IGNORE_INDEX, discover_target_modules
MODEL = "/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16"
CHUNK = 1024
MB = 2
print("=" * 72)
print("loading model")
print("=" * 72, flush=True)
t0 = time.time()
model = AutoModelForCausalLM.from_pretrained(
MODEL, dtype=torch.bfloat16, device_map={"": 0}, attn_implementation="sdpa",
)
targets = discover_target_modules(model)
model = get_peft_model(model, LoraConfig(
r=64, lora_alpha=128, lora_dropout=0.05, target_modules=targets,
bias="none", task_type="CAUSAL_LM",
))
model.enable_input_require_grads()
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
model.train()
print("loaded in %.1fs targets=%d" % (time.time() - t0, len(targets)), flush=True)
base = model.base_model.model if hasattr(model, "base_model") else model
body = base.model
lm_head = base.get_output_embeddings()
softcap = getattr(model.config.get_text_config(), "final_logit_softcapping", None)
print("final_logit_softcapping = %s" % softcap)
print("attn_implementation = %s" % model.config.get_text_config()._attn_implementation)
print(flush=True)
ce_ms = {"fwd": 0.0}
def compute_loss(input_ids, attention_mask, labels, time_ce=False):
"""Byte-for-byte the harness's compute_loss, with optional CE timing."""
hidden = body(input_ids=input_ids, attention_mask=attention_mask,
use_cache=False).last_hidden_state
flat_hidden = hidden[:, :-1, :].reshape(-1, hidden.size(-1))
flat_labels = labels[:, 1:].reshape(-1)
keep = flat_labels != IGNORE_INDEX
kept_hidden = flat_hidden[keep]
kept_labels = flat_labels[keep]
kept = int(kept_labels.numel())
def chunk_loss(chunk_hidden, chunk_labels):
logits = lm_head(chunk_hidden).float()
if softcap is not None:
logits = torch.tanh(logits / softcap) * softcap
return torch.nn.functional.cross_entropy(logits, chunk_labels, reduction="sum")
if time_ce:
s, e = torch.cuda.Event(True), torch.cuda.Event(True)
torch.cuda.synchronize()
s.record()
total = torch.zeros((), device=kept_hidden.device, dtype=torch.float32)
for start in range(0, kept, CHUNK):
total = total + torch.utils.checkpoint.checkpoint(
chunk_loss, kept_hidden[start:start + CHUNK],
kept_labels[start:start + CHUNK], use_reentrant=False,
)
if time_ce:
e.record()
torch.cuda.synchronize()
ce_ms["fwd"] = s.elapsed_time(e)
return total / kept, kept
def make_batch(n, pad_frac=0.0):
"""Synthetic batch. pad_frac trims the SECOND row and right-pads it,
mimicking collate_mixed on a heterogeneous pair."""
ids = torch.randint(100, 200000, (MB, n), device="cuda")
am = torch.ones(MB, n, dtype=torch.long, device="cuda")
labels = ids.clone()
if pad_frac > 0:
keep = int(n * (1 - pad_frac))
am[1, keep:] = 0
labels[1, keep:] = IGNORE_INDEX
# ~40% of real tokens carry loss (measured mean 2188/2752 is higher, but
# rp-dialogue assistant-only masking pulls the mix down); use the measured
# global ratio 57.7M ctx -> 45.9M targets = 0.795
m = torch.rand(labels.shape, device="cuda") > 0.795
labels[m] = IGNORE_INDEX
return ids, am, labels
def timed(n, pad_frac=0.0, reps=2, time_ce=False, label=""):
ids, am, labels = make_batch(n, pad_frac)
for _ in range(1): # warmup
loss, kept = compute_loss(ids, am, labels)
loss.backward()
model.zero_grad(set_to_none=True)
torch.cuda.synchronize()
best = None
for _ in range(reps):
torch.cuda.reset_peak_memory_stats()
t = time.perf_counter()
loss, kept = compute_loss(ids, am, labels, time_ce=time_ce)
loss.backward()
torch.cuda.synchronize()
dt = time.perf_counter() - t
best = dt if best is None else min(best, dt)
model.zero_grad(set_to_none=True)
peak = torch.cuda.max_memory_allocated() / 2**30
print(" %-34s %7.3f s kept=%-6d peak=%5.1f GiB%s" % (
label or ("2x%d pad=%.0f%%" % (n, pad_frac * 100)),
best, kept, peak,
(" CE=%.0f ms (%.1f%%)" % (ce_ms["fwd"], 100 * ce_ms["fwd"] / 1000 / best)) if time_ce else ""))
return best
print("=" * 72)
print("A. SEQUENCE SCALING (no padding - isolates n)")
print("=" * 72, flush=True)
t2048 = timed(2048, 0.0, label="2 x 2,048")
t8192 = timed(8192, 0.0, label="2 x 8,192")
t16384 = timed(16384, 0.0, label="2 x 16,384")
print()
print(" 16384 -> 2048 ratio %.2fx (linear ~8x, launch-bound ~1x, quadratic ~64x)"
% (t16384 / t2048))
print(" 16384 -> 8192 ratio %.2fx (linear ~2x, quadratic ~4x)"
% (t16384 / t8192))
print(flush=True)
print("=" * 72)
print("B. PADDING PENALTY (same real tokens, with vs without pad)")
print("=" * 72, flush=True)
timed(16384, 0.0, label="2 x 16,384 no padding")
timed(16384, 0.5, label="2 x 16,384 50% pad on row 1")
print(flush=True)
print("=" * 72)
print("C. ISOLATED CE WALL CLOCK")
print("=" * 72, flush=True)
timed(16384, 0.0, reps=2, time_ce=True, label="2 x 16,384 (CE timed)")
timed(4096, 0.0, reps=2, time_ce=True, label="2 x 4,096 (CE timed)")
print(flush=True)
print("=" * 72)
print("D. KERNEL TABLE - one fwd+bwd at 2 x 16,384")
print("=" * 72, flush=True)
ids, am, labels = make_batch(16384, 0.0)
loss, _ = compute_loss(ids, am, labels)
loss.backward()
model.zero_grad(set_to_none=True)
torch.cuda.synchronize()
with torch.profiler.profile(
activities=[torch.profiler.ProfilerActivity.CPU,
torch.profiler.ProfilerActivity.CUDA],
record_shapes=False, with_stack=False,
) as prof:
loss, _ = compute_loss(ids, am, labels)
loss.backward()
torch.cuda.synchronize()
model.zero_grad(set_to_none=True)
print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=45))
print()
print("=" * 72)
print("E. LAUNCH COUNTS (grouped_mm: 128/layer sequential = no-op, 1 = grouped)")
print("=" * 72)
rows = []
for ev in prof.key_averages():
if ev.self_device_time_total <= 0:
continue
rows.append((ev.key, ev.count, ev.self_device_time_total / 1000.0))
rows.sort(key=lambda r: -r[2])
print(" %-58s %8s %10s" % ("kernel", "count", "self ms"))
for k, c, ms in rows[:30]:
print(" %-58s %8d %10.1f" % (k[:58], c, ms))
total_ms = sum(r[2] for r in rows)
print()
print(" total self CUDA time %.1f ms" % total_ms)
gemm = sum(ms for k, c, ms in rows if any(t in k.lower() for t in
("gemm", "cutlass", "sm90", "sm100", "sm120", "nvjet", "ampere", "tensor")))
print(" GEMM-ish kernels %.1f ms (%.1f%%)" % (gemm, 100 * gemm / total_ms))
print(" non-GEMM %.1f ms (%.1f%%)" % (total_ms - gemm, 100 * (total_ms - gemm) / total_ms))
scripts/training-probes/step2_padding.py
+77
View File
@@ -0,0 +1,77 @@
"""Step 2 — padding ratio. Data-side, no GPU, no model.
Replicates the exact batching the trainer used: SequentialSampler over the
encode-cache order, per_device_batch_size=2, collate_mixed right-padding to
the pair max. Reports real vs padded token counts and the loss-target count
that sizes the chunked CE.
"""
import json, sys
from collections import Counter
CACHE = "/tank/erp-tune/run-01/encode-cache/encoded-a4b0796de1260930.jsonl"
IGNORE_INDEX = -100
MB = 2 # per_device_batch_size
ACCUM = 8 # gradient_accumulation_steps
lens, kept_counts, kinds = [], [], []
with open(CACHE) as fh:
for line in fh:
row = json.loads(line)
ids = row["input_ids"]
labels = row["labels"]
lens.append(len(ids))
kept_counts.append(sum(1 for x in labels if x != IGNORE_INDEX))
kinds.append(row.get("sample_kind", "?"))
n = len(lens)
print(f"records {n:,}")
print(f"sample_kind mix {dict(Counter(kinds))}")
print()
print(f"seq len min/mean/max {min(lens)} / {sum(lens)/n:.0f} / {max(lens)}")
print(f"loss targets min/mean/max {min(kept_counts)} / {sum(kept_counts)/n:.0f} / {max(kept_counts)}")
print()
# --- micro-batch padding, exactly as collate_mixed builds it ---
real = padded = 0
mb_widths, mb_waste, mb_kept = [], [], []
for i in range(0, n - n % MB, MB):
group = lens[i:i + MB]
width = max(group)
r = sum(group)
p = width * MB
real += r
padded += p
mb_widths.append(width)
mb_waste.append(1 - r / p)
mb_kept.append(sum(kept_counts[i:i + MB]))
nb = len(mb_widths)
print(f"micro-batches (mb={MB}) {nb:,}")
print(f"real tokens {real:,}")
print(f"padded tokens {padded:,}")
print(f"PADDING WASTE {100 * (1 - real / padded):.1f}% ({padded - real:,} pad tokens)")
print()
print(f"mb width min/mean/max {min(mb_widths)} / {sum(mb_widths)/nb:.0f} / {max(mb_widths)}")
srt = sorted(mb_widths)
for q in (50, 75, 90, 95, 99):
print(f" p{q} width {srt[int(nb*q/100)]}")
print(f"mb at max_seq_len 16384 {sum(1 for w in mb_widths if w >= 16384):,} ({100*sum(1 for w in mb_widths if w>=16384)/nb:.1f}%)")
print()
srtw = sorted(mb_waste)
print(f"per-mb waste p50/p90/max {100*srtw[nb//2]:.1f}% / {100*srtw[int(nb*0.9)]:.1f}% / {100*max(mb_waste):.1f}%")
print()
print(f"loss targets per mb min/mean/max {min(mb_kept)} / {sum(mb_kept)/nb:.0f} / {max(mb_kept)}")
print(f" -> CE chunks per mb (1024) min/mean/max {min(mb_kept)//1024+1} / {sum(mb_kept)/nb/1024:.1f} / {max(mb_kept)//1024+1}")
print()
# --- what length-bucketing would recover (sort by length, then batch) ---
order = sorted(range(n), key=lambda i: lens[i])
b_real = b_padded = 0
for i in range(0, n - n % MB, MB):
group = [lens[j] for j in order[i:i + MB]]
b_real += sum(group)
b_padded += max(group) * MB
print("--- counterfactual: length-bucketed sampler ---")
print(f"bucketed padded tokens {b_padded:,}")
print(f"bucketed waste {100 * (1 - b_real / b_padded):.1f}%")
print(f"TOKEN REDUCTION vs current {100 * (1 - b_padded / padded):.1f}%")
scripts/training-probes/step_bucket.py
+112
View File
@@ -0,0 +1,112 @@
"""Measure bucket-to-pair / shuffle-to-mix against the REAL encode cache.
Brokkr's design, validated on measured record lengths rather than a calibrated
length model:
1. sort records by length
2. cut into buckets of BUCKET records
3. form micro-batches of 2 WITHIN each bucket (adjacent after sort)
4. shuffle the resulting MICRO-BATCHES globally, seeded
Padding efficiency is a property of the pairing only, so step 4 costs nothing
and restores root-mixing inside each accumulation window.
Also applies the fitted cost model from the replica scaling test to convert
token savings into predicted wall clock.
"""
import json
import random
from collections import Counter
CACHE = "/tank/erp-tune/run-01/encode-cache/encoded-a4b0796de1260930.jsonl"
IGNORE_INDEX = -100
MB = 2
ACCUM = 8
SEED = 20260824
# fitted on the replica: t(w) = A*w + B*w^2 for a batch of 2 sequences of len w
A = 6.8715e-04
B = 8.8509e-08
rows = []
with open(CACHE) as fh:
for line in fh:
r = json.loads(line)
rows.append((len(r["input_ids"]), r.get("dataset_id", "?"),
r.get("sample_kind", "?")))
n = len(rows)
print("records %d" % n)
print()
def evaluate(order, label, show_roots=False):
real = padded = 0
widths = []
batches = []
for i in range(0, n - n % MB, MB):
grp = [rows[j] for j in order[i:i + MB]]
w = max(g[0] for g in grp)
real += sum(g[0] for g in grp)
padded += w * MB
widths.append(w)
batches.append([g[1] for g in grp])
nb = len(widths)
Ew = sum(widths) / nb
Ew2 = sum(w * w for w in widths) / nb
t_mb = A * Ew + B * Ew2
srt = sorted(widths)
print("--- %s ---" % label)
print(" padded tokens %s" % f"{padded:,}")
print(" waste %.1f%%" % (100 * (1 - real / padded)))
print(" E[w] (per-seq) %.0f" % Ew)
print(" E[w^2] %.3e" % Ew2)
print(" width p50/p90/p99 %d / %d / %d" % (
srt[nb // 2], srt[int(nb * .9)], srt[int(nb * .99)]))
print(" predicted micro-batch %.3f s (lin %.3f + quad %.3f, quad %.0f%%)" % (
t_mb, A * Ew, B * Ew2, 100 * B * Ew2 / t_mb))
print(" predicted step (x%d) %.1f s -> %.2f h over 1312 steps" % (
ACCUM, t_mb * ACCUM, t_mb * ACCUM * 1312 / 3600))
# unpadded micro-batches take the is_causal fast path on the 5 global layers
exact = sum(1 for i in range(0, n - n % MB, MB)
if len(set(rows[j][0] for j in order[i:i + MB])) == 1)
print(" ZERO-PAD micro-batches %d / %d (%.1f%%) <- global layers on is_causal" % (
exact, nb, 100 * exact / nb))
if show_roots:
# root diversity inside an accumulation window
div = []
for i in range(0, nb - nb % ACCUM, ACCUM):
win = [d for b in batches[i:i + ACCUM] for d in b]
div.append(len(set(win)))
print(" roots per accum window mean %.2f min %d (of %d roots)" % (
sum(div) / len(div), min(div), len({r[1] for r in rows})))
print()
return padded, t_mb
# --- current: encode-cache order, SequentialSampler ---
cur_padded, cur_t = evaluate(list(range(n)), "CURRENT (SequentialSampler)", True)
# --- bucket-to-pair + shuffle-to-mix ---
# BUCKET controls the efficiency-vs-diversity trade: records are globally
# sorted, cut into buckets of BUCKET, SHUFFLED WITHIN the bucket (not
# re-sorted), then paired adjacently. BUCKET=2 is a perfect global sort
# (0% waste, worst root mixing); larger buckets admit more length spread
# inside a pair but draw partners from a wider slice of the corpus.
for BUCKET in (2, 8, 32, 128, 512):
by_len = sorted(range(n), key=lambda i: rows[i][0])
rng = random.Random(SEED)
micro = []
for s in range(0, n, BUCKET):
chunk = by_len[s:s + BUCKET]
rng.shuffle(chunk) # mix WITHIN the length bucket
for k in range(0, len(chunk) - len(chunk) % MB, MB):
micro.append(chunk[k:k + MB])
rng.shuffle(micro) # shuffle-to-mix across buckets
order = [i for b in micro for i in b]
placed = set(order)
order += [i for i in by_len if i not in placed]
p, t = evaluate(order, "BUCKET=%d, shuffle within + global micro-batch shuffle" % BUCKET,
True)
print(" >>> vs current: %.1f%% fewer padded tokens, %.1f%% less wall clock" % (
100 * (1 - p / cur_padded), 100 * (1 - t / cur_t)))
print()
stacks/adguard-ana/.env.example
+16
View File
@@ -0,0 +1,16 @@
# AdGuard Home — Anaheim. Copy to `.env` on ana-docker at
# /opt/docker/compose/adguard-ana/.env and chmod 600. Never commit the real file.
# Web UI / API port. compose.yaml defaults to 8053 if unset.
AG_WEB_PORT=8053
# Password for the `infra-ops` AdGuard login, used ONLY by the Homepage
# query/blocked/latency widget (homepage.widget.password in compose.yaml).
# One credential authenticates against all three fleet instances (ANA, ESH,
# NH3) — verified 2026-08-24. Fetch it from the vault rather than copying it
# between boxes:
#
# secret get nh3-dev/adguard-infra-ops-password
#
# (`secret` is services/secrets-broker/secret on nh3-dev.)
ADGUARD_WIDGET_PASSWORD=
stacks/adguard-ana/compose.yaml
+16 -2
View File
@@ -39,12 +39,26 @@ services:
networks:
- tnet
labels:
- homepage.group=Service Networking
- homepage.group=DNS & Filtering
- homepage.name=AdGuard (ANA)
- homepage.icon=mdi-dns
# si-adguard, matching the ESH and NH3 instances. This carried mdi-dns
# and was the only one of the three wearing a different mark.
- homepage.icon=si-adguard
- homepage.description=DNS resolver + .internal zone (colo)
- homepage.href=http://10.250.50.70:${AG_WEB_PORT:-8053}
- homepage.siteMonitor=http://10.250.50.70:${AG_WEB_PORT:-8053}
# Query/blocked/latency strip, same as ESH and NH3. Added 2026-08-24:
# without it this was a short card sitting beside two tall ones, which
# is the whole reason the DNS group exists as its own band — a group's
# members should all carry a widget or none of them should.
# Credential is the fleet AdGuard `infra-ops` login, which authenticates
# against all three instances; it lives in this stack's .env on the host
# (never in git) and is vaulted at
# `secret get nh3-dev/adguard-infra-ops-password`.
- homepage.widget.type=adguard
- homepage.widget.url=http://10.250.50.70:${AG_WEB_PORT:-8053}
- homepage.widget.username=infra-ops
- homepage.widget.password=${ADGUARD_WIDGET_PASSWORD}
volumes:
adguard_work: {}
stacks/arbo/compose.yaml
+1 -1
View File
@@ -85,7 +85,7 @@ services:
networks:
- tnet
labels:
- homepage.group=AI - Image & Media
- homepage.group=AI - Studios
- homepage.name=Arbo
- homepage.icon=mdi-image-multiple-outline
- homepage.description=Catalog-driven ComfyUI engine (irv-ml1)
stacks/comfyui/compose.yaml
+1 -1
View File
@@ -90,7 +90,7 @@ services:
networks:
- tnet
labels:
- homepage.group=AI - Image & Media
- homepage.group=AI - Studios
- homepage.name=ComfyUI
- homepage.icon=mdi-image-auto-adjust
- homepage.description=Node-based SD/Flux inference (irv-ml1)
stacks/dockge/compose.yaml
+1 -1
View File
@@ -27,7 +27,7 @@ services:
networks:
- tnet
labels:
- homepage.group=Service Networking
- homepage.group=Compose Consoles
- homepage.name=Dockge (${DOCKGE_HOST_LABEL})
- homepage.icon=sh-dockge.png
- homepage.description=Compose UI on ${DOCKGE_HOST_LABEL}
stacks/gemma4-charrp/.env.example
+66
View File
@@ -0,0 +1,66 @@
# gemma4-charrp — char-rp seat on ana-ml2 GPU0. Real .env lives on the host.
# ⚠ GPU0 IS SHARED WITH `vllm-gen`. gen runs at --gpu-memory-utilization 0.43
# but actually holds ~45.6 GiB of the 94.97 GiB card — that flag sizes the KV
# cache and does NOT cover CUDA context, graphs and non-torch overhead. The
# predecessor seat sat at 0.51, the pair summed to 0.94, and on 2026-08-24 it
# stopped fitting and crash-looped 13 times with
# `torch.OutOfMemoryError: ... 195.19 MiB is free`.
#
# 0.47 keeps ~4.8 GiB of real margin. This model's weights are only ~15.3 GiB
# (NVFP4) against the predecessor's ~19.5 GiB, so the same budget buys MORE KV
# cache than before, not less. Raising this means lowering gen's in the same
# change — and check the real numbers, not the flags:
# nvidia-smi --query-compute-apps=pid,used_memory --format=csv
GEMMA4_GPU_MEM_UTIL=0.47
# Native context. config.json declares max_position_embeddings 262144, same as
# the outgoing seat, so this is a straight-across swap on context too.
GEMMA4_MAX_MODEL_LEN=262144
GEMMA4_MAX_NUM_SEQS=32
# ⚠ THE NVFP4 QUANT, NOT THE BF16. /tank/aimodels/gemma4-26b-a4b-it-bf16 is the
# QLoRA tuning base and is 48.10 GiB of weights — it does not fit beside gen.
GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4
#
# ACTIVATION-AXIS CONTROL, for one-off benching — swap this line, recreate:
# GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16
# Same NVFP4 weights and the same compressed-tensors loader, but 16-bit
# activations.
#
# ⚠ THE ORIGINAL MOTIVATION FOR THIS CONTROL WAS RETRACTED — see the README's
# "Superseded claims" section. The A16 run itself HELD and is the reason this
# path is documented: activation precision is close to free on this battery,
# every other task identical across builds. But the "12% contradiction
# detection" number that prompted it was an artifact of an ill-posed benchmark
# item, not a model property. Keep the path; do not repeat the reasoning.
# Two minutes down each way; port and both aliases are unchanged, so no
# consumer config moves.
#
# ⚠ Only prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16 is genuinely A16.
# bg-digitalservices and ManniX-ITA both publish repos NAMED "NVFP4A16" whose
# config.json declares input_activations num_bits 4. Check the field, not the
# name, before ever substituting a different repo.
#
# ⚠⚠ THE A16 BUILD ALSO SHIPS A STALE CHAT TEMPLATE — the control run MUST
# override it or it moves two axes instead of one:
# --chat-template /tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja
# Verified by hash on 2026-08-24: upstream google/gemma-4-26B-A4B-it is 390
# lines, the A4 build's is 389 and byte-identical to it once trailing newlines
# are normalised, and the A16 build's is 266 and is NOT. The thinking machinery
# is built differently too — upstream and A4 set
# `enable_thinking | default(false)` at line 186, the A16 template has no such
# set — and its tokenizer_config response_schema has no `thinking` property.
# It was quantized from an older revision. Served with its own template the two
# arms would render DIFFERENT PROMPTS, and a score delta could be the template
# rather than the activations.
# Safe to override because the tokenizers agree: vocab identical at 262,144
# entries, added_tokens identical. Same template over the same vocab renders
# the same token ids.
GEMMA4_PORT=8016
GEMMA4_GPU_ID=0
GEMMA4_CONTAINER=vllm-gemma4-charrp
# Same value as every other vLLM seat on this host — the gateway presents it.
API_KEY=
stacks/gemma4-charrp/README.md
+260
View File
@@ -0,0 +1,260 @@
# gemma4-charrp — the char-rp seat (ana-ml2 GPU0)
`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.
```
char-rp non-thinking -> http://10.250.50.54:8016/v1
char-rp-reasoning thinking -> http://10.250.50.54:8016/v1
```
Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
up, and it cost a peer a mis-attributed benchmark before it was noticed.
## Three model directories, and they are not interchangeable
| path | size | what it is |
|---|---|---|
| `gemma4-26b-a4b-it-bf16` | 49 GB | **Stock BF16.** Unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. Its `chat_template.jinja` is the canonical upstream one; see below. |
| `gemma4-26b-a4b-it-heretic-bf16` | 49 GB | **QLoRA trainee base** (operator's pick, 2026-08-24). llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100. |
| `gemma4-26b-a4b-it-abliterated-bf16` | 49 GB | **Trainee alternate.** TrevorJS, KL 0.09, refusals 1/100 effective and 5/686 cross-dataset — lowest measured damage of the field. |
| `gemma4-26b-a4b-it-nvfp4` | 16 GB | **What is served.** RedHatAI, compressed-tensors, W4**A4**. |
| `gemma4-26b-a4b-it-nvfp4a16` | 17 GB | **Activation-axis control**, for benching only. prithivMLmods, compressed-tensors, W4**A16**. |
All under `/tank/aimodels/`, pulled by the `gemma4-26b-*-dl.py` scripts beside
them with revisions pinned.
### ⚠ Third-party Gemma-4 builds ship STALE CHAT TEMPLATES — this is endemic
Verified by hash on 2026-08-24 across every third-party derivative pulled here.
**Not one of them ships upstream's template:**
| build | lines | sha256 (normalised) |
|---|---|---|
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
| RedHatAI NVFP4 (what is served) | 389 | `6a1015c47ccfcfa6` — the only match |
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
Three independent repos carrying the identical stale 266-line file means it
propagated through the ecosystem, not that one packager slipped.
**Consequences differ by use and both are silent:**
- **Serving** — a different template renders a different prompt. This is why the
production compose pins the template explicitly.
- **Training** — if the harness renders examples through `base/chat_template.jinja`,
you train on a different prompt format than production serves. Train/serve
skew, no error, and it presents as a tuning failure.
For both, point at the upstream file:
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`.
### Measured: abliteration is close to free on this base (2026-08-24)
Isolated properly — stock BF16 against llmfan46 BF16, same precision, same
pinned upstream template, same 192 items, CoT off. **Abliteration was the only
axis that moved.**
| task | stock BF16 | heretic BF16 | items |
|---|---|---|---|
| T1 state / T3 constraint / T4 long-context / T5 control | 100% | 100% | 0 |
| T2 contradiction | 75% | 59% | **−5** |
| T6 spatial | 75% | 88% | **+4** |
| **core** | **90.0%** | **89.4%** | −0.6 pts |
**Net cost 0.6 points — but it MOVED capability rather than removing it.** Five
items lost on contradiction detection, four gained on spatial composition,
nearly cancelling. Nobody predicted a gain anywhere, least of all that
direction.
Consequence for the base choice: **llmfan46 stands.** There is no case for
re-staging on TrevorJS at KL 0.09 over a 0.6-point net difference — the KL gap
between the two builds is smaller than the gap this measurement failed to find.
⚠ Read those as **~5 items and ~4 items at n=32**, not as −15.6/+12.5 percent.
The percentages read more precisely than the measurement supports, and only
marginals were run — no paired per-item analysis.
⚠ **This says nothing about quantization.** Stock BF16 scores T2 75% where stock
NVFP4 scored 94%, but those runs were n=32 and n=16 — different item counts mean
different item sets, and the extra items are not guaranteed equally easy. That
comparison is n-confounded and is not being made.
### Choosing an abliterated base — compare on published damage, not on names
"Low damage" has a measurable proxy and the field spreads widely on it:
| build | method | KL | refusals |
|---|---|---|---|
| TrevorJS | ARA-family | **0.09** | 1/100 effective, 5/686 cross-dataset, manually audited |
| llmfan46 | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
| huihui-ai | remove-refusals-with-transformers | none published | none published |
| trohrbaugh/heretic-ara | — | 0.2999 | 31.4% |
| coder3101/heretic | — | 0.4118 | 15.8% |
Fleet anchor for reading those numbers: our own abliteration work found **Heretic
at KL 0.12 preserved the MTP head at 83.7% acceptance**. Both staged builds sit
at or below that, so neither is an extrapolation past what has been measured
here.
huihui-ai is rejected on this stack despite being the best-known abliteration
house: no published metrics, its card describes the method as "a crude,
proof-of-concept implementation", it states both thinking and non-thinking modes
were "completely abliterated", and its parameter count is 26,544,131,376 against
upstream's 25,805,936,206 — roughly 738M unexplained extra. The operator's
independent read is the same ("huihui produces garbage").
## GPU0 is shared and the budgets must sum under ~0.92
`vllm-gen` runs at `--gpu-memory-utilization 0.43` but actually holds ~45.6 GiB
of the 94.97 GiB card — that flag sizes the KV cache and does **not** cover CUDA
context, graphs and non-torch overhead. The predecessor seat sat at 0.51, the
pair summed to 0.94, and on 2026-08-24 it stopped fitting and crash-looped 13
times with `torch.OutOfMemoryError: ... 195.19 MiB is free`.
This seat runs 0.47. Raising it means lowering gen's in the same change, and
**check the real numbers, not the flags**:
```bash
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
```
There is no room for a second concurrent seat on this card. That is why the A16
control below is a swap rather than a parallel deployment.
## Running the A16 activation-axis control
Why it was run, and what it actually settled: a 2026-08-24 battery appeared to
show a large contradiction-detection deficit with CoT off, which had the exact
shape 4-bit input activations would produce on the most reasoning-dense task.
The control ran and found activation precision **close to free** — every other
task identical across the two builds. That quantization result stands.
⚠ The *deficit* it was chasing does not — the benchmark item was ill-posed. See
"Superseded claims" at the end of this file. The procedure below is kept because
the flip is a genuinely useful capability, not because the original question was
sound.
⚠ **The A16 build ships a STALE CHAT TEMPLATE and the run MUST override it.**
Verified by hash 2026-08-24: upstream `google/gemma-4-26B-A4B-it` is 390 lines,
the A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, the A16 build's is 266 and is not. Upstream and A4 open the thinking
path with `{%- set enable_thinking = enable_thinking | default(false) -%}`; the
A16 template has no such set. Its `tokenizer_config.json` `response_schema` also
lacks the `thinking` property. Served with its own template the two arms render
**different prompts**, and a score delta could be the template rather than the
activations.
Overriding is safe: the tokenizers agree — vocab identical at 262,144 entries,
`added_tokens` identical — so the same template over the same vocab renders the
same token ids. brokkr independently diffed every non-quantization config field
of both builds against the upstream BF16 and found only `transformers_version`
differing, which is a save-time library version rather than a model property.
Residual risk stated honestly on both sides: config identity is not weight
identity, and nobody has done a dequantization pass.
The procedure, ~2 minutes each way:
1. Add the template flag to `compose.yaml` (a **no-op for production** — the A4
build ships this exact file, so pinning it explicitly changes nothing and
guards against precisely the staleness above):
```yaml
- --chat-template
- ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
```
⚠ If `GEMMA4_MODEL` is ever pointed at a *different checkpoint*, this default
must move with it. A pinned template is only correct for the checkpoint it
came from — that is the mistake the outgoing MeroMero seat's hand-patched
template was warning about, inverted.
2. `GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16` in the host `.env`,
then `docker compose up -d --force-recreate vllm-gemma4-charrp`.
3. Tell brokkr; they fire one CoT-off arm, 96 items, under a minute of wall
clock, and report back.
4. Revert `GEMMA4_MODEL` to the `nvfp4` path and recreate.
Only the **CoT-off** arm is worth running. With thinking on the model already
scores 100% on every task it completes, and a ceiling cannot move.
⚠ Displacing this seat is an operator decision, not a routine one. It is live.
## Rollback to MeroMero-v2
`stacks/meromero-charrp/` is retained stopped in `created` state, labelled
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**:
```bash
cd /opt/docker/compose/gemma4-charrp && docker compose stop vllm-gemma4-charrp
cd /opt/docker/compose/meromero-charrp && docker compose up -d vllm-meromero-rp
```
## Gemma-4 flags that are load-bearing
Carried over from the MeroMero seat because they are architecture-level, not
checkpoint-level:
- `--tool-call-parser gemma4` + `--enable-auto-tool-choice` — Gemma-4 emits its
own native tool syntax, not the qwen3_coder XML the other seats use. Without
these, any tools-bearing request 400s outright.
- `--reasoning-parser gemma4` — absorbs the `<|channel>` thought markers; without
it they leak into `content` verbatim on the post-tool turn (vllm #45834).
- `--default-chat-template-kwargs '{"enable_thinking": false}'` — **mandatory
companion to the reasoning parser.** The parser defaults `enable_thinking` to
TRUE, which pre-initialises the engine to REASONING, so all plain prose lands
in `reasoning_content` with a null `content` and every char-rp consumer breaks.
Production runs thinking off; the thinking route reaches it per-request.
## Known defect, not ours
With thinking on, the model does not reliably terminate on constraint-following.
brokkr measured **32 of 96 calls truncating at a 12,000-token cap, with all 16
constraint items among them**, and inspected the traces: the reasoning is sound
right up to the point it fails to stop. A non-termination defect, not a
wrong-answer one. `VLLM_USE_V2_MODEL_RUNNER=0` enables
`thinking_token_budget` and is the first lever to try if a usable thinking seat
is ever wanted — deliberately **not** applied here, since it costs the faster
model runner for a mode production does not serve.
## Superseded claims
Following the repo convention for quant work: when a recorded claim turns out
wrong, it gets a dated row rather than a silent edit, so older notes elsewhere
stop misleading people.
| date | claim, as recorded | status |
|---|---|---|
| 2026-08-24 | "Gemma-4-26B-A4B-it scores 12% on contradiction detection with CoT off, against gen's 81% — the model owns the deficit." | **RETRACTED same day.** The benchmark item was ill-posed: it presented two mutually contradicting statements and asked for "the contradicting statement", but **contradiction is symmetric** — neither was more the contradicting one, and the model consistently named the absolute claim, a defensible reading the labelling scored wrong every time. |
| 2026-08-24 | "Domain tuning costs 43 points of contradiction detection." | **RETRACTED.** Rested entirely on the same item. On the corrected instrument the effect does not shrink, it **reverses**. |
| 2026-08-24 | All pre-fix T2 (contradiction) numbers for Gemma-4, MeroMero-v2, `sec` and `gen`. | **VOID.** |
**The tell, worth internalising: the score was BELOW CHANCE.** 12% on a
five-option task is under the 20% floor. A below-chance score indicts the
instrument before it indicts the model, and that should be the first reaction
rather than a late one. Neither side caught it until the individual items were
read.
A second defect surfaced while fixing the first: all generators shared one RNG,
so rewriting one task reshuffled every task after it. Each task now seeds from
its own name.
**What survived, and it is not nothing:** the A16 control result holds —
activation precision is close to free on this workload, with every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run (the misnamed A16 repos, the stale chat template) were real and
independent of the item defect. The corrected picture, CoT off, n=96:
| arm | core | T2 | T6 | median latency |
|---|---|---|---|---|
| `char-rp` (Gemma-4) | 92.5% | **94%** | 69% | 0.24s |
| `sec` | 92.5% | 81% | 81% | 1.11s |
| `gen` | 86.2% | 50% | 81% | 0.33s |
Gemma leads on the very axis it was suspected of failing. Its actual weak axis
with thinking off is **T6, spatial composition at 69%** — and an earlier run
showed CoT-on takes T6 to 100%, which nothing else benched has managed.
stacks/gemma4-charrp/compose.yaml
+137
View File
@@ -0,0 +1,137 @@
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0.
#
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
# port, same served-model-names, so every gateway route and consumer is
# unchanged. The outgoing G4-MeroMero-v2-31B-NVFP4A16 stack is retained stopped
# for rollback — see stacks/meromero-charrp/.
#
# WHY THE SWAP: the seat's requirements now include chain-of-thought, which makes
# throughput MORE critical rather than less — the user waits through the whole
# reasoning block before the first visible token. The incumbent was the DENSE
# 31B at ~40.7 tok/s @32K; this is the 26B-A4B MoE at ~114 tok/s @32K
# (brokkr-smithy-dev's measurements). Same Gemma-4 family, so the parser and
# chat-template machinery below carries over unchanged.
#
# ⚠ THIS IS THE QUANT, NOT THE TUNING BASE. The BF16 weights live at
# /tank/aimodels/gemma4-26b-a4b-it-bf16 and are for QLoRA — 48.10 GiB of BF16
# weights CANNOT be served here: GPU0 is shared with vllm-gen, which holds
# ~45.6 GiB of the 94.97 GiB card, leaving ~49 GiB. Weights alone would eat all
# of it and the engine would die at KV-cache allocation. Do not "simplify" this
# to the BF16 path.
#
# ⚠ SCHEME IS W4A4, NOT W4A16. The RedHatAI quant declares 4-bit
# input_activations (compressed-tensors, format nvfp4-pack-quantized). The
# incumbent was NVFP4**A16**. Faster, and NOT a like-for-like quality
# comparison on the activation axis — say so when benching the two.
#
# Tunables in .env.
name: gemma4-charrp
services:
vllm-gemma4-charrp:
image: ${GEMMA4_IMAGE:-vllm/vllm-openai:latest}
container_name: ${GEMMA4_CONTAINER:-vllm-gemma4-charrp}
restart: unless-stopped
ipc: host
ports:
- "${GEMMA4_PORT:-8016}:8000"
volumes:
- /tank/aimodels:/tank/aimodels
environment:
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
- VLLM_API_KEY=${API_KEY:-}
command:
- ${GEMMA4_MODEL:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4}
- --quantization
- compressed-tensors
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
# LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` ->
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
# They are two aliases onto ONE seat, not two seats. Renaming either
# breaks the gateway silently.
- --served-model-name
- char-rp
- char-rp-thinking
# Tool-calling: Gemma-4 emits its OWN native syntax
# (<|tool_call>call:name{...}<tool_call|>), NOT the qwen3_coder XML the
# other seats use. Without these two flags any tools-bearing request 400s
# outright. Architecture-level, so it carries over from MeroMero unchanged.
- --tool-call-parser
- gemma4
- --enable-auto-tool-choice
# The gemma4 REASONING parser absorbs the <|channel>...<channel|> thought
# markers; without it they leak into `content` verbatim on the post-tool
# turn (upstream vllm #45834).
- --reasoning-parser
- gemma4
# MANDATORY COMPANION TO THE REASONING PARSER. The parser reads
# enable_thinking from chat_template_kwargs and DEFAULTS IT TO TRUE. True
# makes is_reasoning_end() return False at a new turn, which pre-initialises
# the engine to REASONING — so ALL plain RP prose lands in
# reasoning_content with a NULL content, breaking every char-rp consumer.
# Do not remove. Thinking is still reachable per-request via
# chat_template_kwargs {"enable_thinking": true}, which is what the
# `char-rp-reasoning` alias exercises.
- --default-chat-template-kwargs
- '{"enable_thinking": false}'
# TEMPLATE PINNED EXPLICITLY, AND IT IS A NO-OP FOR THE DEFAULT MODEL.
# The A4 build ships this exact file — verified byte-identical to upstream
# google/gemma-4-26B-A4B-it once trailing newlines are normalised (390 vs
# 389 lines, same hash) — so pinning it changes nothing about what is
# served and permanently removes a real class of bug.
#
# ⚠ WHY IT IS PINNED AT ALL: the A16 control build at
# /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 ships a STALE 266-line
# template. Upstream and A4 open the thinking path with
# `{%- set enable_thinking = enable_thinking | default(false) -%}`; the
# A16 one has no such set, and its tokenizer_config response_schema lacks
# the `thinking` property. Serving it with its own template would render a
# DIFFERENT PROMPT, turning a one-axis activation-precision control into a
# two-axis comparison — a result that would look like a finding.
# Safe to force across both builds because the tokenizers are identical:
# vocab 262,144 entries, added_tokens identical.
#
# ⚠ IF GEMMA4_MODEL EVER POINTS AT A DIFFERENT CHECKPOINT, THIS DEFAULT
# MUST MOVE WITH IT. A pinned template is only correct for the checkpoint
# it came from. That is the inverse of the mistake the outgoing MeroMero
# seat warned about, where a hand-patched template was assumed to transfer.
- --chat-template
- ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
- --max-model-len
- "${GEMMA4_MAX_MODEL_LEN:-262144}"
- --max-num-seqs
- "${GEMMA4_MAX_NUM_SEQS:-32}"
- --gpu-memory-utilization
- "${GEMMA4_GPU_MEM_UTIL:-0.47}"
- --kv-cache-dtype
- fp8
- --trust-remote-code
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${GEMMA4_GPU_ID:-0}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 600s
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
- homepage.icon=mdi-drama-masks
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs
networks:
tnet:
name: traefik-net
external: true
stacks/gemma4-trainee-bench/compose.yaml
+105
View File
@@ -0,0 +1,105 @@
# gemma4-trainee-bench — BF16 abliterated trainee base, served for benchmarking.
#
# EPHEMERAL BY DESIGN. This is not a production seat. It exists so the trainee
# base can be measured on the same battery as the served char-rp seat, and it
# takes GPU0 to itself while it runs.
#
# ⚠ IT CANNOT COEXIST WITH `vllm-gen`. The weights are BF16 — 48.07 GiB of a
# 94.97 GiB card — and gen actually holds ~45.6 GiB. 48.07 + 45.6 = 93.7 GiB
# before a single byte of KV cache, activations, CUDA graphs or non-torch
# overhead. Running this means gen is stopped, and gen coming back means this
# is stopped. There is no arrangement where both are up.
#
# ⚠ NO `--quantization` FLAG, AND THAT IS THE POINT.
# The production gemma4-charrp compose hardcodes `--quantization
# compressed-tensors` because it serves an NVFP4 build. Pointing that stack at
# these BF16 weights crash-loops immediately:
# TypeError: CompressedTensorsConfig.__init__() missing 3 required
# positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
# vLLM tries to read a quantization config out of a checkpoint that has none.
# 35 restarts before it was caught. Hence a separate stack rather than another
# variable on the production one.
#
# SAME PORT AND SAME SERVED-MODEL-NAMES AS char-rp, deliberately: brokkr's
# battery targets the `char-rp` / `char-rp-reasoning` gateway aliases, so
# serving under those names means their harness needs no edit and the numbers
# are directly comparable to the runs already recorded. Only one of this stack
# and gemma4-charrp can be up at a time — both bind :8016.
#
# Tunables in .env.
name: gemma4-trainee-bench
services:
vllm-gemma4-trainee-bench:
image: ${BENCH_IMAGE:-vllm/vllm-openai:v0.26.0}
container_name: ${BENCH_CONTAINER:-vllm-gemma4-trainee-bench}
# `no`, not unless-stopped. A bench seat that resurrects itself after the
# window closes would silently hold 48 GiB and block gen's restore.
restart: "no"
ipc: host
ports:
- "${BENCH_PORT:-8016}:8000"
volumes:
- /tank/aimodels:/tank/aimodels
environment:
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
- VLLM_API_KEY=${API_KEY:-}
command:
- ${BENCH_MODEL:-/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16}
- --served-model-name
- char-rp
- char-rp-thinking
# Architecture-level Gemma-4 flags, identical to the production seat —
# see stacks/gemma4-charrp/README.md for why each is load-bearing.
- --tool-call-parser
- gemma4
- --enable-auto-tool-choice
- --reasoning-parser
- gemma4
- --default-chat-template-kwargs
- '{"enable_thinking": false}'
# ⚠ UPSTREAM TEMPLATE, NOT THE ONE THIS CHECKPOINT SHIPS.
# Every third-party Gemma-4 derivative carries a stale template: this
# abliterated build ships a 365-line file, the stock upstream is 390
# lines. Benching through a different template than production serves
# would make the comparison meaningless — the same reason the production
# seat pins it.
- --chat-template
- ${BENCH_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
- --max-model-len
- "${BENCH_MAX_MODEL_LEN:-262144}"
- --max-num-seqs
- "${BENCH_MAX_NUM_SEQS:-32}"
# Card to itself. 0.92 leaves ~7 GiB of real headroom on top of 48 GiB of
# weights — deliberately not 0.95, after a seat crash-looped this
# afternoon on 0.6 GiB of margin.
- --gpu-memory-utilization
- "${BENCH_GPU_MEM_UTIL:-0.92}"
- --kv-cache-dtype
- fp8
- --trust-remote-code
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${BENCH_GPU_ID:-0}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 900s
networks:
- tnet
# No homepage labels. This is a transient bench seat; giving it a dashboard
# card would leave a permanently-offline entry once the window closes.
networks:
tnet:
name: traefik-net
external: true
stacks/gitea-runner/compose.yaml
+1 -1
View File
@@ -27,7 +27,7 @@ services:
networks:
- tnet
labels:
- homepage.group=Toolchain
- homepage.group=Agents (no UI)
- homepage.name=gitea-runner
- homepage.icon=mdi-cog-play
- homepage.description=Gitea Actions self-hosted runner (${GITEA_RUNNER_NAME})
stacks/homepage/README.md
+231 -10
View File
@@ -33,11 +33,44 @@ recreated.
## Layout
`conf/settings.yaml` owns tabs, group order, and column counts — `services.yaml`
owns *what exists*, `settings.yaml` owns *where it sits*. Four tabs: Main, AI,
Infrastructure, Toolchain. A group listed in `layout:` with no members simply
renders empty, so a group can look "dead" when its provider host is unreachable
rather than when the group is wrong.
`conf/settings.yaml` owns tabs, group order, column counts and collapse state —
`services.yaml` owns *what exists*, `settings.yaml` owns *where it sits*. Four
tabs: Main, AI, Infrastructure, Toolchain. A group listed in `layout:` with no
members simply renders empty, so a group can look "dead" when its provider host
is unreachable rather than when the group is wrong.
**The organising question is "do I open this?", not "what is it?"** (operator,
2026-08-24). Every group is either **tools** — expanded, top of its tab — or
**endpoints** (an API, a broker, a background agent, an href that is a `/docs`
page or a `/ping` or nothing) — `initiallyCollapsed: true`, bottom of its tab.
A collapsed group still renders its eyebrow and rule, so the tab tells you the
thing exists without spending a row on it. `AI - Inference` holds the seats the
whole fleet runs on and it is collapsed, because you reach them through the
gateway, not by clicking them.
A second, quieter rule shapes the same block: **a group's members should all
have widgets or all not have them.** A stat strip makes a card ~50px taller, so
one widget card in a row of plain ones opens a void under the plain ones. That
is why AdGuard (3 widget cards) and Traefik (2 widget cards) are separate groups
from Dockge (5 plain cards) rather than one `Service Networking` band.
Group *membership* is not in this file — it is the `homepage.group=` label on
each container, and labels are read at container **creation**. The 2026-08-24
pass moved 28 services and the playbooks that did it are rerunnable:
```bash
scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml
scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
```
⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`,
`AI - Eval & Retrieval`, `AI - Speech (TTS)` and `AI - Audio Tools`.** Clearer
names would cost a recreate on sixteen seats — multi-minute model reloads on
endpoints peers reach through the gateway. Order and `initiallyCollapsed` buy
the same separation for free. Don't spend that recreate on a label.
## Foot-guns found in the 2026-08-17 audit
@@ -82,14 +115,144 @@ Three fixes, all in this stack's config except where noted:
they had members, so the last row of each was mostly dead space (Notes: 1
card in a 4-wide row). Columns now track member counts; see the rule in
`settings.yaml`. Check with `GET /api/services`, which prints live per-group
counts.
counts. ⚠ **Superseded 2026-08-24** — see below.
## 2026-08-24 layout + theme pass
- **Columns are 4 everywhere now; "columns = member count" is retired.**
`columns:` sets `lg:grid-cols-N` for one group, so tuning it per group fixes
that group's *card width* — and the width then changed at every group
boundary. Measured on the live board: Notes rendered a single 1464px card,
News and Media 728px, the AI tab alternated 360 / 728 / 286 down the page.
The old rule was avoiding dead cells in a short last row and bought a worse
defect. A short last row is what a grid looks like; a card wider than its
neighbours is what a mistake looks like. Rule and reasoning are at the top of
`settings.yaml`'s `layout:` block.
- **Scriberr's `AI Systems` group was on all four tabs** — the same untabbed-
group behaviour as UltraSeedbox in 2026-08-18, arriving this time from a
container label rather than from this file. Relabelled to `AI - Audio Tools`
in `stacks/scriberr/compose.yaml` (its real home, alongside Parakeet and
Speaches) and the container recreated. The `homepage.group=AI Systems` sample
in the repo-root `CLAUDE.md` was the source and now carries the constraint.
- **Long service names printed under their own status pill.** `.service-name`
reserved a 78px right gutter with `padding-right` and relied on
`overflow: hidden` to hold it — but overflow clips at the *padding* box, so
the gutter was spill room, not a guard. Every long name on the AI tab
collided. It holds by wrapping now; see the comment on the rule.
- **Descriptions are clamped to three lines** (floor of two, unchanged). Four-
and five-line descriptions were dragging their whole row 30-50px taller than
the cards beside them.
- **Icons were grey smudges.** Homepage masks every mdi glyph over
`--color-logo-start/stop`, stock slate-400 → slate-700, and the dark stop
sank the bottom of each glyph into the card fill. Overridden to ice → aurora
blue. ⚠ The override must sit on `html[class]`, not `:root` — Homepage sets
the same variables on `.theme-slate`, which is on `<html>`, and a class beats
`:root` on the same element.
- **Bookmark groups and Jellyfin's trailing stream rows** were the last two
un-themed components; both now use the card/eyebrow vocabulary.
Verified with Playwright against the live board — per-group card width, card
height spread, and a geometric title-vs-status collision check. Script pattern
is in the "iteration loop" note below.
## The tab bar goes missing after a recreate, then comes back on its own
**Status: self-healing, cause not pinned. Do not chase it.** After a recreate
the client render comes up with no tab bar, no wallpaper and no i18n (the
search box shows the raw key `search.search`), and groups fall back to
side-by-side columns. It restores itself with no intervention.
**Status: intermittent, self-healing, and now HALF-DIAGNOSED. Still don't chase
it — but you can now tell in one command whether you are in it.**
After a recreate the client render can come up with no tab bar, no wallpaper
and no i18n (the search box shows the raw key `search.search`), with groups
falling back to side-by-side columns. It restores itself with no intervention.
### The one-command test
```bash
curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'
```
`initialSettings":{"…` — healthy. `initialSettings":{}` — you are in it.
### What `initialSettings":{}` actually means (found 2026-08-24)
It is **not** a warm-up, a cache, or a partial render. It is the **catch branch**
of the page's data loader. Decompiled from `/app/.next/server/pages/index.js`:
```js
async function K() {
let a;
try {
a = logger("index"); // <-- assigned INSIDE the try
let { providers, ...d } = getSettings();
... await servicesResponse(), bookmarksResponse(), widgetsResponse()
return { props: { initialSettings: d, ... } };
} catch (b) {
return a && b && a.error(b), // <-- guarded on `a`
{ props: { initialSettings: {}, ... } };
}
}
```
Two consequences worth knowing:
1. **Something in that try block is throwing.** The page is not "still warming
up"; it has already failed and returned a degraded shape.
2. **The error can be swallowed completely.** `a` is assigned inside the same
try, and the catch only logs `if (a)`. If the *logger itself* is what threw,
`a` is `undefined` and nothing is written anywhere — which is exactly what
was observed on 2026-08-24: catch branch demonstrably taken, and not one
`index`-tagged line in `docker logs` or `conf/homepage/logs/homepage.log`.
### ONE CAUSE IS NOW KNOWN: a missing `theme:` key
**Removing `theme:` from `settings.yaml` reproduces this deterministically.**
Six force-recreates over seven minutes all served `initialSettings":{}` with the
key absent; restoring `theme: dark` rendered correctly on the next recreate in
12 seconds (2026-08-24). So the loader really can be thrown by config — just not
by the parts you would suspect, and never with a message.
That does **not** explain every occurrence: the same symptom has appeared with
`theme:` present and correct. Treat the missing key as one confirmed trigger,
not the whole story.
### What it is NOT — ruled out by measurement, don't re-run these
- **Not a downstream data failure.** `/api/services`, `/api/bookmarks`,
`/api/widgets` and `/api/hash` all return **200 with fully correct content**
while the page serves `initialSettings":{}` — including the brand-new group
structure, in the right order. Every input the loader awaits works when called
directly.
- **Not the 2026-08-24 layout rewrite.** Restoring the *previous, known-good*
`settings.yaml` and recreating reproduced the empty payload identically. (This
matches the 2026-08-19 finding that the pre-adoption backup config reproduces
it too.)
- **Not `/api/validate`,** which returns `[]` throughout.
- **Not disk, not permissions.** 206 GB free; the container runs as root and a
write test into `/app/config/logs` succeeds.
- **Probably not the log file.** Rolling the 8.6 MB `homepage.log` aside once
coincided with an immediate recovery, which looked like a lead — but the same
move did nothing during the `theme:`-key episode. Recorded so nobody chases
it twice; the coincidence was almost certainly just the intermittency.
### Timing, measured rather than assumed
Wildly variable, which is the whole trap. On 2026-08-24 one recreate came up
correct **within 10 seconds**, and three consecutive recreates ~40 minutes later
were still empty after 60-120s each. The 2026-08-19 session measured a fresh
container still tab-less at 4m30s twice, and healthy again after roughly an hour.
`docker ps` reporting `healthy` says nothing about it — the container is serving,
the page is just wrong.
**Practical rule: recreate, run the one-command test, and if it is empty, go do
something else and re-check.** Do not start editing config — that is how an hour
got spent in 2026-08-19 ruling out four causes that were never the cause (the
config, the v2.0.0 release, `PUID`/`PGID` and Docker discovery, and the server
side). Every one of those remains ruled out.
**First thing to check, now that one cause is confirmed:** diff `settings.yaml`
against the last version that rendered. A key that Homepage's loader needs and
cannot find will do this silently — `theme:` is the one we know about, and
there may be others. `git log -p -- stacks/homepage/conf/settings.yaml` is
faster than any amount of container archaeology.
**Timing, measured rather than assumed:** five minutes is NOT enough — a fresh
container was still tab-less at 4m30s, twice. It was observed healthy again
@@ -169,6 +332,64 @@ theme/build.py inlines the fonts + tokens -> conf/custom.css
**Do not hand-edit `conf/custom.css`.** Change `skyfall.css.in`, run
`python3 stacks/homepage/theme/build.py`, then deploy.
**Do not hand-edit the three vendored files either.** `build.py` records their
SHA-256 and **fails the build** on a mismatch — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name. Put the
override in `skyfall.css.in`, which is expressed entirely through Skyfall's
semantic layer (`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/
--warning`) and never against a raw family token or a literal colour. That is
not fussiness: the theme this one replaced built a parallel palette "derived
from the philosophy" and had to be torn out twice.
### Light and dark (2026-08-24)
Both themes are first-class. Skyfall Day is the bundle's own light ramp —
surfaces at `--sea-94/96/98`, text at `--sea-15`, and every chromatic family
dropping to its `-deep` (L 0.48) step. Nothing about it was derived here.
Precedence, highest first:
1. **an explicit choice** — the toggle in the header strip, stored in
`localStorage` under `skyfall-theme`;
2. **the OS preference** — `@media (prefers-color-scheme: light)`, applied only
while no explicit choice exists;
3. **dark** — Skyfall's default.
Two pieces of plumbing make that work, and both are load-bearing:
- **`build.py` re-emits the vendored `[data-theme="light"]` blocks** in both
forms — `[data-theme="light"], html.light` for an explicit choice, and a
copy inside the media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])`. That `:not()` pair
is what lets a stored *dark* choice survive a light-mode OS.
- **`conf/custom.js` renders the toggle**, because Homepage will not give us
its own.
⚠ **Homepage's built-in theme toggle is unreachable, and reaching for it breaks
the dashboard.** The toggle renders only when `settings.yaml` leaves `theme:`
unpinned — but with the key absent, the page's data loader throws and serves
`initialSettings: {}` (no tab bar, no layout, no i18n). Measured 2026-08-24:
six force-recreates over seven minutes all came up empty with the key removed;
restoring `theme: dark` rendered correctly on the next recreate in 12 seconds.
`/api/services` stays 200 and correct the whole time, which is exactly why this
looks like a caching problem and is not one. **Leave `theme: dark` pinned.**
### Type: one canonical face, two documented substitutions
Skyfall names Bespoke Sans (display) / Supreme (body, UI) / Victor Mono Nerd
Font (data, code). Only **Supreme** was ever vendored into this repo, and
Skyfall's own notes call Victor Mono "user-supplied", so the other two are
stand-ins rather than deviations:
| role | Skyfall | here |
|---|---|---|
| display | Bespoke Sans | **Space Grotesk** (variable, latin subset) |
| body / UI | Supreme | **Supreme 400/500/700** — canonical |
| data / mono | Victor Mono Nerd Font | **JetBrains Mono** (variable) |
Swapping in the real faces is a two-line change: `FONTS_*` in `build.py` and
the `--font-display` / `--font-mono` overrides at the top of `skyfall.css.in`.
### Iterating on the theme — do NOT recreate the container
`custom.css` is fetched per request from `/api/config/custom.css`, so a CSS
stacks/homepage/conf/custom.css
+874 -257
View File
File diff suppressed because one or more lines are too long
stacks/homepage/conf/custom.js
+124 -2
View File
@@ -1,2 +1,124 @@
// Custom JS for homepage. Empty placeholder — keep file present so
// homepage doesn't 404 on the asset.
// Homepage — Australis Skyfall theme switch.
//
// WHY THIS EXISTS INSTEAD OF HOMEPAGE'S OWN TOGGLE
// Homepage renders a built-in light/dark switch only when settings.yaml does
// NOT pin `theme:`. We cannot unpin it: removing the key makes the page's data
// loader throw, and its catch branch serves `initialSettings: {}` — a dashboard
// with no tab bar, no layout and no i18n. Measured on 2026-08-24: with
// `theme: dark` present the payload is correct within ~12s of a recreate;
// with the key removed, six recreates over seven minutes all came up empty,
// and putting the key back fixed it on the next try. So the key stays, and the
// switch is ours.
//
// HOW IT WORKS
// Skyfall keys its light theme off `[data-theme="light"]` on <html>. Homepage
// keeps its own `dark` class there regardless — that is fine and was verified:
// with `class="dark scheme-dark theme-slate"` AND `data-theme="light"`, every
// themed surface resolves to Skyfall Day, because our rules carry !important
// on the surfaces Tailwind's `dark:` variants would otherwise claim.
//
// Precedence, highest first:
// 1. an explicit choice stored here in localStorage
// 2. the OS preference, via @media (prefers-color-scheme) in custom.css
// 3. dark — Skyfall's first-class default
//
// The stylesheet handles 2 and 3 on its own, so this file is only responsible
// for 1. It writes `data-theme` ONLY when there is a stored choice; leaving the
// attribute absent is what lets the media query take over.
(() => {
"use strict";
const KEY = "skyfall-theme"; // "light" | "dark" | absent = follow the OS
const root = document.documentElement;
const stored = () => {
try {
const v = localStorage.getItem(KEY);
return v === "light" || v === "dark" ? v : null;
} catch {
return null; // private mode / storage disabled — fall through to the OS
}
};
const systemPrefersLight = () =>
window.matchMedia?.("(prefers-color-scheme: light)").matches ?? false;
const apply = (mode) => {
if (mode) root.setAttribute("data-theme", mode);
else root.removeAttribute("data-theme");
};
// Run before first paint where possible, so a stored light choice does not
// flash dark on load.
apply(stored());
const effective = () => stored() ?? (systemPrefersLight() ? "light" : "dark");
const ICON = {
// Lucide sun / moon, 1.75 stroke per Skyfall's iconography note.
light:
'<circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.93 4.93l1.41 1.41M17.66 17.66l1.41 1.41M2 12h2M20 12h2M6.34 17.66l-1.41 1.41M19.07 4.93l-1.41 1.41"/>',
dark: '<path d="M12 3a6 6 0 0 0 9 9 9 9 0 1 1-9-9Z"/>',
};
const button = document.createElement("button");
button.type = "button";
button.id = "skyfall-theme-toggle";
button.setAttribute("aria-label", "Toggle light and dark theme");
const paint = () => {
const mode = effective();
// Show the icon for the mode you would switch TO, which is the convention
// every OS theme switch uses.
const next = mode === "light" ? "dark" : "light";
button.title = `Switch to ${next} theme`;
button.setAttribute("aria-pressed", String(mode === "light"));
button.innerHTML =
'<svg viewBox="0 0 24 24" width="16" height="16" fill="none" ' +
'stroke="currentColor" stroke-width="1.75" stroke-linecap="round" ' +
`stroke-linejoin="round">${ICON[next]}</svg>`;
};
button.addEventListener("click", () => {
const next = effective() === "light" ? "dark" : "light";
try {
localStorage.setItem(KEY, next);
} catch {
/* storage unavailable — the choice just will not survive a reload */
}
apply(next);
paint();
});
// Track the OS while no explicit choice is stored, so the icon stays honest.
window
.matchMedia?.("(prefers-color-scheme: light)")
.addEventListener?.("change", () => {
if (!stored()) paint();
});
// Homepage renders client-side and rebuilds its header, so a one-shot
// querySelector at load usually finds nothing and would silently no-op.
// Watch until the header strip exists, then stop watching.
const mount = () => {
if (document.getElementById("skyfall-theme-toggle")?.isConnected) return true;
const host = document.querySelector(
"div.flex.flex-row.self-center.flex-wrap.justify-between",
);
if (!host) return false;
paint();
host.appendChild(button);
return true;
};
if (!mount()) {
const observer = new MutationObserver(() => {
if (mount()) observer.disconnect();
});
observer.observe(document.body, { childList: true, subtree: true });
// Backstop: never leave an observer running forever on a page that will
// not produce the host element.
setTimeout(() => observer.disconnect(), 30000);
}
})();
stacks/homepage/conf/services.yaml
+25 -19
View File
@@ -27,24 +27,37 @@
icon: mdi-filmstrip
siteMonitor: http://10.100.10.50:8090/healthz
description: Media drop + upload-for-pickup + the standing agent link board — nh3-dev, 24h TTL except kept boards
- Voice Design Studio:
href: http://10.100.79.3:8216/
icon: mdi-microphone
siteMonitor: http://10.100.79.3:8216/health
description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only
- The Henge:
href: http://park.phasefinal.com:8420/
icon: mdi-clipboard-check
siteMonitor: http://park.phasefinal.com:8420/healthz
description: Durable needs-attention / idea parking (stonehenge-park) — ana-docker
# Was its own one-card `Games` group, which burned a full 4-wide row on a
# single panel. It is an app you open; this is where apps you open live.
- Pterodactyl:
href: http://10.250.50.55/
icon: mdi-gamepad-square
siteMonitor: http://10.250.50.55
description: Game server panel
# The AI tab is fully Docker-auto-discovered. Each inference service carries
# a homepage.group=AI - <role> label on its compose file (AI - Inference,
# AI - Eval & Retrieval, AI - Gateways & Chat, AI - Speech (TTS),
# AI - Audio Tools, AI - Image & Media). Tab assignment, group order, and
# column counts live in settings.yaml. Do not add entries here or they'll
# double up. To move a service between AI groups, change the label on its
# compose file and recreate the container (labels only apply on recreate).
# The AI tab is otherwise fully Docker-auto-discovered. Each service carries a
# homepage.group=AI - <role> label on its compose file (AI - Gateways & Chat,
# AI - Studios, AI - Inference, AI - Eval & Retrieval, AI - Speech (TTS),
# AI - Audio Tools, AI - Dormant). Tab assignment, group order, columns and
# collapse state live in settings.yaml. Do not add a labelled container here as
# well or it renders twice. To move a service between AI groups, change the
# label on its compose file and recreate the container — labels are read at
# creation, so `restart` will not do it.
- AI - Studios:
# Manual entry — Voice Design Studio is a user-level systemd service on
# irv-ml1, not a Docker-labeled stack, so it cannot auto-discover. It sits
# with the other studios rather than in Apps: it is a workspace you open and
# produce something in, which is exactly what that group is for.
- Voice Design Studio:
href: http://10.100.79.3:8216/
icon: mdi-microphone
siteMonitor: http://10.100.79.3:8216/health
description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only
- Media:
- Plex:
@@ -67,13 +80,6 @@
key: '{{HOMEPAGE_VAR_JELLYFIN_KEY}}'
enableBlocks: true
- Games:
- Pterodactyl:
href: http://10.250.50.55/
icon: mdi-gamepad-square
siteMonitor: http://10.250.50.55
description: Game server panel
- Infra - ANA:
- ANA-Firewall:
href: https://10.250.250.1
stacks/homepage/conf/settings.yaml
+187 -85
View File
@@ -1,10 +1,28 @@
---
# https://gethomepage.dev/latest/configs/settings
title: ... all my base ...
# ⚠ `theme:` MUST STAY PINNED. REMOVING IT BREAKS THE WHOLE DASHBOARD.
#
# Unpinning this is the documented way to make Homepage render its own
# light/dark toggle, and it was tried on 2026-08-24 for exactly that reason.
# It does not work here: with the key absent, the page's data loader throws and
# its catch branch serves `initialSettings: {}` — no tab bar, no layout, no
# i18n, just a flat list of every group at once. Measured, not inferred: six
# force-recreates over seven minutes all came up empty with the key removed,
# and putting it back rendered correctly on the next recreate in 12 seconds.
# `/api/services` stays 200 and correct throughout, which is what makes this so
# easy to misdiagnose as a caching or warm-up problem.
#
# Light/dark still works — it just is not Homepage's switch. The stylesheet is
# dual-theme (Australis Skyfall ships both), driven by `data-theme` on <html>:
# conf/custom.js adds a toggle that writes it and remembers the choice, and
# theme/build.py also emits an OS-preference copy so an unset choice follows
# `prefers-color-scheme`. Dark stays Skyfall's first-class default.
#
# `color:` is pinned too, for a different reason: unpinning it adds a
# colour-ramp picker, and the ramp is not ours to choose — conf/custom.css
# carries the real palette. slate is the nearest cool neutral underneath it.
theme: dark
# `color:` only accepts Homepage's built-in Tailwind ramps, and none of them is
# the Australis Sea ramp. slate is the nearest cool neutral and acts as the
# base under conf/custom.css, which is where the real palette lives.
color: slate
# NO `background:` BLOCK — DELIBERATE, DO NOT RE-ADD WITHOUT ASKING.
@@ -14,15 +32,13 @@ color: slate
# for an Arbo-generated background and came out on 2026-08-19 when the
# operator called the result ugly.
#
# The image was also against canon on its own terms: Australis says "solid
# fills only on chrome — no full-bleed photography, no decorative gradients",
# and permits the aurora motif "never as a background fill behind text". A
# whole dashboard is text. The predecessor knew and dialled the opacity down
# instead of dropping it, which is how you end up with a quiet version of the
# wrong thing. The asset is still in `images/` if this is ever revisited.
#
# The aurora now appears exactly where canon sanctions it: a 1px accent edge
# under the tab bar, in conf/custom.css.
# The image was also against canon on its own terms: Skyfall says "flat
# semantic surfaces; no photography, no textures", and permits its one
# decorative motif — a subtle aurora gradient — on hero and empty-state areas
# only, "never behind body text blocks". A whole dashboard is a body text
# block. The predecessor knew and dialled the opacity down instead of dropping
# it, which is how you end up with a quiet version of the wrong thing. The
# asset is still in `images/` if this is ever revisited.
headerStyle: boxedWidgets
providers:
@@ -46,81 +62,129 @@ statusStyle: ""
# reads as a rendering fault.
useEqualHeights: false
# Function-first layout, four-tab split:
# Main - daily-use apps, media, bookmarks, monitoring
# AI - the inference fleet, grouped by role (see below)
# Infrastructure - hardware, hypervisors, BMCs (per site)
# Toolchain - backend services running but rarely clicked
# ===========================================================================
# THE ORGANISING QUESTION IS "DO I OPEN THIS?" — NOT "WHAT IS IT?"
# (operator, 2026-08-24: "most of the issues are that tools I use and have a
# UI are interspersed with API endpoints which are largely informational
# only. They might even go in their own cards or start collapsed.")
#
# The AI tab splits the fleet by function so a 20+ service list reads as
# sorted groups instead of one endless column. Group membership is set by
# the homepage.group=AI - <role> label on each service's compose file:
# AI - Inference LLM seats you call (gen, char-rp, char-rp-reasoning, summarizer)
# AI - Eval & Retrieval judges, reward, rerank, embed, image-quality
# AI - Gateways & Chat routing gateway, control plane, chat frontends
# AI - Speech (TTS) text-to-speech engines
# AI - Audio Tools speech-to-text + audio dataset tooling
# AI - Image & Media image/video generation + pipelines
# AI - Dormant stopped stacks (rollback seats, retired auditions)
# Every group on this board is one of two kinds, and they never mix:
#
# AI TAB ORDER IS BY CLICKABILITY, NOT BY IMPORTANCE (operator, 2026-08-18).
# Groups render in the order they appear in this block, so the top of the tab
# is prime real estate and it should hold the things you actually open in a
# browser — chat frontends, ComfyUI, the control plane. Most of the model
# seats below them are vLLM API endpoints whose href is a `/docs` page: they
# are worth SEEING (status at a glance) but not worth reaching for, so they
# sink. Order is therefore:
# interactive UIs -> mixed -> API-only seats -> dormant
# If you add an AI group, place it by asking "would I click this?", not by
# how central the service is to the fleet.
# TOOLS — you click the card and do something in the thing it opens.
# Expanded, and placed at the TOP of its tab.
# ENDPOINTS — an API, a broker, a background agent. Its href is a `/docs`
# page, a `/ping`, or nothing at all. The only thing you want from
# the card is "is it alive". `initiallyCollapsed: true`, and placed
# at the BOTTOM of its tab.
#
# COLUMN COUNTS ARE NOT A STYLE CHOICE — they are the member count.
# `columns: N` lays the group out N-per-row and leaves the remainder of the
# last row as dead space. A 1-member group at columns:4 renders one card and
# three empty cells, which is what made this dashboard look ragged before
# 2026-08-18. Rule: set columns to the member count, or to the divisor that
# leaves the smallest remainder. Re-check when a group gains or loses a
# service — `GET /api/services` prints the live per-group counts.
# A collapsed group is not hidden — the eyebrow and its rule still render, so
# the tab still tells you the thing exists, and one click expands it. That is
# the whole point: presence without cost.
#
# EVERY GROUP NEEDS A `tab:` — including bookmark groups. A group with no tab
# assignment renders on ALL FOUR TABS. That is how UltraSeedbox ended up
# repeated at the bottom of every tab (fixed 2026-08-18); it is Homepage
# behaviour, not a bug, and it will happen again to the next group added
# without a tab.
# When you add a service, ask "would I open this in a browser to get work
# done?" If no, it belongs in a collapsed endpoint group, no matter how
# central it is to the fleet. `AI - Inference` holds the seats the entire
# fleet runs on and it is collapsed, because you consume them through the
# gateway rather than by clicking them.
#
# Four tabs:
# Main - what you actually open day to day
# AI - AI tools up top, model/API seats collapsed below
# Infrastructure - hardware, hypervisors, BMCs (per site) — all consoles
# Toolchain - the plumbing, split by kind of plumbing
#
# ---------------------------------------------------------------------------
# GROUP MEMBERSHIP LIVES ON THE CONTAINER, NOT HERE.
#
# This block controls tab, order, columns and collapse. WHICH services are in
# a group is set by `homepage.group=` on each container's compose file, and
# labels only apply at container CREATION — moving a service between groups
# means editing the label and running `docker compose up -d <service>`, not
# `restart`. The 2026-08-24 pass did 28 of those; the playbooks that did it
# are `playbooks/homepage-regroup-<host>.yaml` and they are rerunnable.
#
# ⚠ THE MODEL SEATS ARE DELIBERATELY STILL NAMED `AI - Inference`,
# `AI - Eval & Retrieval`, `AI - Speech (TTS)` AND `AI - Audio Tools`.
# Renaming them to something like "AI API - …" would be clearer, and it would
# cost a recreate on sixteen GPU-backed seats — multi-minute model reloads on
# endpoints peers reach through the gateway. Order and `initiallyCollapsed`
# buy the same separation for free. Do not spend that recreate on a label.
#
# ---------------------------------------------------------------------------
# COLUMNS ARE 4 EVERYWHERE. DO NOT TUNE THEM PER GROUP.
#
# `columns: N` is not a density dial — it sets `lg:grid-cols-N` on that one
# group, so it fixes the CARD WIDTH for that group alone. Varying it per group
# means the card width changes every time you cross a group boundary, and a
# page whose grid keeps resizing as you scroll reads as broken layout even when
# every individual group is fine.
#
# The predecessor rule here was "columns = the member count", written to avoid
# the dead cells a 1-member group leaves in a 4-wide row. It trades one flaw
# for a worse one: at columns:1 a single service becomes a 1500px-wide bar
# holding six words, and at columns:2 a three-member group orphans its third
# card onto a half-empty row. Measured on the live board 2026-08-24 — the
# Notes, News, Media, AI - Image & Media and AI - Audio Tools groups were all
# rendering cards two to four times wider than the groups above and below them.
#
# A short last row is what a grid looks like. A card that is wider than its
# neighbours is what a mistake looks like. Uniform wins.
#
# Homepage's own responsive ramp (`grid-cols-1 md:grid-cols-2 lg:grid-cols-N`)
# still collapses this to 2-up and 1-up on narrow viewports, so 4 is a desktop
# maximum, not a hard floor.
#
# ---------------------------------------------------------------------------
# GROUPS ARE ALSO KEPT UNIFORM IN CARD HEIGHT, WHICH IS WHY ADGUARD AND
# TRAEFIK GOT THEIR OWN GROUPS.
#
# A service with a widget (AdGuard's query counts, Traefik's router counts,
# Uptime Kuma's uptime) renders a stat strip that makes its card ~50px taller
# than a plain link card. Put one of those in a row of three plain cards and
# you get a void under the plain ones — which is what made the old 13-member
# `Service Networking` group look broken. Split so that a group's members all
# have widgets or all do not, and every row comes out flush. That is the real
# reason `DNS & Filtering` (3 widget cards) and `Reverse Proxies` (2 widget
# cards) are separate from `Compose Consoles` (5 plain cards).
#
# ---------------------------------------------------------------------------
# EVERY GROUP NEEDS A `tab:` — including bookmark groups, and including groups
# that arrive from a `homepage.group=` container label rather than from this
# file. A group with no tab assignment renders on ALL FOUR TABS. That is how
# UltraSeedbox ended up repeated at the bottom of every tab (fixed 2026-08-18)
# and how Scriberr's `AI Systems` label did the same from 2026-08-23 (fixed
# 2026-08-24). It is Homepage behaviour, not a bug, and it will happen again to
# the next container labelled with a group name that does not appear below.
# `GET /api/services` prints the live group list — anything in it that is not a
# key here is currently leaking onto all four tabs.
# ===========================================================================
layout:
Notes:
icon: mdi-note-text-outline
# ---- Main: what you actually open -------------------------------------
# Replaces the old Notes (1 member) and News (2) bands, which each burned a
# full 4-wide row on a single card.
Daily:
icon: mdi-coffee-outline
tab: Main
style: row
columns: 1
News:
icon: mdi-rss
tab: Main
style: row
columns: 2
columns: 4
Monitoring:
icon: mdi-chart-line
tab: Main
style: row
columns: 4
# Absorbed the old one-card Games band (Pterodactyl). Lost SearXNG to Daily,
# the two chat frontends to the AI tab, and Mosquitto + the RustDesk relay to
# Agents (no UI) — neither of those has a page to open.
Apps:
icon: mdi-apps
tab: Main
style: row
columns: 4
# 2 wide, not 3 — Plex and Jellyfin carry stat-block widgets that get
# squeezed to unreadable at 4-across, and 4 members / 3 columns orphaned
# Jellyfin onto a row of its own.
Media:
icon: mdi-play-box-multiple
tab: Main
style: row
columns: 2
Games:
icon: mdi-gamepad-square
tab: Main
style: row
columns: 1
columns: 4
# Bookmarks. Listed here for the tab pin above all else — without it this
# group appears on every tab. `style: row` also turns the eight entries
# from full-width stacked bars into a compact grid.
@@ -129,43 +193,53 @@ layout:
tab: Main
style: row
columns: 4
# --- AI tab: ordered interactive -> API-only -> dormant (see note above) ---
# Things you open: chat frontends, the control plane, the LiteLLM UI.
# ---- AI: tools, then collapsed endpoints ------------------------------
# Chat frontends and the control plane — Lobe Chat and the ESH Open WebUI
# joined from Main on 2026-08-24; they are chat frontends and belong with the
# other chat frontends.
AI - Gateways & Chat:
icon: mdi-router-network
tab: AI
style: row
columns: 4
# ComfyUI is a full node editor and Arbo has a real UI — both get clicked.
AI - Image & Media:
icon: mdi-image-multiple
# Replaces `AI - Image & Media`. Everything here is a workspace you open and
# produce something in: ComfyUI's node editor, Arbo, Waterland, the YT
# Voice Clipper audition console, Scriberr's transcription UI.
AI - Studios:
icon: mdi-palette-outline
tab: AI
style: row
columns: 2
# Mixed: YT Voice Clipper has an audition console, Parakeet is an API.
AI - Audio Tools:
icon: mdi-waveform
tab: AI
style: row
columns: 2
# Below here: model seats whose href is a vLLM `/docs` page. Status at a
# glance is the whole value; you consume these through the gateway, not by
# clicking them.
columns: 4
# ---- collapsed from here down: seats you call, not pages you open ------
# Named `AI - Inference` rather than something clearer on purpose — see the
# warning above about what renaming these costs.
AI - Inference:
icon: mdi-brain
tab: AI
style: row
columns: 4
initiallyCollapsed: true
AI - Eval & Retrieval:
icon: mdi-scale-balance
tab: AI
style: row
columns: 5
columns: 4
initiallyCollapsed: true
AI - Speech (TTS):
icon: mdi-account-voice
tab: AI
style: row
columns: 4
initiallyCollapsed: true
# What is left of the old Audio Tools group after Scriberr and the YT Voice
# Clipper moved to Studios: the two ASR API seats.
AI - Audio Tools:
icon: mdi-waveform
tab: AI
style: row
columns: 4
initiallyCollapsed: true
# Stopped stacks kept for rollback / superseded seats / retired auditions.
# They stay 'created' (not running) via `docker compose up --no-start`, so
# they show here as offline cards and revive with `docker compose start`.
@@ -174,6 +248,9 @@ layout:
tab: AI
style: row
columns: 4
initiallyCollapsed: true
# ---- Infrastructure: every card is a console --------------------------
Infra - ANA:
icon: si-proxmox
tab: Infrastructure
@@ -183,24 +260,49 @@ layout:
icon: si-proxmox
tab: Infrastructure
style: row
columns: 3
columns: 4
Infra - IRV:
icon: mdi-brain
tab: Infrastructure
style: row
columns: 2
columns: 4
Infra - ESH:
icon: si-proxmox
tab: Infrastructure
style: row
columns: 4
Service Networking:
# ---- Toolchain: the plumbing, split by kind ---------------------------
# The old `Service Networking` group was thirteen members mixing three
# AdGuards, five Dockges, two Traefiks and four headless agents — widget
# cards next to plain ones next to things with no href at all. Split four
# ways on 2026-08-24.
DNS & Filtering:
icon: mdi-dns
tab: Toolchain
style: row
columns: 4
Reverse Proxies:
icon: mdi-transit-connection-variant
tab: Toolchain
style: row
columns: 4
Compose Consoles:
icon: mdi-docker
tab: Toolchain
style: row
columns: 4
Toolchain:
icon: mdi-toolbox
tab: Toolchain
style: row
columns: 4
# No href, or an href that is an API. CrowdSec, Mailrise, the restic
# rest-server, the Gitea Actions runner, the MQTT broker, the RustDesk relay.
# You never open these; you only ever want to know they are up.
Agents (no UI):
icon: mdi-cog-transfer-outline
tab: Toolchain
style: row
columns: 4
initiallyCollapsed: true
stacks/homepage/theme/australis.css.in
-528
View File
@@ -1,528 +0,0 @@
/* Homepage — Australis.
*
* SOURCE OF TRUTH: this file. Do not hand-edit `conf/custom.css`; it is
* generated. Run `theme/build.py` after changing anything here, then
* `scripts/deploy-stack.sh esh-docker-vm homepage --conf`.
*
* WHY A BUILD STEP: Homepage serves exactly two files out of its config
* directory, `/api/config/custom.css` and `/api/config/custom.js`. There is
* no static route for anything else, so a `@font-face` pointing at
* `theme/fonts/*.woff2` would 404 — the faces have to arrive inside the
* stylesheet as data: URIs. The build inlines them.
*
* TOKENS ARE CANONICAL, NOT DERIVED. Every hex below is copied verbatim from
* `~/.claude/skills/australis-design/colors_and_type.css`. The predecessor
* theme built a parallel OKLCH palette "derived from the Australis
* philosophy" rather than using the system's own values, which is how a
* design system quietly forks. `build.py` re-checks these against the skill
* file at build time when it is present and warns on drift.
*
* DIRECTION: instrument panel. This is engineering chrome, not a page with
* wallpaper. Canon: "the screen is the polar sky — empty, with light coming
* through it." Three moves carry it:
*
* 1. Group headings become the Australis eyebrow (mono / uppercase / 11px /
* 0.08em, cyan) with a hairline running to the right edge. Canon names
* the eyebrow a system signature and says to use it often; here it turns
* the groups into register bands instead of headings floating over a
* grid.
* 2. Status stops shouting. The filled emerald chips read louder than the
* service names they annotate; they become a semantic dot plus a mono
* micro-label at tertiary contrast.
* 3. Cards are bordered and opaque, sized to their own content. Canon puts
* border above shadow for grouping in chrome, and equal-height rows turn
* short cards into hollow boxes.
*
* WHAT WAS REMOVED, AND WHY IT IS NOT A REGRESSION: the previous theme
* carried a generated full-bleed aurora image behind the whole dashboard,
* added on the operator explicitly asking for an Arbo-generated background.
* Canon forbids exactly that ("solid fills only on chrome — no full-bleed
* photography, no decorative gradients", and the aurora motif "never as a
* background fill behind text"). The operator called the result ugly on
* 2026-08-19 and asked for a canonical-colors rebuild, so the image is gone.
* The aurora survives where canon actually sanctions it: a single 1px accent
* edge under the tab bar. If a background is ever wanted again, that is the
* conversation to reopen — not a quiet re-add.
*
* TYPE: Space Grotesk (display) / Inter (UI, body) / JetBrains Mono (all
* metadata) — canonical per the system README. Vendored as latin-subset
* VARIABLE woff2, one file per family covering the whole weight axis: 102 KB
* for three families against 56 KB for the three static Supreme cuts the
* predecessor shipped, and no Google Fonts request at page load.
*/
/* @@FONTS@@ */
:root {
/* Ice */
--aus-black: #222531;
--aus-white: #a9bcc3;
--aus-bright-white: #cce7ec;
/* Sea */
--aus-bright-black: #373b46;
--aus-dark-30: #414751;
--aus-dark-40: #565f69;
--aus-dark-50: #6e7882;
--aus-dark-60: #86929d;
--aus-bright-70: #9daeb6;
--aus-bright-80: #b3cbcf;
/* Aurora */
--aus-blue: #6388d8;
--aus-bright-blue: #a4c4ff;
--aus-cyan: #00b1a8;
--aus-bright-cyan: #42dcd1;
--aus-green: #16b866;
--aus-bright-green: #51e08a;
/* Dawn */
--aus-red: #ff491a;
--aus-yellow: #e1c631;
--aus-magenta: #9d78ff;
--bg-0: var(--aus-black);
--bg-1: var(--aus-bright-black);
--bg-2: var(--aus-dark-30);
--border-subtle: var(--aus-dark-30);
--border-default: var(--aus-dark-40);
--fg-0: var(--aus-bright-white);
--fg-1: var(--aus-white);
--fg-2: var(--aus-bright-70);
--fg-3: var(--aus-dark-60);
--fg-muted: var(--aus-dark-50);
--font-display: "Space Grotesk", ui-sans-serif, system-ui, sans-serif;
--font-sans: "Inter", ui-sans-serif, system-ui, sans-serif;
--font-mono: "JetBrains Mono", ui-monospace, "SF Mono", Menlo, monospace;
--tracking-caps: 0.08em;
--radius-sm: 4px;
--radius-md: 6px;
--radius-lg: 10px;
--shadow-1: 0 1px 2px rgba(10, 12, 18, 0.4);
--dur-fast: 120ms;
--ease-out: cubic-bezier(0.2, 0.8, 0.2, 1);
}
/* ---- Canvas --------------------------------------------------------------
Solid fill. Canon forbids full-bleed imagery and decorative gradient fills
behind text. The single sanctioned aurora moment is the hairline under the
header chrome, further down. */
html,
body,
#page_wrapper,
#background,
body > div {
background-color: var(--bg-0) !important;
background-image: none !important;
}
/* `#background` above is load-bearing, not defensive tidiness. Homepage
server-renders the wallpaper as an INLINE style on `<div id="background">`,
and Next.js caches that rendered page — so removing the `background:` block
from settings.yaml is not enough on its own. A restart does not clear it;
only a full recreate does, and recreating this container has its own cost
(the tab bar and i18n go missing for up to an hour and then heal on their
own). Enforcing it in the stylesheet is deterministic, immediate, and also
means the canvas stays correct if the setting is ever re-added by accident.
The deliberate absence of the config block is documented in settings.yaml. */
body {
font-family: var(--font-sans) !important;
color: var(--fg-1);
font-size: 15px;
-webkit-font-smoothing: antialiased;
}
/* ---- Group heading — the Australis eyebrow --------------------------------
Canon calls mono/uppercase/11px/0.08em the system's signature and says to
use it often. It replaces the oversized sans heading, and a hairline runs
from the label to the right edge so the groups read as register bands
rather than as titles floating over a grid. */
.services-group > button.group {
gap: 0;
margin-bottom: 4px;
margin-top: 10px;
}
.service-group-icon {
display: none !important;
}
h2.service-group-name {
font-family: var(--font-mono) !important;
font-size: 11px !important;
font-weight: 500 !important;
line-height: 1 !important;
text-transform: uppercase;
letter-spacing: var(--tracking-caps);
color: var(--aus-cyan) !important;
white-space: nowrap;
}
.services-group > button.group::after {
content: "";
flex: 1;
height: 1px;
margin-left: 12px;
background: var(--border-subtle);
}
/* The chevron sits after the rule, muted. */
.services-group > button.group svg {
color: var(--fg-muted);
margin-left: 8px;
width: 14px;
height: 14px;
}
/* ---- Service cards -------------------------------------------------------
Border > shadow for grouping in chrome (canon). Opaque fill, 1px border,
one soft shadow. Height is content-driven — equal-height stretching turns
short cards into hollow boxes. */
ul.services-list {
align-items: start !important;
row-gap: 8px;
}
li.service > .service-card {
height: auto !important;
background: var(--bg-1) !important;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-lg) !important;
box-shadow: var(--shadow-1) !important;
margin-bottom: 0 !important;
transition: background-color var(--dur-fast) var(--ease-out),
border-color var(--dur-fast) var(--ease-out);
}
li.service > .service-card:hover {
background: var(--bg-2) !important;
border-color: var(--border-default);
}
/* Service name — display face, tight. Description — body face, secondary. */
.service-name {
font-family: var(--font-display) !important;
font-size: 14px !important;
font-weight: 500 !important;
/* Pinned so the status cluster can be centred on this exact line rather
than floating against the card's top edge. Changing one without the
other breaks the alignment. */
line-height: 17px !important;
letter-spacing: -0.015em;
color: var(--fg-0) !important;
/* Reserve the status gutter. Matches --status-gutter below; a long name
ellipsises into it instead of colliding with the pill. */
padding-right: 78px !important;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
p.service-description {
font-family: var(--font-sans) !important;
font-size: 12.5px !important;
font-weight: 400 !important;
line-height: 17px !important;
color: var(--fg-3) !important;
margin-top: 3px;
/* Two lines minimum. Cards in a row now bottom-align for the common
one-line/two-line mix, which is what made the grid look ragged, without
bringing back equal-height stretching — that inflated short cards to
match a widget card twice their height, which was far worse. Descriptions
longer than two lines still grow; nothing is truncated. */
min-height: 34px;
white-space: normal;
}
.service-icon {
opacity: 0.9;
}
/* ---- Status — dots, not chips --------------------------------------------
The filled emerald pills read louder than the service names they annotate.
Replaced with a semantic dot plus a mono micro-label at tertiary contrast:
present when you look for it, silent when you are not. */
/* The status cluster is CENTRED ON THE SERVICE NAME'S LINE, not parked in the
card's top-right corner. Homepage positions it `absolute top-0 right-0`,
which floats it ~7px above the title's optical centre and reads as a
misalignment on every card. The offsets below reconstruct the title line's
geometry: .service-name sits 5px below the card's top edge and carries 8px
of its own top padding, so its 17px line box starts at 13px and centres at
21.5px. Giving the cluster that top and that height, and centring inside it,
puts the pill and the name on one optical line.
These three numbers move together — if the name's font-size or line-height
changes, recompute all of them. */
.service-tags {
top: 13px !important;
height: 17px;
align-items: center !important;
gap: 12px !important;
margin-top: 0 !important;
/* 14px right inset, against the 8px of text padding on the left — the
optical match, since the pill has no glyph hard against its edge. */
margin-right: 14px !important;
}
/* Every child of the cluster shares one baseline: status pills, latency
badges and container-stat buttons alike. */
.service-tag {
display: flex !important;
align-items: center !important;
height: 17px;
line-height: 17px !important;
}
.service-tag,
.docker-status,
.site-monitor-status,
.service-tag > div:not(.sr-only) {
background: transparent !important;
background-color: transparent !important;
padding: 0 !important;
border-radius: 0 !important;
box-shadow: none !important;
}
.docker-status > div,
.service-tag .text-\[8px\] {
display: flex !important;
align-items: center !important;
font-family: var(--font-mono) !important;
font-size: 9px !important;
font-weight: 500 !important;
line-height: 1 !important;
letter-spacing: var(--tracking-caps);
color: var(--fg-3) !important;
}
/* Flex-centred rather than nudged with vertical-align: the dot is an optical
round on a cap-height uppercase run, so it needs true centring, not a magic
pixel offset that only looks right at one font size. */
.docker-status > div::before {
content: "\25CF"; /* ● */
display: inline-block;
margin-right: 6px;
font-size: 7px;
line-height: 1;
color: var(--aus-dark-50);
}
.docker-status-healthy > div::before,
.docker-status-running > div::before {
color: var(--aus-green);
}
.docker-status-unhealthy > div::before,
.docker-status-exited > div::before,
.docker-status-dead > div::before {
color: var(--aus-red);
}
.docker-status-starting > div::before,
.docker-status-paused > div::before {
color: var(--aus-yellow);
}
/* Ping / latency — mono metadata, tabular, no chip. */
.site-monitor-status > div {
font-family: var(--font-mono) !important;
font-size: 9px !important;
font-weight: 400 !important;
letter-spacing: 0.04em;
text-transform: lowercase !important;
font-variant-numeric: tabular-nums;
color: var(--fg-muted) !important;
}
/* ---- Widget stat strip ---------------------------------------------------
Was nested boxes-in-a-box. Now a footer strip: hairline above, cells
divided by hairlines, value in the display face, label as a mono eyebrow.
The card stays one object instead of three. */
.service-container {
margin-top: 8px;
padding-top: 2px;
border-top: 1px solid var(--border-subtle);
gap: 0 !important;
}
.service-container > div {
background: transparent !important;
border-radius: 0 !important;
margin: 0 !important;
padding: 8px 4px 6px !important;
border-left: 1px solid var(--border-subtle);
}
.service-container > div:first-child {
border-left: 0;
}
/* value — nowrap because a long one (AdGuard's "32.767 ms") otherwise wraps
mid-cell and drags the whole card taller than its row-mates. */
.service-container > div > div:not(.uppercase) {
font-family: var(--font-display) !important;
font-size: 17px !important;
font-weight: 600 !important;
letter-spacing: -0.015em;
line-height: 1.2 !important;
white-space: nowrap;
font-variant-numeric: tabular-nums;
color: var(--fg-0) !important;
}
.service-container > div {
min-width: 0; /* let a long value shrink the cell instead of overflowing */
}
/* label */
.service-container > div > div.uppercase {
font-family: var(--font-mono) !important;
font-size: 9px !important;
font-weight: 400 !important;
letter-spacing: var(--tracking-caps);
color: var(--fg-3) !important;
margin-top: 2px;
}
/* ---- Tab bar -------------------------------------------------------------
The filled pill is replaced by a 2px top edge in aurora blue — canon's one
sanctioned accent-border pattern (and explicitly never a left border). */
ul.sm\:flex {
position: relative;
background: transparent !important;
background-color: transparent !important;
border-radius: 0 !important;
gap: 0;
overflow: visible;
}
/* The one moment of colour on the page. Canon sanctions the aurora glow —
bright-blue → bright-cyan → green — as an accent EDGE, never as a fill
behind text. A 1px rule under the tab bar is that edge, and it is the only
place the full aurora appears. */
ul.sm\:flex::after {
content: "";
position: absolute;
left: 0;
right: 0;
bottom: 0;
height: 1px;
background: linear-gradient(
90deg,
var(--aus-blue) 0%,
var(--aus-cyan) 42%,
var(--aus-green) 68%,
var(--border-subtle) 100%
);
opacity: 0.9;
}
ul.sm\:flex > li {
height: 40px !important;
border-radius: 0 !important;
background: transparent !important;
overflow: visible;
}
ul.sm\:flex > li > button {
margin: 0 !important;
height: 100%;
border-radius: 0 !important;
background: transparent !important;
background-color: transparent !important;
box-shadow: none !important;
border-top: 2px solid transparent;
font-family: var(--font-mono) !important;
font-size: 11px !important;
font-weight: 500 !important;
text-transform: uppercase;
letter-spacing: var(--tracking-caps);
color: var(--fg-3) !important;
transition: color var(--dur-fast) var(--ease-out);
}
ul.sm\:flex > li > button:hover {
color: var(--fg-1) !important;
background: transparent !important;
}
ul.sm\:flex > li > button.dark\:bg-white\/10 {
border-top-color: var(--aus-blue) !important;
color: var(--fg-0) !important;
background: linear-gradient(
180deg,
rgba(99, 136, 216, 0.12) 0%,
rgba(99, 136, 216, 0) 72%
) !important;
}
/* ---- Header chrome: resource widget + search -----------------------------
Same card treatment as everything else, so the header belongs to the page
instead of hovering above it. */
.information-widget-resource,
div.flex.flex-col.justify-center.mt-2 {
background: var(--bg-1) !important;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-lg) !important;
box-shadow: var(--shadow-1) !important;
}
.information-widget-resource {
border: 0 !important;
border-radius: 0 !important;
box-shadow: none !important;
}
/* The header strip that holds the resource widgets. */
div.flex.flex-row.self-center.flex-wrap.justify-between {
background: var(--bg-1) !important;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-lg);
box-shadow: var(--shadow-1);
}
.information-widget-resource .text-xs,
.information-widget-resource span {
font-family: var(--font-mono) !important;
font-size: 11px !important;
letter-spacing: 0.02em;
color: var(--fg-2) !important;
}
.resource-usage,
[class*="resource-usage"] {
background: var(--aus-black) !important;
height: 3px !important;
border-radius: 2px !important;
}
.resource-usage > div,
[class*="resource-usage"] > div {
background: var(--aus-cyan) !important;
border-radius: 2px !important;
}
/* Search input */
input[type="text"],
input[type="search"] {
font-family: var(--font-sans) !important;
font-size: 14px !important;
color: var(--fg-1) !important;
}
input::placeholder {
color: var(--fg-muted) !important;
font-family: var(--font-mono) !important;
font-size: 12px !important;
letter-spacing: 0.04em;
}
/* ---- Focus — the aurora glow, canon's signature interaction motif -------- */
a:focus-visible,
button:focus-visible,
input:focus-visible {
outline: none !important;
box-shadow: 0 0 0 3px rgba(99, 136, 216, 0.35) !important;
border-radius: var(--radius-sm);
}
stacks/homepage/theme/build.py
+157 -56
View File
@@ -1,46 +1,126 @@
#!/usr/bin/env python3
"""Generate conf/custom.css from the Australis source in this directory.
"""Generate conf/custom.css from the Australis Skyfall source in this directory.
Homepage serves exactly two files out of its config dir — custom.css and
custom.js — with no static route for anything alongside them. A @font-face
pointing at a vendored .woff2 would therefore 404, so the faces have to be
inlined as data: URIs. That is the whole reason this build step exists.
pointing at a vendored .woff2 would therefore 404, and an @import of a sibling
token file would 404 too. So this build does three things that cannot be done
in plain CSS:
1. inlines the font faces as data: URIs;
2. concatenates the vendored token files into one sheet;
3. bridges Skyfall's `[data-theme="light"]` selector to the `light` CLASS
Homepage actually sets on <html>.
Usage: python3 stacks/homepage/theme/build.py
Then: scripts/deploy-stack.sh esh-docker-vm homepage --conf
(a settings.yaml change needs a container recreate; a custom.css
change needs only a browser reload — custom.css is served per request)
"""
import base64
import hashlib
import pathlib
import re
import sys
HERE = pathlib.Path(__file__).resolve().parent
SRC = HERE / "australis.css.in"
OUT = HERE.parent / "conf" / "custom.css"
# The canonical token file. Present on a box with the australis-design skill
# installed; absent elsewhere, in which case the drift check is skipped rather
# than failing the build.
CANON = pathlib.Path.home() / ".claude/skills/australis-design/colors_and_type.css"
# ---------------------------------------------------------------------------
# VENDORED, VERBATIM, FROM THE SKYFALL BUNDLE. Concatenated in this order:
# families before semantics, tokens before bindings.
#
# The hashes are the anti-fork guard. The theme this one replaces built a
# parallel palette "derived from the Australis philosophy" instead of using the
# system's own values, and it had to be torn out twice. A vendored file is
# either byte-identical to the bundle or it is a fork wearing the bundle's
# name; there is no third state, so a mismatch FAILS the build rather than
# warning. To legitimately update one, drop in the new file and record its new
# hash in the same commit.
# ---------------------------------------------------------------------------
VENDORED = {
"colors.css": "75f78b674dd7fd87ac414cf04165f0e99b016e54d3b546f0e96378c08fdd7b88",
"layout.css": "a3826169b7a6b604d9c456c073ba3788297ae88bb703862a95171e3858f94c56",
"typography.css": "2ec1a667ced0653292eed0f724b8bc294765d740531ee10e48ebda5f5ca6f2f8",
}
# One VARIABLE face per family — the whole 100-900 axis in a single file, so
# there is no per-weight fan-out. Latin subset only.
FONTS = [
("Inter", "Inter-Variable.woff2"),
("Space Grotesk", "SpaceGrotesk-Variable.woff2"),
("JetBrains Mono", "JetBrainsMono-Variable.woff2"),
# Our own bindings — the only file in this directory that is ours to edit.
BINDINGS = "skyfall.css.in"
# ---------------------------------------------------------------------------
# Skyfall keys light mode off `[data-theme="light"]` on <html>. Homepage never
# touches data attributes, and — this is the constraint that shapes everything
# below — it will not give us its own theme toggle either: rendering that
# toggle requires settings.yaml to leave `theme:` unpinned, and an unpinned
# `theme:` makes the page's data loader throw (measured 2026-08-24; see
# conf/custom.js for the evidence). So `theme: dark` stays pinned, Homepage's
# class is always `dark`, and light mode is driven by `data-theme` written by
# our own toggle in custom.js, or by the OS preference when nothing is stored.
#
# Editing the vendored file to add those selectors would fork it; hand-copying
# its light block into the bindings would drift the moment the bundle updates.
# So re-emit the block here, mechanically, in three forms:
#
# [data-theme="light"], html.light explicit choice (our toggle; also the
# spelling a real Skyfall app would use,
# kept so the sheet stays portable)
# @media (prefers-color-scheme: light)
# html:not([data-theme=...]) the OS preference, and ONLY while no
# explicit choice is present — the :not()
# pair is what makes a stored "dark"
# survive a light-mode OS.
# ---------------------------------------------------------------------------
LIGHT_BLOCK = re.compile(r'^\[data-theme="light"\]\s*\{\n(.*?)^\}', re.M | re.S)
def expand_light(match: "re.Match[str]") -> str:
body = match.group(1).rstrip("\n")
return (
'[data-theme="light"],\nhtml.light {\n' + body + "\n}\n\n"
"@media (prefers-color-scheme: light) {\n"
' html:not([data-theme="dark"]):not([data-theme="light"]) {\n'
+ body
+ "\n }\n}"
)
# One VARIABLE face per stand-in family — the whole 100-900 axis in a single
# file — plus Supreme's three static cuts, which is how Supreme ships. Latin
# subset only. See the type note in skyfall.css.in for why two of the three
# families are substitutions.
FONTS_VARIABLE = [
("Space Grotesk", "SpaceGrotesk-Variable.woff2"), # stands in for Bespoke Sans
("JetBrains Mono", "JetBrainsMono-Variable.woff2"), # stands in for Victor Mono
]
FONTS_STATIC = [
("Supreme", "Supreme-400.woff2", 400),
("Supreme", "Supreme-500.woff2", 500),
# Supreme ships no 600 cut; 600 and 700 both resolve to this file.
("Supreme", "Supreme-700.woff2", "600 700"),
]
# Tokens the generated sheet cannot work without. A vendoring mistake that
# drops the :root block produces a stylesheet that parses fine and renders
# nothing, so assert on the values rather than trusting the copy.
REQUIRED_TOKENS = ["--aus-black", "--aus-cyan", "--bg-1", "--font-mono"]
# drops a block produces a stylesheet that parses fine and renders nothing, so
# assert on the values rather than trusting the copy. One from each vendored
# file, plus proof the light theme survived.
REQUIRED = [
"--sea-15", # colors.css, family layer
"--surface-card", # colors.css, semantic layer
"--shadow-sm", # layout.css
"--tracking-caps", # typography.css
"html.light", # the explicit-choice selector was emitted
"prefers-color-scheme: light", # the OS-preference copy was emitted
"#skyfall-theme-toggle", # the toggle has styling, not just behaviour
]
def sha256(path: pathlib.Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def font_faces() -> str:
out = []
for family, filename in FONTS:
for family, filename in FONTS_VARIABLE:
path = HERE / "fonts" / filename
if not path.exists():
sys.exit(f"missing font: {path}")
@@ -54,58 +134,79 @@ def font_faces() -> str:
f" font-display: swap;\n"
f"}}"
)
for family, filename, weight in FONTS_STATIC:
path = HERE / "fonts" / filename
if not path.exists():
sys.exit(f"missing font: {path}")
b64 = base64.b64encode(path.read_bytes()).decode("ascii")
out.append(
f"@font-face {{\n"
f' font-family: "{family}";\n'
f" src: url(data:font/woff2;base64,{b64}) format('woff2');\n"
f" font-weight: {weight};\n"
f" font-style: normal;\n"
f" font-display: swap;\n"
f"}}"
)
return "\n".join(out)
def check_canon_drift(css: str) -> None:
"""Warn if a token here no longer matches the Australis skill's value.
def vendored_css() -> str:
"""Concatenate the vendored token files, refusing to build if one moved."""
parts, drift = [], []
for name, expected in VENDORED.items():
path = HERE / name
if not path.exists():
sys.exit(
f"missing vendored token file: {path}\n"
f"Recover it with: git show 45c1995:stacks/homepage/theme/{name}"
)
actual = sha256(path)
if actual != expected:
drift.append((name, actual, expected))
parts.append(f"/* ==== vendored: {name} ({actual[:12]}) ==== */\n{path.read_text()}")
Non-fatal on purpose: the skill is a per-workstation install, so a missing
file is normal and must not break a build on a box that lacks it. A *value*
mismatch is worth shouting about — that is the palette forking, which is
exactly how the theme this one replaced went wrong.
"""
if not CANON.exists():
print(f"note: {CANON} not present — canonical drift check skipped")
return
canon = dict(re.findall(r"(--aus-[a-z0-9-]+):\s*(#[0-9a-fA-F]{6});", CANON.read_text()))
ours = dict(re.findall(r"(--aus-[a-z0-9-]+):\s*(#[0-9a-fA-F]{6});", css))
if not canon:
print(f"note: parsed no --aus-* tokens from {CANON} — drift check skipped")
return
drift = [(k, v, canon[k]) for k, v in ours.items() if k in canon and v.lower() != canon[k].lower()]
missing = sorted(set(ours) - set(canon))
for key, mine, theirs in drift:
print(f"DRIFT: {key} is {mine} here, {theirs} in the skill", file=sys.stderr)
for key in missing:
print(f"note: {key} is not a canonical token", file=sys.stderr)
if drift:
print(
f"WARNING: {len(drift)} token(s) diverge from the canonical palette. "
"Reconcile before deploying — a derived palette is how the previous "
"theme forked from the design system.",
file=sys.stderr,
for name, actual, expected in drift:
print(f"VENDOR DRIFT: {name}", file=sys.stderr)
print(f" on disk : {actual}", file=sys.stderr)
print(f" expected: {expected}", file=sys.stderr)
sys.exit(
"A vendored Skyfall token file has been modified. These are copied "
"verbatim from the bundle and are not ours to edit — put the "
"override in skyfall.css.in instead. If the bundle itself was "
"legitimately updated, record the new hash in VENDORED in the same "
"commit as the file."
)
else:
print(f"canonical palette check: {len(ours)} tokens match {CANON.name}")
print(f"vendored token files verified: {', '.join(VENDORED)}")
return "\n\n".join(parts)
def main() -> None:
if not SRC.exists():
sys.exit(f"missing source: {SRC}")
src = SRC.read_text()
src_path = HERE / BINDINGS
if not src_path.exists():
sys.exit(f"missing source: {src_path}")
src = src_path.read_text()
if "/* @@FONTS@@ */" not in src:
sys.exit(f"{SRC.name} lost its @@FONTS@@ placeholder")
sys.exit(f"{BINDINGS} lost its @@FONTS@@ placeholder")
css = src.replace("/* @@FONTS@@ */", font_faces())
tokens = vendored_css()
for token in REQUIRED_TOKENS:
bridged, n = LIGHT_BLOCK.subn(expand_light, tokens)
if not n:
sys.exit(
'no [data-theme="light"] rule found in the vendored tokens — the '
"light theme would silently never apply"
)
print(f'expanded {n} [data-theme="light"] block(s): explicit + OS-preference')
css = bridged + "\n\n" + src.replace("/* @@FONTS@@ */", font_faces())
for token in REQUIRED:
if token not in css:
sys.exit(f"generated css has no {token} — the token block did not survive")
sys.exit(f"generated css has no {token} — a token block did not survive")
check_canon_drift(css)
OUT.write_text(css)
print(f"wrote {OUT} ({len(css) / 1024:.0f} KB)")
stacks/homepage/theme/colors.css
+208
View File
@@ -0,0 +1,208 @@
/* ============================================================
Australis Skyfall — Color Tokens v2
A ground-up web palette derived from the Australis Dark
PHILOSOPHY (not its ANSI values). Built in OKLCH so the laws
are explicit and every family stays perceptually consistent:
- All hues are cooler than neutral; nothing warmer than amber.
- One lightness law across every chromatic family:
deep L 0.48 — light-theme fills & colored text
base L 0.66 — dark-theme fills (the working color)
bright L 0.80 — dark-theme colored text & highlights
- Chroma is tuned per hue so no family shouts over another.
- Neutrals (Sea ramp) drift in hue from ice-blue (265°) toward
ocean green (195°) as they brighten — the aurora signature.
- Medium / medium-high contrast for eye comfort.
Families: Aurora (blue, cyan, green — use generously, in that
order) and Dawn (amber, red, violet — semantic accents only:
warning, danger, AI).
============================================================ */
:root {
/* ---- Sea — neutral ramp (hue 265° → 195° as L rises) ---- */
--sea-10: oklch(0.23 0.02 265);
--sea-15: oklch(0.27 0.024 268); /* dark canvas — the Australis anchor */
--sea-20: oklch(0.31 0.022 262);
--sea-25: oklch(0.35 0.02 258);
--sea-30: oklch(0.4 0.018 252);
--sea-40: oklch(0.48 0.016 245);
--sea-50: oklch(0.56 0.015 235);
--sea-60: oklch(0.64 0.014 228);
--sea-70: oklch(0.72 0.014 220);
--sea-75: oklch(0.78 0.015 212);
--sea-80: oklch(0.83 0.017 205);
--sea-90: oklch(0.9 0.02 198);
--sea-94: oklch(0.94 0.014 195);
--sea-96: oklch(0.96 0.01 195);
--sea-98: oklch(0.98 0.006 195);
/* ---- Ice — main colors (aliases into the ramp) ---- */
--ice-black: var(--sea-15);
--ice-white: var(--sea-75);
--ice-bright-white: var(--sea-90);
/* ---- Aurora — primary families (blue > cyan > green) ---- */
--blue-deep: oklch(0.48 0.12 262);
--blue-base: oklch(0.66 0.12 262);
--blue-bright: oklch(0.8 0.09 262);
--cyan-deep: oklch(0.48 0.08 200);
--cyan-base: oklch(0.66 0.1 200);
--cyan-bright: oklch(0.8 0.1 200);
--green-deep: oklch(0.48 0.11 158);
--green-base: oklch(0.66 0.13 158);
--green-bright: oklch(0.8 0.13 158);
/* ---- Dawn — accent families (semantic use only) ---- */
--amber-deep: oklch(0.52 0.1 75);
--amber-base: oklch(0.7 0.12 78);
--amber-bright: oklch(0.82 0.13 82);
--red-deep: oklch(0.5 0.15 25);
--red-base: oklch(0.62 0.16 25);
--red-bright: oklch(0.78 0.11 28);
--violet-deep: oklch(0.5 0.15 292);
--violet-base: oklch(0.66 0.14 292);
--violet-bright: oklch(0.8 0.1 292);
}
/* ============================================================
Semantic aliases — DARK (default, first-class)
============================================================ */
:root {
color-scheme: dark;
/* Surfaces */
--surface-sunken: var(--sea-10);
--surface-base: var(--sea-15);
--surface-raised: var(--sea-20);
--surface-overlay: oklch(0.34 0.022 260);
--surface-card: var(--sea-20);
--surface-input: var(--sea-10);
--surface-scrim: oklch(0.17 0.02 265 / 0.72);
/* Text */
--text-heading: var(--sea-90);
--text-body: var(--sea-75);
--text-muted: var(--sea-60);
--text-faint: var(--sea-50);
--text-inverse: var(--sea-15);
--text-link: var(--blue-bright);
--text-link-hover: var(--sea-90);
/* Borders */
--border-subtle: var(--sea-25);
--border-default: var(--sea-30);
--border-strong: var(--sea-40);
--border-focus: var(--blue-base);
/* Accent (primary = aurora blue) */
--accent: var(--blue-base);
--accent-hover: oklch(0.71 0.11 262);
--accent-active: oklch(0.6 0.13 262);
--accent-bright: var(--blue-bright);
--accent-text: var(--blue-bright);
--accent-contrast: var(--sea-10);
--accent-soft: color-mix(in oklab, var(--blue-base) 16%, transparent);
--accent-soft-hover: color-mix(in oklab, var(--blue-base) 26%, transparent);
/* Secondary accent (aurora cyan) */
--secondary: var(--cyan-base);
--secondary-bright: var(--cyan-bright);
--secondary-text: var(--cyan-bright);
--secondary-soft: color-mix(in oklab, var(--cyan-base) 14%, transparent);
/* Semantic status */
--success: var(--green-base);
--success-text: var(--green-bright);
--success-soft: color-mix(in oklab, var(--green-base) 14%, transparent);
--warning: var(--amber-base);
--warning-text: var(--amber-bright);
--warning-soft: color-mix(in oklab, var(--amber-base) 13%, transparent);
--danger: var(--red-base);
--danger-hover: oklch(0.67 0.15 25);
--danger-text: var(--red-bright);
--danger-contrast: var(--sea-10);
--danger-soft: color-mix(in oklab, var(--red-base) 13%, transparent);
--info: var(--blue-base);
--info-text: var(--blue-bright);
--info-soft: color-mix(in oklab, var(--blue-base) 14%, transparent);
--ai: var(--violet-base);
--ai-text: var(--violet-bright);
--ai-soft: color-mix(in oklab, var(--violet-base) 13%, transparent);
/* Selection */
--selection-bg: var(--blue-base);
--selection-fg: var(--sea-10);
}
/* ============================================================
Semantic aliases — LIGHT ("Skyfall Day")
Same laws, inverted: surfaces at L 0.94–0.98, text at L 0.27–0.45,
chromatic fills & colored text drop to the deep (L 0.48) step.
============================================================ */
[data-theme="light"] {
color-scheme: light;
/* Surfaces */
--surface-sunken: var(--sea-94);
--surface-base: var(--sea-96);
--surface-raised: var(--sea-98);
--surface-overlay: #ffffff;
--surface-card: var(--sea-98);
--surface-input: #ffffff;
--surface-scrim: oklch(0.27 0.024 268 / 0.4);
/* Text */
--text-heading: var(--sea-15);
--text-body: oklch(0.38 0.02 255);
--text-muted: var(--sea-50);
--text-faint: var(--sea-60);
--text-inverse: var(--sea-90);
--text-link: var(--blue-deep);
--text-link-hover: var(--sea-15);
/* Borders */
--border-subtle: oklch(0.89 0.014 210);
--border-default: oklch(0.84 0.016 215);
--border-strong: var(--sea-75);
--border-focus: var(--blue-deep);
/* Accent */
--accent: var(--blue-deep);
--accent-hover: oklch(0.53 0.12 262);
--accent-active: oklch(0.44 0.12 262);
--accent-bright: var(--blue-base);
--accent-text: var(--blue-deep);
--accent-contrast: #ffffff;
--accent-soft: color-mix(in oklab, var(--blue-deep) 10%, transparent);
--accent-soft-hover: color-mix(in oklab, var(--blue-deep) 18%, transparent);
/* Secondary accent */
--secondary: var(--cyan-deep);
--secondary-bright: var(--cyan-base);
--secondary-text: var(--cyan-deep);
--secondary-soft: color-mix(in oklab, var(--cyan-deep) 9%, transparent);
/* Semantic status */
--success: var(--green-deep);
--success-text: var(--green-deep);
--success-soft: color-mix(in oklab, var(--green-deep) 10%, transparent);
--warning: var(--amber-deep);
--warning-text: var(--amber-deep);
--warning-soft: color-mix(in oklab, var(--amber-base) 16%, transparent);
--danger: var(--red-deep);
--danger-hover: oklch(0.55 0.16 25);
--danger-text: var(--red-deep);
--danger-contrast: #ffffff;
--danger-soft: color-mix(in oklab, var(--red-deep) 8%, transparent);
--info: var(--blue-deep);
--info-text: var(--blue-deep);
--info-soft: color-mix(in oklab, var(--blue-deep) 9%, transparent);
--ai: var(--violet-deep);
--ai-text: var(--violet-deep);
--ai-soft: color-mix(in oklab, var(--violet-deep) 9%, transparent);
/* Selection */
--selection-bg: var(--blue-base);
--selection-fg: #ffffff;
}
stacks/homepage/theme/fonts/Supreme-400.woff2
BIN
View File
Binary file not shown.
stacks/homepage/theme/fonts/Supreme-500.woff2
BIN
View File
Binary file not shown.
stacks/homepage/theme/fonts/Supreme-700.woff2
BIN
View File
Binary file not shown.
stacks/homepage/theme/layout.css
+72
View File
@@ -0,0 +1,72 @@
/* Australis Skyfall — spacing, radii, shadows, motion (v3 "calm depth") */
:root {
/* Spacing (4px base) */
--space-1: 4px;
--space-2: 8px;
--space-3: 12px;
--space-4: 16px;
--space-5: 20px;
--space-6: 24px;
--space-8: 32px;
--space-10: 40px;
--space-12: 48px;
--space-16: 64px;
--space-20: 80px;
--space-24: 96px;
--space-32: 128px;
/* Semantic spacing — airy content, compact chrome */
--pad-chrome: var(--space-2); /* app chrome: rails, toolbars, list rows */
--pad-content: var(--space-8); /* work surfaces: page bodies, card interiors breathe */
--gap-pane: var(--space-4); /* gutter between panes/cards on the canvas */
--measure: 68ch; /* long-form reading width */
/* Radii — moderate, calm */
--radius-xs: 6px;
--radius-sm: 8px;
--radius-md: 10px; /* buttons, inputs, menu items */
--radius-lg: 14px; /* cards, panes, popovers */
--radius-xl: 16px; /* dialogs, drawers, command palette */
--radius-full: 999px;
/* Depth — crisp 1px hairline + layered soft shadow underneath.
Every elevated surface pairs border: 1px solid var(--border-subtle)
with one of these. Shadows are cool-tinted and two-layer:
a tight contact shadow + a wide ambient falloff. */
--shadow-xs: 0 1px 2px oklch(0.13 0.02 265 / 0.22);
--shadow-sm: 0 1px 2px oklch(0.13 0.02 265 / 0.2), 0 2px 8px -2px oklch(0.13 0.02 265 / 0.24);
--shadow-md: 0 2px 4px oklch(0.13 0.02 265 / 0.22), 0 10px 28px -6px oklch(0.13 0.02 265 / 0.36);
--shadow-lg: 0 4px 8px oklch(0.13 0.02 265 / 0.26), 0 28px 64px -12px oklch(0.13 0.02 265 / 0.5);
--shadow-drawer: 0 8px 16px oklch(0.13 0.02 265 / 0.28), 0 32px 80px -8px oklch(0.13 0.02 265 / 0.55);
/* Glows — hero moments ONLY: empty states, featured cards, command palette */
--glow-accent: 0 0 72px -16px color-mix(in oklab, var(--blue-base) 42%, transparent);
--glow-secondary: 0 0 72px -16px color-mix(in oklab, var(--cyan-base) 38%, transparent);
--glow-ai: 0 0 72px -16px color-mix(in oklab, var(--violet-base) 40%, transparent);
/* Active rail — left accent bar on active nav / selected items */
--rail-active: inset 2px 0 0 0 var(--accent);
/* Focus ring */
--focus-ring: 0 0 0 2px var(--surface-base), 0 0 0 4px var(--border-focus);
/* Motion — calm, no bounce */
--ease-out: cubic-bezier(0.22, 1, 0.36, 1); /* @kind other */
--ease-in-out: cubic-bezier(0.65, 0, 0.35, 1); /* @kind other */
--duration-fast: 120ms; /* @kind other */
--duration-base: 180ms; /* @kind other */
--duration-slow: 300ms; /* @kind other */
--duration-drawer: 260ms; /* @kind other */
}
[data-theme="light"] {
--shadow-xs: 0 1px 2px oklch(0.3 0.02 255 / 0.07);
--shadow-sm: 0 1px 2px oklch(0.3 0.02 255 / 0.06), 0 2px 8px -2px oklch(0.3 0.02 255 / 0.08);
--shadow-md: 0 2px 4px oklch(0.3 0.02 255 / 0.06), 0 10px 28px -6px oklch(0.3 0.02 255 / 0.12);
--shadow-lg: 0 4px 8px oklch(0.3 0.02 255 / 0.07), 0 28px 64px -12px oklch(0.3 0.02 255 / 0.18);
--shadow-drawer: 0 8px 16px oklch(0.3 0.02 255 / 0.08), 0 32px 80px -8px oklch(0.3 0.02 255 / 0.22);
--glow-accent: 0 0 72px -16px color-mix(in oklab, var(--blue-deep) 26%, transparent);
--glow-secondary: 0 0 72px -16px color-mix(in oklab, var(--cyan-deep) 24%, transparent);
--glow-ai: 0 0 72px -16px color-mix(in oklab, var(--violet-deep) 25%, transparent);
}
stacks/homepage/theme/skyfall.css.in
+718
View File
@@ -0,0 +1,718 @@
/* Homepage — Australis Skyfall, dual theme.
*
* SOURCE OF TRUTH: this file plus the three VENDORED token files beside it.
* Do not hand-edit `conf/custom.css`; it is generated. Run `theme/build.py`
* after changing anything here, then
* `scripts/deploy-stack.sh esh-docker-vm homepage --conf`.
*
* ── WHAT IS VENDORED, AND WHY IT IS UNTOUCHABLE ──────────────────────────
* colors.css Skyfall v2 palette — Sea/Ice/Aurora/Dawn in OKLCH, plus
* the semantic layer for BOTH themes.
* layout.css v3 "calm depth" — spacing, radii, the shadow system,
* glows, the active rail, motion.
* typography.css families, scale, weights, leading, tracking.
*
* These three are copied VERBATIM from the Skyfall bundle and must stay that
* way. `build.py` hashes them and fails the build if a byte moves. Everything
* this dashboard needs on top of them lives in THIS file, expressed through
* the semantic layer — never against a raw family token, and never a literal
* colour. That rule is not fussiness: the theme this one replaces built its
* own parallel palette "derived from the philosophy", which is how a design
* system quietly forks, and it had to be torn out twice.
*
* ── DUAL THEME, AND THE ONE PIECE OF PLUMBING IT NEEDS ───────────────────
* Skyfall keys light mode off `[data-theme="light"]`. Homepage keys it off a
* `light` / `dark` CLASS on <html>, and does not touch data attributes. Rather
* than edit the vendored file (fork) or duplicate its light block by hand
* (drift), `build.py` rewrites the selector at build time so both spellings
* are honoured. See BRIDGE_SELECTOR there.
*
* Dark is `:root` and remains first-class. Light is Skyfall Day. Both are
* reachable from Homepage's own theme toggle, which appears because
* settings.yaml no longer pins `theme:`.
*
* ── TYPE: TWO DOCUMENTED SUBSTITUTIONS ───────────────────────────────────
* Skyfall names Bespoke Sans (display) / Supreme (body, UI) / Victor Mono
* Nerd Font (data, code). Only Supreme was ever vendored into this repo, and
* Skyfall's own notes call Victor Mono "user-supplied", so substituting is
* expected rather than a deviation:
* Supreme 400/500/700 — CANONICAL, the body and UI face.
* Space Grotesk variable — stands in for Bespoke Sans (display).
* JetBrains Mono variable — stands in for Victor Mono (data, metadata).
* Both stand-ins are neutral grotesques in the right register. Swap them the
* moment the real faces arrive: it is a two-line change in build.py's FONTS
* and the --font-display / --font-mono overrides below.
*
* ── WHERE THIS DEPARTS FROM THE PREVIOUS THEME, DELIBERATELY ─────────────
* 1. Depth is now the RECIPE, not a choice: every elevated surface carries a
* 1px hairline AND a two-layer shadow, never one without the other.
* 2. Radii move to Skyfall's scale — cards at --radius-lg (14px), controls at
* --radius-md (10px).
* 3. The full-width aurora ribbon under the tab bar is GONE. Skyfall sanctions
* exactly two accent expressions — the active rail and hero-only glows —
* and a decorative gradient across the chrome is neither. The colour it was
* carrying now lands where canon puts it: a 2px accent bar plus an
* --accent-soft fill on the active tab. If you want the ribbon back, that
* is a deliberate documented deviation, not an oversight.
* 4. Widget stat values move from the display face to mono. Skyfall is
* explicit: numbers and telemetry are always --font-mono.
*/
/* @@FONTS@@ */
/* ---- Family substitutions ------------------------------------------------
Overriding typography.css's declarations rather than editing it, so the
vendored file stays byte-identical and the swap is visible in one place. */
:root {
--font-display: "Space Grotesk", "Bespoke Sans", ui-sans-serif, sans-serif;
--font-body: "Supreme", ui-sans-serif, system-ui, sans-serif;
--font-mono: "JetBrains Mono", "VictorMono Nerd Font", ui-monospace, monospace;
/* Width the status cluster occupies in a card's top-right. Load-bearing in
two places that must agree: the inset reserved on the title line, and the
negative margin the description uses to opt back out of it. Sized for the
widest real cluster — a latency badge plus a status word. */
--status-gutter: 96px;
}
/* ---- Icon colour ---------------------------------------------------------
Homepage draws every mdi and simple-icons glyph as a MASK over a gradient
fill, so this ramp IS the icon colour — `color:` does nothing to them.
⚠ These must be space-separated sRGB CHANNELS, because Homepage consumes
them as `rgb(var(--color-logo-start))`. An oklch() token cannot be fed
through that, so the four values below are the exact sRGB conversions of
real Skyfall tokens rather than colours anyone picked by eye:
dark --sea-80 oklch(0.83 0.017 205) -> 187 203 204
--blue-base oklch(0.66 0.12 262) -> 105 145 220
light --sea-40 oklch(0.48 0.016 245) -> 86 95 102
--blue-deep oklch(0.48 0.12 262) -> 54 91 161
Recompute with theme/oklch_to_srgb.py if a token ever changes.
Light uses the deep (L 0.48) step for both stops, per Skyfall's lightness
law: deep is the light-theme value for fills and coloured marks.
⚠ NOT `:root`. Homepage sets these same two variables on `.theme-slate`,
and that class sits on <html> — the very element `:root` matches. Same
element, so the cascade decides, and `.theme-slate` (0,1,0) outranks
`:root` (0,0,1): a `:root` override here is silently ignored. `html[class]`
is (0,1,1) and wins without hard-coding which `theme-*` class is active.
Specificity alone is not enough either — a custom property resolves from
the NEAREST ancestor that sets it, so this has to land on <html>. */
html[class] {
--color-logo-start: 187 203 204;
--color-logo-stop: 105 145 220;
}
html[data-theme="light"],
html.light {
--color-logo-start: 86 95 102;
--color-logo-stop: 54 91 161;
}
/* ---- Canvas --------------------------------------------------------------
The app canvas is --surface-sunken and work floats on it as cards. Flat
semantic fill only: Skyfall forbids full-bleed imagery and decorative
gradients behind text.
`#background` is load-bearing, not defensive tidiness. Homepage
server-renders the wallpaper as an INLINE style on `<div id="background">`
and Next.js caches that rendered page, so removing the `background:` block
from settings.yaml is not enough on its own — a restart does not clear it,
only a full recreate does. Enforcing it here is deterministic and immediate,
and keeps the canvas correct if the setting is ever re-added by accident. */
html,
body,
#page_wrapper,
#background,
body > div {
background-color: var(--surface-sunken) !important;
background-image: none !important;
}
body {
font-family: var(--font-body) !important;
font-size: var(--text-base);
line-height: var(--leading-normal);
color: var(--text-body);
-webkit-font-smoothing: antialiased;
}
/* ---- Group heading — the Skyfall eyebrow ---------------------------------
Mono, uppercase, --text-2xs, --tracking-caps, in the secondary accent, with
a hairline running to the right edge so groups read as register bands
rather than titles floating over a grid. ALL-CAPS is sanctioned here
specifically: Skyfall allows it for tiny tracked eyebrow labels and table
headers, and nowhere else.
Bookmark groups get the identical treatment. They are a separate component
with a parallel class family, and none of these selectors matched it before
— so UltraSeedbox rendered with Homepage's stock 20px sans heading while
every service group above it wore the eyebrow. One un-themed heading in a
column of themed ones is more conspicuous than the whole page being
un-themed. */
.services-group > button.group,
.bookmark-group > button.group {
gap: 0;
margin-bottom: var(--space-1);
margin-top: var(--space-6);
}
/* …except the first group on a tab, which should sit up against the tab bar
rather than floating below it. */
.services-group:first-of-type > button.group {
margin-top: var(--space-2);
}
.service-group-icon,
.bookmark-group-icon {
display: none !important;
}
h2.service-group-name,
h2.bookmark-group-name {
font-family: var(--font-mono) !important;
font-size: var(--text-2xs) !important;
font-weight: var(--weight-medium) !important;
line-height: 1 !important;
text-transform: uppercase;
letter-spacing: var(--tracking-caps);
color: var(--secondary-text) !important;
white-space: nowrap;
display: flex;
}
.services-group > button.group::after,
.bookmark-group > button.group::after {
content: "";
flex: 1;
height: 1px;
margin-left: var(--space-3);
background: var(--border-subtle);
}
/* The chevron sits after the rule, muted. */
.services-group > button.group svg,
.bookmark-group > button.group svg {
color: var(--text-faint);
margin-left: var(--space-2);
width: 14px;
height: 14px;
}
/* ---- Service cards — the calm-depth recipe -------------------------------
HAIRLINE + TWO-LAYER SHADOW, NEVER ONE WITHOUT THE OTHER. That pairing is
Skyfall's defining move; a card with a border and no shadow, or a shadow
and no border, is off-system. Height stays content-driven — equal-height
stretching turns short cards into hollow boxes. */
ul.services-list {
align-items: start !important;
row-gap: var(--space-2);
}
li.service > .service-card {
height: auto !important;
background: var(--surface-card) !important;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-lg) !important;
box-shadow: var(--shadow-sm) !important;
margin-bottom: 0 !important;
transition: background-color var(--duration-fast) var(--ease-out),
border-color var(--duration-fast) var(--ease-out),
box-shadow var(--duration-fast) var(--ease-out);
}
/* Hover: surface lifts a step and the shadow deepens sm -> md, per canon.
No scale, no translate — Skyfall has no bounce anywhere. */
li.service > .service-card:hover {
background: var(--surface-raised) !important;
border-color: var(--border-default);
box-shadow: var(--shadow-md) !important;
}
/* Service name — display face, tight. Description — body face, muted.
*
* ⚠ `.service-name` IS NOT THE TITLE. Homepage puts the title in as a bare
* text node and then nests `<p class="service-description">` as a sibling
* inside the SAME div, so every rule here lands on a box containing both. The
* title has no element of its own and cannot be selected — which is why it
* cannot be given its own truncation, and why the description needs an
* explicit escape hatch from anything the title needs. */
.service-name {
font-family: var(--font-display) !important;
font-size: var(--text-base) !important;
font-weight: var(--weight-medium) !important;
/* Pinned so the status cluster can be centred on this exact line rather
than floating against the card's top edge. Changing one without the
other breaks the alignment. */
line-height: 17px !important;
letter-spacing: var(--tracking-tight);
color: var(--text-heading) !important;
/* THE GUTTER MUST BE HELD BY WRAPPING, NOT BY CLIPPING.
This rule used to read `padding-right: 78px; overflow: hidden;
text-overflow: ellipsis; white-space: nowrap` and it did not work: CSS
clips overflow at the PADDING box, not the content box, so the reserved
78px was spill room the title printed straight through. Six cards on the
AI tab rendered their name underneath their own status pill. The ellipsis
never appeared either: it is painted by the block whose own line
overflows, and that block here is the anonymous box around the text node,
which does not carry the `overflow`. Wrapping has neither problem — line
breaking respects the content box, so the gutter genuinely holds. */
padding-right: var(--status-gutter) !important;
white-space: normal;
overflow: visible;
}
p.service-description {
font-family: var(--font-body) !important;
font-size: var(--text-xs) !important;
font-weight: var(--weight-regular) !important;
line-height: 17px !important;
color: var(--text-muted) !important;
margin-top: 3px;
white-space: normal;
/* Opt back out of the title's gutter. The status cluster is absolutely
positioned over the FIRST line only (top: 13px, height: 17px, so it
occupies y 13-30); the description never starts above y 33. Nothing can
collide, so it takes the full card width back and the gutter costs no
description space at all. */
margin-right: calc(var(--space-2) - var(--status-gutter));
/* Floor of two lines, ceiling of three. The floor keeps the common
one-line/two-line mix bottom-aligned, which is most of what made rows
look ragged. The ceiling is the other half: a handful of four- and
five-line descriptions were dragging their entire row 30-50px taller than
every card beside them, and at four cards across that gap is the most
visible defect on the page. Not equal-heights — that inflates short cards
into hollow boxes to match a widget card twice their height. */
min-height: 34px;
display: -webkit-box;
-webkit-line-clamp: 3;
-webkit-box-orient: vertical;
overflow: hidden;
}
/* ---- Status — semantic dots, not chips -----------------------------------
The stock filled emerald pills read louder than the service names they
annotate. A semantic dot plus a mono micro-label at faint contrast: present
when you look for it, silent when you are not. The dot colours come from
--success/--danger/--warning-text, so they take the deep step automatically
in Skyfall Day rather than staying dark-theme bright on a white card.
The cluster is CENTRED ON THE SERVICE NAME'S LINE, not parked in the card's
top-right corner. Homepage positions it `absolute top-0 right-0`, which
floats it ~7px above the title's optical centre and reads as a misalignment
on every card. The offsets below reconstruct the title line's geometry:
.service-name sits 5px below the card's top edge and carries 8px of its own
top padding, so its 17px line box starts at 13px and centres at 21.5px.
These three numbers move together — if the name's font-size or line-height
changes, recompute all of them. */
.service-tags {
top: 13px !important;
height: 17px;
align-items: center !important;
gap: var(--space-3) !important;
margin-top: 0 !important;
/* 14px right inset, against the 8px of text padding on the left — the
optical match, since the pill has no glyph hard against its edge. */
margin-right: 14px !important;
}
/* Every child of the cluster shares one baseline: status pills, latency
badges and container-stat buttons alike. */
.service-tag {
display: flex !important;
align-items: center !important;
height: 17px;
line-height: 17px !important;
}
.service-tag,
.docker-status,
.site-monitor-status,
.service-tag > div:not(.sr-only) {
background: transparent !important;
background-color: transparent !important;
padding: 0 !important;
border-radius: 0 !important;
box-shadow: none !important;
}
.docker-status > div,
.service-tag .text-\[8px\] {
display: flex !important;
align-items: center !important;
font-family: var(--font-mono) !important;
font-size: 9px !important;
font-weight: var(--weight-medium) !important;
line-height: 1 !important;
letter-spacing: var(--tracking-caps);
color: var(--text-muted) !important;
}
/* Flex-centred rather than nudged with vertical-align: the dot is an optical
round on a cap-height uppercase run, so it needs true centring, not a magic
pixel offset that only looks right at one font size. */
.docker-status > div::before {
content: "\25CF"; /* ● */
display: inline-block;
margin-right: 6px;
font-size: 7px;
line-height: 1;
color: var(--text-faint);
}
/* ⚠ Homepage emits `docker-status-<state>`, not `status-<state>`. Selectors
matching the latter silently hit nothing — which is how an earlier pass
shipped "themed" status pills that were still stock green. */
.docker-status-healthy > div::before,
.docker-status-running > div::before {
color: var(--success-text);
}
.docker-status-unhealthy > div::before,
.docker-status-exited > div::before,
.docker-status-dead > div::before {
color: var(--danger-text);
}
.docker-status-starting > div::before,
.docker-status-paused > div::before {
color: var(--warning-text);
}
/* Ping / latency — mono metadata, tabular, no chip. Deliberately the info
family and not a status colour: "how fast" must not read as "is it alive". */
.site-monitor-status > div {
font-family: var(--font-mono) !important;
font-size: 9px !important;
font-weight: var(--weight-regular) !important;
letter-spacing: var(--tracking-wide);
text-transform: lowercase !important;
font-variant-numeric: tabular-nums;
color: var(--text-faint) !important;
}
/* ---- Widget stat strip ---------------------------------------------------
A footer strip inside the card: hairline above, cells divided by hairlines,
value then a mono eyebrow label. The card stays one object instead of three
nested boxes.
Values are MONO, not the display face. Skyfall is explicit that numbers and
telemetry always take --font-mono; hierarchy against the label comes from
weight and size, which is the same tool the type system uses everywhere
else. Padding is tight because a stat strip adds its full height to its own
card and to nothing else in the row — at four cards across, one AdGuard
card opens a void beside three ordinary ones. It cannot be made free, but
it can be made small. */
.service-container {
margin-top: var(--space-2);
padding-top: 2px;
border-top: 1px solid var(--border-subtle);
gap: 0 !important;
}
.service-container > div {
background: transparent !important;
border-radius: 0 !important;
margin: 0 !important;
padding: 6px 4px 5px !important;
border-left: 1px solid var(--border-subtle);
min-width: 0; /* let a long value shrink the cell instead of overflowing */
}
.service-container > div:first-child {
border-left: 0;
}
/* value — nowrap because a long one (AdGuard's "32.767 ms") otherwise wraps
mid-cell and drags the whole card taller than its row-mates. */
.service-container > div > div:not(.uppercase) {
font-family: var(--font-mono) !important;
font-size: var(--text-sm) !important;
font-weight: var(--weight-bold) !important;
letter-spacing: var(--tracking-tight);
line-height: 1.2 !important;
white-space: nowrap;
font-variant-numeric: tabular-nums;
color: var(--text-heading) !important;
}
/* label */
.service-container > div > div.uppercase {
font-family: var(--font-mono) !important;
font-size: 9px !important;
font-weight: var(--weight-regular) !important;
letter-spacing: var(--tracking-caps);
color: var(--text-muted) !important;
margin-top: 2px;
}
/* Trailing widget rows — Jellyfin's "No Active Streams" strip and its empty
second row, which Homepage emits AFTER .service-container as its own block
of Tailwind-tinted pills. Untouched they broke out of the card's language:
two light grey bars with their own radius hanging under an otherwise
hairline-ruled footer. Not hidden — "no active streams" is the answer to
the question the card exists to answer. */
li.service .service-container ~ div > div {
background: transparent !important;
border-radius: 0 !important;
border-top: 1px solid var(--border-subtle);
height: auto !important;
min-height: 16px;
margin-top: 0 !important;
}
li.service .service-container ~ div span {
position: static !important;
display: block;
padding: 3px 4px 2px !important;
margin: 0 !important;
font-family: var(--font-mono) !important;
font-size: 9px !important;
letter-spacing: var(--tracking-caps);
text-transform: uppercase;
color: var(--text-muted) !important;
}
/* ---- Bookmarks -----------------------------------------------------------
Same depth recipe as a service card. The name takes the display face and
the description drops to mono metadata, so a bookmark row reads as a
quieter sibling of a service card rather than a different widget that
wandered in. */
li.bookmark > a {
background: var(--surface-card) !important;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-lg) !important;
box-shadow: var(--shadow-sm) !important;
margin-bottom: var(--space-2) !important;
overflow: hidden;
transition: background-color var(--duration-fast) var(--ease-out),
border-color var(--duration-fast) var(--ease-out),
box-shadow var(--duration-fast) var(--ease-out);
}
li.bookmark > a:hover {
background: var(--surface-raised) !important;
border-color: var(--border-default);
box-shadow: var(--shadow-md) !important;
}
.bookmark-icon {
background: transparent !important;
border-radius: 0 !important;
border-right: 1px solid var(--border-subtle);
}
.bookmark-name {
font-family: var(--font-display) !important;
font-size: var(--text-sm) !important;
font-weight: var(--weight-medium) !important;
color: var(--text-heading) !important;
}
.bookmark-description {
font-family: var(--font-mono) !important;
font-size: 10px !important;
letter-spacing: var(--tracking-wide);
color: var(--text-muted) !important;
}
/* ---- Tab bar — the active rail -------------------------------------------
Skyfall sanctions exactly two accent expressions: the active rail (a 2px
accent bar on the active item, paired with an --accent-soft fill) and
hero-only glows. This is the rail, turned through 90° for a horizontal bar.
The previous theme ran a full-width blue -> cyan -> green aurora gradient
under this bar. It was the nicest thing on the page and it is gone on
purpose: a decorative gradient across the chrome is neither of the two
sanctioned expressions, and Skyfall permits the aurora motif only on
hero/empty-state areas, never behind body text. A plain hairline separates
chrome from content instead, and the colour lands on the active tab. */
ul.sm\:flex {
position: relative;
background: transparent !important;
background-color: transparent !important;
border-radius: 0 !important;
gap: 0;
overflow: visible;
}
ul.sm\:flex::after {
content: "";
position: absolute;
left: 0;
right: 0;
bottom: 0;
height: 1px;
background: var(--border-subtle);
}
ul.sm\:flex > li {
height: 40px !important;
border-radius: 0 !important;
background: transparent !important;
overflow: visible;
}
ul.sm\:flex > li > button {
margin: 0 !important;
height: 100%;
border-radius: 0 !important;
background: transparent !important;
background-color: transparent !important;
box-shadow: none !important;
border-top: 2px solid transparent;
font-family: var(--font-mono) !important;
font-size: var(--text-2xs) !important;
font-weight: var(--weight-medium) !important;
text-transform: uppercase;
letter-spacing: var(--tracking-caps);
color: var(--text-muted) !important;
transition: color var(--duration-fast) var(--ease-out),
background-color var(--duration-fast) var(--ease-out);
}
ul.sm\:flex > li > button:hover {
color: var(--text-body) !important;
background: var(--accent-soft) !important;
}
/* The active rail: 2px accent bar + --accent-soft fill. `dark:bg-white/10` is
the class Homepage puts on the selected tab; it is the only handle it
gives. */
ul.sm\:flex > li > button.dark\:bg-white\/10 {
border-top-color: var(--accent) !important;
color: var(--text-heading) !important;
background: var(--accent-soft) !important;
}
/* ---- Header chrome: resource widget + search -----------------------------
Same depth recipe as everything else, so the header belongs to the page
instead of hovering above it. --pad-chrome, not --pad-content: this is app
chrome, and Skyfall's negative-space rule is compact chrome / airy content. */
div.flex.flex-col.justify-center.mt-2,
div.flex.flex-row.self-center.flex-wrap.justify-between {
background: var(--surface-card) !important;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-lg) !important;
box-shadow: var(--shadow-sm) !important;
}
/* The resource widget is INSIDE that strip, so it must not repeat the recipe
— nested hairline-and-shadow reads as a box in a box. */
.information-widget-resource {
border: 0 !important;
border-radius: 0 !important;
box-shadow: none !important;
background: transparent !important;
}
/* The CPU / memory / disk glyphs. Homepage colours them from its own Tailwind
theme, which lands almost invisible on Skyfall Day's near-white chrome. */
.information-widget-resource svg {
color: var(--text-muted) !important;
opacity: 1 !important;
}
.information-widget-resource .text-xs,
.information-widget-resource span {
font-family: var(--font-mono) !important;
font-size: var(--text-2xs) !important;
letter-spacing: var(--tracking-wide);
font-variant-numeric: tabular-nums;
color: var(--text-muted) !important;
}
.resource-usage,
[class*="resource-usage"] {
background: var(--surface-input) !important;
height: 3px !important;
border-radius: var(--radius-full) !important;
}
.resource-usage > div,
[class*="resource-usage"] > div {
background: var(--secondary) !important;
border-radius: var(--radius-full) !important;
}
/* Search input */
input[type="text"],
input[type="search"] {
font-family: var(--font-body) !important;
font-size: var(--text-base) !important;
color: var(--text-body) !important;
background: transparent !important;
}
input::placeholder {
color: var(--text-faint) !important;
font-family: var(--font-mono) !important;
font-size: var(--text-xs) !important;
letter-spacing: var(--tracking-wide);
}
/* ---- Theme toggle --------------------------------------------------------
Our own switch, injected by conf/custom.js. Homepage's built-in one is
unavailable: rendering it needs `theme:` unpinned in settings.yaml, and an
unpinned `theme:` makes the page's data loader throw — see the note in
custom.js for the measurement. Sized and shaped like an IconButton from the
Skyfall component set: --radius-md, chrome padding, hairline, and the same
accent-soft hover every interactive surface uses. */
#skyfall-theme-toggle {
display: inline-flex;
align-items: center;
justify-content: center;
width: 30px;
height: 30px;
margin-left: auto;
align-self: center;
flex: 0 0 auto;
border: 1px solid var(--border-subtle);
border-radius: var(--radius-md);
background: transparent;
color: var(--text-muted);
cursor: pointer;
transition: color var(--duration-fast) var(--ease-out),
background-color var(--duration-fast) var(--ease-out),
border-color var(--duration-fast) var(--ease-out);
}
#skyfall-theme-toggle:hover {
color: var(--text-heading);
background: var(--accent-soft);
border-color: var(--border-default);
}
#skyfall-theme-toggle:active {
background: var(--accent-soft-hover);
}
/* ---- Focus — Skyfall's ring ---------------------------------------------- */
a:focus-visible,
button:focus-visible,
input:focus-visible {
outline: none !important;
box-shadow: var(--focus-ring) !important;
border-radius: var(--radius-md);
}
/* ---- Version / footer ---------------------------------------------------- */
#version,
#version * {
font-family: var(--font-mono) !important;
font-size: var(--text-2xs) !important;
color: var(--text-faint) !important;
}
/* ---- Reduced motion — canon requires honouring it ------------------------ */
@media (prefers-reduced-motion: reduce) {
*,
*::before,
*::after {
transition-duration: 0.01ms !important;
animation-duration: 0.01ms !important;
}
}
stacks/homepage/theme/typography.css
+40
View File
@@ -0,0 +1,40 @@
/* Australis Skyfall — Typography tokens */
:root {
/* Families */
--font-display: "Bespoke Sans", "Avenir Next", "Segoe UI", sans-serif;
--font-body: "Supreme", "Helvetica Neue", "Segoe UI", sans-serif;
--font-mono: "VictorMono Nerd Font", "SF Mono", "Cascadia Code", monospace;
/* Scale (web-app oriented; 14px UI base) */
--text-2xs: 11px;
--text-xs: 12px;
--text-sm: 13px;
--text-base: 14px;
--text-md: 16px;
--text-lg: 18px;
--text-xl: 22px;
--text-2xl: 28px;
--text-3xl: 36px;
--text-4xl: 48px;
--text-5xl: 64px;
/* Weights */
--weight-regular: 400;
--weight-medium: 500;
--weight-semibold: 600;
--weight-bold: 700;
--weight-extrabold: 800;
/* Line heights */
--leading-tight: 1.15;
--leading-snug: 1.35;
--leading-normal: 1.55;
--leading-relaxed: 1.7;
/* Tracking */
--tracking-tight: -0.02em;
--tracking-normal: 0;
--tracking-wide: 0.06em;
--tracking-caps: 0.1em;
}
stacks/lobe-chat/compose.yaml
+1 -1
View File
@@ -120,7 +120,7 @@ services:
networks:
- tnet
labels:
- homepage.group=Apps
- homepage.group=AI - Gateways & Chat
- homepage.name=Lobe Chat
- homepage.icon=mdi-chat-processing
- homepage.description=Chat frontend over the LiteLLM gateway (eval)
stacks/memos/compose.yaml
+1 -1
View File
@@ -33,7 +33,7 @@ services:
networks:
- tnet
labels:
- homepage.group=Notes
- homepage.group=Daily
- homepage.name=Memos
- homepage.icon=mdi-note-text-outline
- homepage.description=Self-hosted note + memo server (ana-docker)
stacks/meromero-charrp/.env.example
+28 -1
View File
@@ -1,4 +1,31 @@
# ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host.
MEROMERO_GPU_MEM_UTIL=0.52
#
# ⚠ GPU0 IS SHARED WITH `vllm-gen` AND THE BUDGETS MUST SUM UNDER ~0.92.
# gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the
# 94.97 GiB card, because the utilization figure sizes the KV cache and does not
# cover CUDA context, graphs and non-torch overhead. This seat was at 0.51:
# 0.43 + 0.51 = 0.94, which left ~0.6 GiB of real headroom and worked right up
# until it did not. On 2026-08-24 it stopped fitting and the seat entered a
# crash-loop — 13 restarts, `torch.OutOfMemoryError: ... 195.19 MiB is free`,
# both the `char-rp` and `char-rp-reasoning` gateway aliases returning 500s
# (they share this one seat; see the LiteLLM routes for :8016).
#
# 0.47 restores a real margin (~4.8 GiB free on the card) and costs nothing you
# can use: KV cache goes 27.36 GiB -> 23.56 GiB, which is 430,825 -> 371,023
# tokens against a max-model-len of 262,144. The pool still holds 1.4x a
# full-length sequence; what you lose is concurrent long requests, not context.
#
# Raising this again means lowering gen's in the same change. Check the real
# numbers, not the flags: `nvidia-smi --query-compute-apps=pid,used_memory`.
#
# ⚠ AND CHECK THEM ON A FRESHLY-RESTARTED gen. Measured 2026-08-24: gen held
# 46,726 MiB (45.6 GiB) after ~3 days of uptime, and 39,424 MiB (38.5 GiB)
# immediately after a restart — the same container, the same
# `--gpu-memory-utilization 0.43`, ~7 GiB apart. Its footprint GROWS WITH
# UPTIME, which is the missing half of why this seat "fit on the 21st and
# stopped fitting on the 24th": nothing about this seat changed, gen crept up
# underneath it. Headroom arithmetic done against a long-running gen is
# measuring a moving number.
MEROMERO_GPU_MEM_UTIL=0.47
MEROMERO_MAX_MODEL_LEN=262144
MEROMERO_GPU_ID=0
stacks/meromero-charrp/compose.yaml
+7 -3
View File
@@ -86,10 +86,14 @@ services:
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=char-rp (MeroMero-v2 NVFP4, multimodal)
# STOPPED 2026-08-24 — superseded by stacks/gemma4-charrp (gemma-4-26B-A4B
# -it NVFP4 MoE) on the same port and the same served-model-names. Kept as
# the rollback seat: `docker compose up -d vllm-meromero-rp` here after
# stopping the gemma4 one, since both want :8016.
- homepage.group=AI - Dormant
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
- homepage.icon=mdi-drama-masks
- homepage.description=G4-MeroMero-v2-31B NVFP4A16 non-thinking prose seat, vision-enabled, 256K (ana-ml2 GPU0)
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
networks:
stacks/miniflux/compose.yaml
+1 -1
View File
@@ -44,7 +44,7 @@ services:
networks:
- tnet
labels:
- homepage.group=News
- homepage.group=Daily
- homepage.name=Miniflux
- homepage.icon=mdi-rss
- homepage.description=RSS reader (subreddits, blogs, HN)
stacks/open-webui/compose.yaml
+2 -2
View File
@@ -106,8 +106,8 @@ services:
networks:
- tnet
labels:
- homepage.group=Apps
- homepage.name=Open WebUI
- homepage.group=AI - Gateways & Chat
- homepage.name=Open WebUI (esh)
- homepage.icon=mdi-chat-question
- homepage.description=Env-declarative chat frontend over the LiteLLM gateway (eval vs Lobe)
- homepage.href=http://10.0.50.45:${OPENWEBUI_PORT:-3211}
stacks/rest-server-ana/compose.yaml
+1 -1
View File
@@ -52,7 +52,7 @@ services:
networks:
- tnet
labels:
- homepage.group=Service Networking
- homepage.group=Agents (no UI)
- homepage.name=Restic (rest-server)
- homepage.icon=mdi-cloud-upload
- homepage.description=Anaheim restic endpoint (data on NFS)
stacks/scriberr/.env.example
+44
View File
@@ -0,0 +1,44 @@
# Scriberr — copy to .env on the host at /opt/docker/compose/scriberr/.env
# Real .env is gitignored and lives only on ana-ml2.
# ── Image ────────────────────────────────────────────────────────────────
# Built locally from Dockerfile.cuda.12.9 — see the compose header for why
# the published scriberr-cuda image is NOT usable on these Blackwell cards.
SCRIBERR_IMAGE=scriberr:local-blackwell
# ── Network ──────────────────────────────────────────────────────────────
SCRIBERR_PORT=8080
SCRIBERR_BIND=0.0.0.0
# CORS. Must list every origin the UI is actually reached from, or the
# browser blocks the API calls. Comma-separated, no spaces, no trailing /.
SCRIBERR_ALLOWED_ORIGINS=http://10.250.50.54:8080,http://scriberr.ana.internal:8080
# ── GPU ──────────────────────────────────────────────────────────────────
# GPU0 is fully committed to the `gen` seat; GPU1 is the one with headroom.
SCRIBERR_GPU_ID=1
# ── Storage (on /tank — NOT the root pool, weights are multi-GB) ─────────
SCRIBERR_DATA_DIR=/tank/scriberr/data
SCRIBERR_ENV_DIR=/tank/scriberr/whisperx-env
# ── Runtime ──────────────────────────────────────────────────────────────
# ⚠ 10001, not the fleet-usual 1000. The Blackwell image's `appuser` IS 10001
# and its PUID remapping is broken — at 1000 the app cannot open its SQLite DB
# and crash-loops. The /tank dirs are chowned to 10001:10001 to match.
# See README "The PUID trap".
SCRIBERR_PUID=10001
SCRIBERR_PGID=10001
# Keep false while the app is served over plain HTTP. Setting this true
# without TLS makes login silently fail (cookie marked Secure, dropped).
SCRIBERR_SECURE_COOKIES=false
# ── Optional: summarisation / transcript chat ────────────────────────────
# Scriberr speaks the OpenAI API. Point it at the LiteLLM gateway so this
# costs nothing and stays on-prem, rather than a paid vendor key.
# Configure the base URL in the Scriberr UI (Settings -> AI provider):
# base URL : http://10.250.50.70:4000/v1
# model : summarizer (or gen / gen-reasoning)
# The key below is the shared all-agents gateway key.
# ⚠ That key also reaches PAID passthrough models (GLM, Kimi) on a shared
# tab — keep the configured model on a free local seat.
# SCRIBERR_OPENAI_API_KEY=
stacks/scriberr/README.md
+135
View File
@@ -0,0 +1,135 @@
# scriberr — self-hosted transcription + diarization (ana-ml2, GPU1)
Web UI for transcribing audio/video locally. WhisperX (Whisper + pyannote
speaker diarization) with NVIDIA Parakeet/Canary also selectable; SQLite for
state; optional summarisation and transcript chat against any OpenAI-compatible
endpoint.
- **Host:** `ana-ml2` (10.250.50.54) — GPU1
- **URL:** http://10.250.50.54:8080
- **Upstream:** https://github.com/rishikanthc/Scriberr
## The image is built locally, and that is not incidental
ana-ml2's RTX PRO 6000 Blackwell cards are **sm_120**. Upstream's published
images do not cover that:
| image | built for | usable here |
|---|---|---|
| `ghcr.io/rishikanthc/scriberr` | CPU | yes, but no GPU |
| `ghcr.io/rishikanthc/scriberr-cuda` | sm_61 … sm_89 (Pascal→Ada) | **no** — no sm_120 kernels |
| `ghcr.io/rishikanthc/scriberr-cuda-blackwell` | sm_120 | **does not exist** — documented in the upstream README but never published; GHCR returns no tags (checked 2026-08-23) |
The sm_120 path upstream actually ships is `Dockerfile.cuda.12.9`
(CUDA 12.9.1 + cuDNN, `PYTORCH_CUDA_VERSION=cu128`), built from source. So we
build it. **Do not "simplify" the compose back to the published `scriberr-cuda`
image** — it will fail on these cards or quietly fall back to CPU.
### Rebuilding
```bash
ssh ana-ml2
cd /tank/scriberr/src/Scriberr
git pull
docker build -f Dockerfile.cuda.12.9 -t scriberr:local-blackwell .
cd /opt/docker/compose/scriberr && docker compose up -d
```
Source checkout lives on `/tank`, not the root pool — see storage below.
## Deploy
```bash
# from this workstation
scripts/deploy-stack.sh ana-ml2 scriberr
```
Then on the host, the usual:
```bash
cd /opt/docker/compose/scriberr
docker compose config # dry parse first
docker compose up -d scriberr # target the service, not the whole stack
```
## Storage — deliberately on /tank
`/var/lib/docker` on ana-ml2 sits on `zroot` at ~87% used. Whisper, pyannote
and NeMo weights are multi-GB and land in the `whisperx-env` volume, so both
mounts are bind-mounted onto `/tank` (4+ TB) instead of named volumes:
| host path | container path | holds |
|---|---|---|
| `/tank/scriberr/data` | `/app/data` | SQLite DB, uploads, transcripts |
| `/tank/scriberr/whisperx-env` | `/app/whisperx-env` | Python env + model weights |
| `/tank/scriberr/src/Scriberr` | — | build checkout |
Both are owned by uid/gid 1000 to match `PUID`/`PGID`.
## First run takes a while
On first start the container builds a Python environment and downloads several
GB of model weights before the port answers — upstream says "several minutes".
The healthcheck therefore has a **600 s `start_period`**; the container will
show `starting`, not `unhealthy`, during that window. Watch it with:
```bash
docker logs -f scriberr
```
Subsequent starts are fast because the env volume persists.
## The PUID trap — read this before "fixing" the uid
This stack runs as **uid/gid 10001**, not the fleet-usual 1000, and the
`/tank/scriberr` dirs are chowned to match. That is deliberate.
`Dockerfile.cuda.12.9` creates `appuser` at **uid 10001** — Ubuntu 24.04's base
image already owns uid 1000 as `ubuntu`, so upstream moved their app user out of
the way. It then `chown`s `/app` to 10001. But the entrypoint's `PUID` remapping
only chowns `/app/data` and `/app/whisperx-env` — **not `/app` itself**. So
running with `PUID=1000` leaves the app unable to open its SQLite database and
it crash-loops with:
```
Failed to connect to database: unable to open database file: out of memory (14)
```
That message is a red herring twice over: error 14 is `SQLITE_CANTOPEN`, not an
OOM, and the machine has 566 GB of RAM. Diagnosis notes from 2026-08-23:
- SQLite itself writes fine to `/tank` as uid 1000 — the mount is not at fault.
- The app fails on a plain Docker **named volume** too — storage is not at fault.
- The **published CPU image runs fine at `PUID=1000`**, because in `Dockerfile`
(the non-CUDA one) `appuser` *is* uid 1000. Only the CUDA 12.9 variant moved it.
- Same image at `PUID=10001` starts clean. That is the whole difference.
If you ever want host files owned by 1000 instead, the fix is to patch
`Dockerfile.cuda.12.9` to `userdel ubuntu` and recreate `appuser` at 1000, then
rebuild — a local patch to carry, which is why it was not done.
## Gotchas
- **`SECURE_COOKIES` must stay `false` while served over plain HTTP.** At the
production default of `true` the session cookie is marked `Secure`, the
browser drops it, and login appears to succeed then bounces you straight back
to the login page with nothing useful in the logs.
- **`ALLOWED_ORIGINS` must list the real origin.** Upstream defaults to
`localhost` only; reaching the UI by host IP fails CORS until it is set.
- **Never add `NVIDIA_VISIBLE_DEVICES=all`.** Upstream's compose sets it, but
here it would override the `device_ids` reservation and expose both cards —
GPU0 belongs to the `gen` seat.
- **This stack is a guest on GPU1**, which it shares with the `sec` seat. If
VRAM gets tight, this is the thing that should yield.
## Optional: summarisation via the LiteLLM gateway
Scriberr speaks the OpenAI API, so point it at the fleet gateway instead of a
paid vendor. In the UI under the AI provider settings:
- base URL: `http://10.250.50.70:4000/v1`
- model: `summarizer` (or `gen` / `gen-reasoning`)
- key: the shared all-agents gateway key
⚠ That key also reaches **paid** passthrough models (GLM, Kimi) on a shared
tab. Keep the configured model on a free local seat.
stacks/scriberr/compose.yaml
+111
View File
@@ -0,0 +1,111 @@
# Scriberr — self-hosted audio/video transcription with speaker diarization.
# Upstream: https://github.com/rishikanthc/Scriberr (Go + SvelteKit, SQLite).
#
# Transcription runs locally via WhisperX (Whisper + pyannote diarization);
# NVIDIA Parakeet / Canary models are also selectable in the UI. Optional
# summarisation / transcript chat talks to any OpenAI-compatible endpoint —
# point it at the LiteLLM gateway rather than a paid API (see README).
#
# ── IMAGE: BUILT LOCALLY, ON PURPOSE ──────────────────────────────────────
# ana-ml2's RTX PRO 6000 Blackwell cards are **sm_120**. Upstream publishes
# `scriberr-cuda` (built for sm_61…sm_89 — no sm_120 kernels) and documents a
# `scriberr-cuda-blackwell` image that **has never actually been published**
# (GHCR returns no tags for it, checked 2026-08-23). The sm_120 path upstream
# ships is `Dockerfile.cuda.12.9` (CUDA 12.9.1 + cu128 torch), built locally.
# Do NOT "simplify" this to the published `scriberr-cuda` image — it will
# fail on these cards or silently fall back to CPU.
# Rebuild: see README "Rebuilding" — checkout lives at
# /tank/scriberr/src/Scriberr on ana-ml2.
#
# ── GPU PINNING ───────────────────────────────────────────────────────────
# Pinned to **GPU1** via explicit device_ids, per the house convention and
# because GPU0 is fully committed to the `gen` seat. GPU1 shares space with
# the `sec` seat, so this stack is a guest there — keep an eye on VRAM.
# NOTE: do NOT add `NVIDIA_VISIBLE_DEVICES=all` (as upstream's compose does).
# It overrides the device_ids reservation and exposes both cards.
#
# All tunables live in .env — edit that, not this file.
services:
scriberr:
image: ${SCRIBERR_IMAGE:-scriberr:local-blackwell}
container_name: scriberr
restart: unless-stopped
ports:
- "${SCRIBERR_BIND:-0.0.0.0}:${SCRIBERR_PORT}:8080"
volumes:
# Bind mounts rather than named volumes: /var/lib/docker on ana-ml2
# lives on zroot with limited headroom, while /tank has terabytes.
# Model weights (Whisper, pyannote, NeMo) land in whisperx-env and are
# multi-GB — they must not go anywhere near the root pool.
- ${SCRIBERR_DATA_DIR}:/app/data
- ${SCRIBERR_ENV_DIR}:/app/whisperx-env
environment:
# ⚠ 10001, NOT the fleet-usual 1000 — this is load-bearing.
# Dockerfile.cuda.12.9 creates `appuser` at uid 10001 (Ubuntu 24.04's
# base image already owns uid 1000 as `ubuntu`, so upstream moved it) and
# chowns /app to 10001. The entrypoint's PUID remapping only chowns
# /app/data + /app/whisperx-env, not /app itself, so running as 1000
# leaves the app unable to open its SQLite DB and it crash-loops with
# `unable to open database file: out of memory (14)` — which is
# SQLITE_CANTOPEN wearing a misleading message, not a real OOM.
# The host bind-mount dirs are therefore chowned to 10001:10001 too.
# Verified 2026-08-23: PUID=1000 crash-loops, PUID=10001 starts clean.
- PUID=${SCRIBERR_PUID:-10001}
- PGID=${SCRIBERR_PGID:-10001}
- APP_ENV=production
# Served over plain HTTP on the LAN. Left at the production default of
# `true`, the session cookie is marked Secure and the browser silently
# drops it — you log in, get bounced back to the login page, and the
# logs show nothing wrong. This must stay false while access is HTTP.
- SECURE_COOKIES=${SCRIBERR_SECURE_COOKIES:-false}
# Upstream defaults to localhost origins only, which fails CORS when
# reached by host IP. Keep this in sync with how the app is reached.
- ALLOWED_ORIGINS=${SCRIBERR_ALLOWED_ORIGINS}
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
# Scriberr builds each model backend's Python env with `uv` at runtime.
# uv's default link mode reflink/hardlinks out of its cache, which fails
# on this overlayfs+ZFS combination with a misleading
# "Failed to clone ... Resource temporarily unavailable (os error 11)"
# and takes out the Parakeet + Sortformer backends (WhisperX survives).
# `copy` trades a little disk and time for it actually working.
- UV_LINK_MODE=${SCRIBERR_UV_LINK_MODE:-copy}
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["${SCRIBERR_GPU_ID:-1}"]
capabilities: [gpu]
healthcheck:
# 127.0.0.1 rather than localhost — the IPv6-first resolution trap has
# bitten news-digest and chatterbox in this fleet before.
# start_period is generous: first boot builds a Python env and pulls
# several GB of model weights before the port answers.
test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8080/ >/dev/null || exit 1"]
interval: 30s
timeout: 5s
retries: 3
start_period: 600s
networks:
- tnet
labels:
# ⚠ The group name MUST match a key in the dashboard's settings.yaml
# `layout:` block. A group that appears nowhere in that block gets no
# `tab:` assignment, and Homepage renders an untabbed group on EVERY tab.
# This label read `AI Systems` — a group that existed nowhere — from
# 2026-08-23 until it was caught on 2026-08-24.
# `AI - Studios` and not one of the ASR groups because Scriberr is a
# transcription UI you open and work in, which is what Studios collects;
# the bare ASR endpoints (Parakeet, Speaches) live in the collapsed
# `AI - Audio Tools` group instead.
- homepage.group=AI - Studios
- homepage.name=Scriberr
- homepage.icon=mdi-microphone-message
- homepage.description=Audio/video transcription + diarization (ana-ml2, GPU1)
- homepage.href=http://10.250.50.54:${SCRIBERR_PORT}
networks:
tnet:
name: traefik-net
external: true
stacks/searxng/compose.yaml
+1 -1
View File
@@ -87,7 +87,7 @@ services:
- traefik.http.routers.searxng.tls=true
- traefik.http.routers.searxng.service=searxng
- traefik.http.services.searxng.loadbalancer.server.port=8080
- homepage.group=Apps
- homepage.group=Daily
- homepage.name=SearXNG
- homepage.icon=si-searxng
- homepage.description=Privacy-respecting meta-search
stacks/waterland-studio/compose.yaml
+1 -1
View File
@@ -82,7 +82,7 @@ services:
networks:
- tnet
labels:
- homepage.group=AI - Image & Media
- homepage.group=AI - Studios
- homepage.name=Waterland Studio
- homepage.icon=mdi-watercolor
- homepage.description=Watercolour plate + reveal animation renderer (irv-ml1, A6000)
stacks/zed-fim-proxy/compose.yaml
+1 -1
View File
@@ -29,7 +29,7 @@ services:
retries: 3
start_period: 10s
labels:
- homepage.group=AI - Gateways & Chat
- homepage.group=AI - Inference
- homepage.name=Zed FIM Proxy
- homepage.icon=mdi-code-braces-box
- homepage.description=Keyless /v1/completions for Zed edit-predictions (coder-fast, ana-docker)