memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure

2026-09-05 06:36:56 -07:00
parent 5237efa299
commit 82158e7e36
13 changed files with 958 additions and 651 deletions
@@ -3410,3 +3410,634 @@ strictly better than softdog; treat a real firing as unconfirmed until tested.
 - **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
  that are demonstrably online). `stat/sta` is the live view.
 _Archived 2026-09-03._
 - `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`)
 _Archived 2026-09-05._
 # `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
 The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
 using the MTP-aware, vision-preserving **Robinson formula** (documented in
 `docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
 Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
 ## Why this model, why abliterate it ourselves
 Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
 llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
 **~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
 4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
 over-refusal 0. So there is a real creative-content refusal surface to remove.
 This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
 we abliterate it ourselves.
 **It is additive over the current gen seat.** The live Heretic seat
 (`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
 the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
 not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
 residual-write matrices), and the MTP head is what gates speculative acceptance.
 That in-band MTP edit is the delta this experiment tests.
 ## Recipe maps 1:1 — dry-run PASSED
 Against the staged bf16: 1199 tensors, 333 vision preserved,
 `down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
 **131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
 name drift. Two hard gates in the harness halt before any write: the coverage
 identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
 **dim 3994** (orthogonalizing a direction living there bricks the model).
 ## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
 **The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
 The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
 AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
 Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
 produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
 finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
 range, so this is **precision-driven catastrophic cancellation, not overflow** —
 **fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
 prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
 input path through the unstable recurrence.
 Fixes, all in the committed harness (`7abd301`):
 - **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
 - **A finite-gate aborts on a non-finite direction** — the sink screen alone
  can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
  saved silently on the first run).
 - `attn_implementation="sdpa"` pinned.
 **fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
 forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
 `vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
 ⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
 grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
 meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
 into the remainder. All three restored to healthy.
 ⚠ **fla lives in a side dir, not the venv.** The shared
 `/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
 `--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
 reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
 ## Result
 Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
 in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
 Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
 ⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
 Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
 0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
 (HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
 is valid and sink-clean but noisier than ideal; abliterating on it risks
 under-removing refusals or nicking capability. **Expand the sets to a few
 hundred each and re-capture** before the `--out` write.
 ## Sequence from here
 1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
 2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
 3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
   (the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
   (gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
 4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
   **Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
   holds through real multi-turn use.
 bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
 provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
 re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
 `7abd301`.
 _Archived 2026-09-05._
 # `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
 Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
 keep track of local .internal dns names that can be agent managed and is
 lightweight."* Built and live in one session; commit `b8003c7`.
 ## Shape
 ```
 dns/internal.yaml     source of truth — 38 hosts + 4 service aliases
 scripts/dns-sync.py   reconciles AdGuard resolvers against it
 stacks/adguard-ana/   the colo's resolver, which did not exist
 dns/README.md         workflow, naming, the IPv6 caveat
 ```
 Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
 resolvers are derived state, you see a diff before anything changes.
 `--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
 `nothing to do`.
 Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
 (operator's call). `.internal` is ICANN-reserved for private use since 2024;
 `.local` is reserved for mDNS, which is why the pre-existing
 `searxng.pfi.local` was a standards collision that merely happened to work.
 Every name is published to **every** resolver — the site label says where a
 host *is*, not which resolver knows about it.
 ## The framing correction that mattered most
 The ask reads as "I can't memorise v6 addresses", but the deeper problem is
 that **v6 addresses are derived, not assigned**, so they cannot reliably be
 *written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
 privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
 DHCP reservation** — so a hand-maintained v6 table rots on its own.
 ⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
 server-class hosts, (2) then the name table is just a file. Surfaced to the
 operator before building.
 **Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
 ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
 `ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
 EMPTY and correct, and the naming layer was built first rather than blocking
 on v6. Names established now need no renaming when addresses land.
 Suggested convention when they do (awaiting operator): each server static at
 its site's `/64` with low-order bits echoing the v4 host octet —
 `esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
 ## Two properties not to break
 **Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
 `.internal` are managed. ESH's resolver turned out to carry three hand-made
 `esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
 read of the config missed them**, because an `awk` range on `rewrites:` matched
 an empty-looking block. A resolver-wide authoritative sync would have silently
 deleted all three on first run. Verified intact after sync.
 **Within `.internal` it IS authoritative** — names added by hand in the AdGuard
 UI get deleted by the next sync. That is the point: one place to look.
 ## The colo had no resolver at all
 ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
 `1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
 `stacks/adguard-ana/`.
 ⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
 that busy host. The port is therefore carried **per-site in the yaml**, not
 assumed by the script, so the odd one out cannot be forgotten.
 ⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
 human browsing; this one resolves for a rack of servers, where a blocklist
 false-positive breaks service-to-service calls at 3am for no upside.
 First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
 the conf volume before first start, so the container comes up configured
 instead of sitting in the setup wizard.
 ## Credential — service account, not the operator's
 Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
 than asking for the `lkraven` password (per the standing migrate-off-operator-
 creds directive). Password vaulted at
 `nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
 backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
 resolvers kept answering across the restart.
 Two landmines worth keeping:
 - **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
  marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
  the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
  on a live resolver.
 - **The vault appends a trailing newline on `get`.** A password carrying a
  stray `\n` fails auth in a way that looks exactly like a wrong password.
  `dns-sync.py` strips it.
 ## `pfi.local` migration — and the one that must NOT move
 `searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
 alongside** in the Traefik rule so nothing breaks mid-migration; both return
 200. Drop the fallback once the access log shows the old name unused.
 **`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
 baked into every user ID, room ID and signing key, and federation identity
 derives from it — renaming it is not a DNS change, it is rebuilding the
 homeserver's identity and invalidating its history. The operator approved
 "migrate pfi.local" generally; this was surfaced as a deliberate exclusion
 rather than executed blindly.
 ## Still open
 Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
 — it only answers what asks it directly. Repointing a whole site's DNS is a
 bigger change than standing the service up, and is the operator's to schedule.
 See also [[2026-08-17-fleet-ipv6-mesh]].
 _Archived 2026-09-05._
 # `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
 Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
 a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
 `nohup` (PID 1283383) that would not survive a reboot. Now
 `stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
 irv-ml1:8410. Commits `a2b5b58`, `8189076`.
 Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
 pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
 **Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
 a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
 own guards already neutralised both landmines and the project was in
 wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
 with a real reason behind it, and one `update.sh` run carried both. Verified
 end to end after the update: healthy, `backend: cupy`, and a real 256² plate
 render completes warm — the kernel-cache volume survived the image swap.
 ## Build context lives OUTSIDE the compose dir — on purpose
 `/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
 out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
 rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
 `compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
 refreshes source → rebuild → recreate → health, and is verified end to end.
 ## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
 `gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
 silently drops to the numpy path at ~21x wall time — it does not error, it
 just gets slow. waterland-dev warned about the build side.
 The runtime side is worse and was not in the handover: **`studio/jobs.py`
 shells the renderer out as a literal `uv run waterland ...` with no `--extra`
 flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
 mid-job to its default extras and prunes cupy back out from under a correctly
 built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
 pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
 slower environment.
 **Fixed upstream in `464dfc2`:** the server now spawns
 `sys.executable -m waterland.cli` directly — no resolver in the render path at
 all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
 if any future code path re-enters `uv` inside the container, the job fails
 loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
 in the image regardless — it performs the build-time `uv sync` /
 `uv pip install`, and this is a single-stage build.
 ## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
 Every render died 1.7s in with:
 ```
 RuntimeError: Failed to find CUDA headers.
 ```
 printed **through argparse's usage banner**, which makes it read like a CLI
 argument bug rather than a missing toolkit. That misdirection is the reason
 this is written down.
 cupy compiles kernels at runtime through NVRTC, which needs the toolkit
 **headers** — not just the driver and the runtime libs bundled in the
 `cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
 bare `nohup` process found them **by accident**; a slim image has none.
 Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
 hundred MB against ~6 GB for a `-devel` base image. It runs **after**
 `uv sync`, because sync prunes what it does not know about.
 Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
 and anyone running this without a system toolkit hits it. **Declared upstream
 in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
 stays in the Dockerfile**: the header requirement is a property of *this*
 image — a slim base with no system CUDA toolkit — so it belongs in the file
 that creates the problem, not inherited from an extra two repos away. It also
 survives any future restructuring of the `gpu` extra. Cost of keeping it is
 now measured, not assumed: since `uv sync` satisfies it first, the line
 reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
 that documents a non-obvious requirement is worth 0.3s. (waterland-dev
 independently agreed they would keep it too.)
 ## Landmine 3 — the GPU index inside the container is not the host's
 The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
 `CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
 `nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
 (`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
 container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
 the host's value selects a device that does not exist. Host device 0 is the
 3090, which carries the TTS zoo and must not be touched.
 ## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
 | job | wall |
 |---|---|
 | 256² + anim, cold container | 23.3 s |
 | 256² + anim, warm | **6.1 s** |
 | 256² plate only (`--codec none`) | 3.9 s |
 | 512² plate only | 6.4 s |
 Warm beats the **7.4 s** recorded against the bare-metal process, so
 containerising cost nothing. Verified the cache volume properly: recreate
 (fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
 it every restart makes the next user wait 4x and the service merely *looks*
 slow.
 ## Upstream finding — the on-disk job store grows without bound
 `JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
 startup**. Consequences:
 1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
   how this was spotted: the API reported **1 job** while the volume held all
   **16 directories, 60.6 MB**. Not data loss.
 2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
   so directories orphaned by a restart are **never reclaimed**. The
   handover's "bounded around 500 MB" holds within a single process lifetime;
   across restarts the store grows monotonically at ~12 MB per animated job.
 Reported to waterland-dev with evidence; **not patched from the infra side** —
 it is their code. Prune the volume by hand if it bites first.
 **waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
 claim holds within one process lifetime and nowhere else, which on a
 `restart: unless-stopped` service is the wrong lifetime to have bounded. They
 have **surfaced a startup-rehydrate fix to the operator** rather than opening a
 third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
 and is DEPLOYED (2026-08-19).**
 Startup rehydrate, as recommended — and waterland-dev deliberately went
 further than the framing I sent them. I had said a directory the scan cannot
 parse "just does not enter the index"; they made the opposite call, because a
 directory that never enters the index is exactly the one that never gets
 reclaimed. **That is the sharper reading and it is the reason the fix works on
 this volume at all** — the 16 pre-existing dirs have no sidecar. Their
 adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
 dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
 sidecar → degrades to inference, no startup crash; **neither source nor
 sidecar → skipped on purpose**, since adopting it would turn eviction into a
 delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
 go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
 unreachable via the artifact route.
 They also closed a second leak I never saw, because it needs a restart
 *mid-render* to surface: a job left `running`/`queued` in its sidecar is
 non-terminal forever, and eviction skips non-terminal jobs — so it is a
 phantom that is never reclaimed and `queue_depth` over-reports for the life of
 the process. Adoption now marks those `failed`.
 **Verified on this host after the update:** `/api/jobs` went **1 → 16** while
 the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
 Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
 made them visible. A subsequent real render took both to 17. From here the
 store is bounded **across** restarts, not merely within a process.
 ## Access
 Repo is not anonymously readable (a bare clone 403s). Operator granted
 **`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
 pull: True`. Token on irv-ml1 at
 `/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
 **repo-scoped** credential helper; `.git/config` carries no token (verified),
 so the remote stays clean in any diff or backup. The operator's `vh`
 site-admin token was used only for the initial clone and the grant itself and
 was **never written to disk on that host** — a site-admin credential on a GPU
 box is a blast radius nobody needs for a read-only fetch.
 ## Constraints honoured as stated (not inferred)
 - **Serial by design — one replica, one card.** A render is 20–45s of near-full
  GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
  Throughput is a hardware conversation, not a replica-count one.
 - **No authentication, arbitrary file uploads** ⇒ stays inside the
  LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
  waterland-dev offered to add a real auth layer if wider reach is ever needed.
 _Archived 2026-09-05._
 # `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
 Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
 ## The cleanup (three real defects)
 - **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
  `settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
  is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
  ⚠️ This will happen again to the next group added without a `tab:` — the rule
  is now written at the top of the layout block.
 - **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
  Monitoring *and* `homepage.group=Apps` on the container. Exactly the
  "never list a labelled container manually" failure the stack README warns
  about; it survived the previous day's audit because a duplicate reads as two
  plausible cards rather than as an error. Manual block deleted, label moved to
  `Monitoring`, `homepage.siteMonitor` added.
 - **Column counts were fiction** — several groups declared more columns than
  they had members, so the last row of each was dead space (Notes: 1 card in a
  4-wide row). Columns now track member counts; `GET /api/services` prints the
  live per-group counts and is the check.
 Later, on operator instruction, the **AI tab was reordered by clickability**:
 Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
 seats and TTS endpoints. Reasoning written into the config so it survives:
 order by "would I click this?", not by how central the service is.
 ## ⚠️ The expensive red herring — the tab bar after a recreate
 After a recreate the client render comes up with **no tab bar, no wallpaper and
 no i18n** (search box shows the raw key `search.search`), groups falling back to
 side-by-side columns. **It restores itself with no intervention.**
 Timing, measured rather than assumed: a fresh container was still tab-less at
 **4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
 reporting `healthy` says nothing about it — the container is serving, the page
 is just wrong.
 An hour went into ruling out four causes that were never the cause:
 1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
   their committed versions reproduces it, as does the pre-adoption backup in
   `/opt/docker-bu/conf/homepage/`.
 2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
   identical symptoms, and the image never changed anyway (working and broken
   both report `v2.0.0` / rev `17456f2`).
 3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
   socket unmounted entirely.
 4. **Not server-side** — the server-rendered HTML still contains the tab
   markup, the background URL and `useEqualHeights`; `GET /api/validate`
   returns `[]`. The loss is client-side, with no page error, no failed chunk
   and no non-200.
 Every throwaway container in that list was judged within ~30s of starting, so
 they were all inside the same window — and that consistency **read as a
 reproduction when it was the same measurement mistake five times over.**
 **Operative rule: recreate, walk away, re-check later. Do not chase it.**
 ## ⚠️ The iteration loop that would have prevented the overcook
 `custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
 change needs a **browser reload** — not a container recreate, and it never owed
 the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
 per attempt (operator called this out directly).
 Faster still, and how the final pass was done: **inject candidate CSS into the
 running page and screenshot it** —
 `await p.addStyleTag({content: css})` in Playwright against the live
 dashboard. Seconds per iteration, no deploy. Build + deploy only once the
 render looks right.
 ## The theme — Australis Skyfall
 Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
 Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
 family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
 Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
 "calm depth" language of **hairline + two-layer shadow on every elevated
 surface, never one without the other**.
 ```
 theme/colors.css layout.css typography.css   vendored VERBATIM from the bundle
 theme/fonts/Supreme-{400,500,700}.woff2      the body/UI face
 theme/skyfall.css.in                         the Homepage bindings (ours)
 theme/build.py            → conf/custom.css  (generated — do not hand-edit)
 ```
 The build step exists for one reason: **Homepage serves only `custom.css` and
 `custom.js` out of its config dir**, with no static route beside them, so a
 `@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
 as a data URI. The background image takes the other road, because
 `/app/public/images` **is** a real static route (mounted read-only in
 `compose.yaml`).
 Only Supreme is embedded: a link dashboard has no display type, and Victor
 Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
 handful of latency figures.
 ## The background is generated, not stock
 **Arbo as an image-gen engine** (the operator's actual ask, which I first
 misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
 workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
 Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
 WebP** (smooth gradients compress absurdly well).
 ⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
 bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
 ## Two documented deviations from the design system
 1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
   photography, no textures", with one permitted motif — a subtle aurora
   gradient on hero/empty-state areas only, *"never behind body text blocks"*.
   A dashboard is a body-text block. Present on the operator's explicit
   instruction, mitigated rather than excused: abstract, no subject, strictly
   cool temperature, held at **`opacity: 30`**. That number is load-bearing —
   at 14 the aurora was invisible, and turning it up makes the cards fight the
   ribbon.
 2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
   only makes them illegible.
 ## Overcorrection, and the colour pass
 First stat-well pass went from `font-thin` 13px straight to **bold 22px in
 heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
 The principle missed: a stat only has to out-rank **its own label**, not the
 service name above it. Now `--text-md` medium in cyan.
 Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
 cyan, green) the *primary* families, "used generously, in that order", while
 Dawn (amber/red/violet) is semantic-only. So group markers cycle
 blue→cyan→green down the page (icons full strength, names at 0.72), service
 icons take a single cool wash, latency tags move to the info family so
 "how fast" stops looking like "is it alive". **No Dawn colour is used
 decoratively anywhere.**
 Two DOM findings that made it possible:
 - **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
  via `background`, not `color`.
 - **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
  original selectors matched nothing, so every green pill up to that point was
  stock colouring rather than the theme. Both forms are now matched.
 _Archived 2026-09-05._
 # `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
 Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
 ## The pattern worth remembering
 Chasing two bad-looking cards on the dashboard turned up **four stacks running
 on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
 known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
 ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
 ⇒ **A dashboard card is a cheap census of what is actually running.** When
 something on it looks wrong, check whether the stack behind it is even in
 `stacks/` before debugging the symptom — twice here the answer was "no", and
 the fix belonged in version control as much as on the host.
 Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
 `stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
 **deliberately left unmanaged** — adopting three live resolvers while also
 introducing a new DNS naming system is two risky changes at once.
 ## SearXNG — the healthcheck was eating itself
 Card flapped UNHEALTHY; the container was fine the whole time. The compose
 passed `--tries` and `--spider` as **two separate argv entries**, so wget
 consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
 which means every probe since April **downloaded** the healthz response to a
 file:
 ```
 295,287 healthz.N files in the container's working directory
 ```
 With that many files, wget's scan for the next free filename is what
 intermittently blew the 10s timeout. **Self-worsening — every probe made the
 next one slower.** Restored `--tries=1`; the junk lived in the writable layer
 so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
 Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
 silently swallows the next argument turns a liveness probe into a workload.
 ## SeaFile — not broken, never restarted
 Card showed EXITED for three months. **None of the three services declared a
 restart policy**, so Docker defaulted them to `no`. On
 **2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
 other — a daemon restart or host reboot — and nothing brought them back.
 ⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
 SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
 it as one sends you hunting a bug that does not exist. The tell was all three
 services stopping within 200ms.
 Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
 healthcheck exactly as the existing `depends_on` comments intended, seahub
 started without the race, `302` → login page. Data was in local named volumes,
 not on the ana-nas NFS, so nothing was at risk.
 Three months of silent downtime whose only signal was a card nobody read as an
 outage — the argument for semantic status colour on the dashboard (see
 [[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
 mis-grouped AI seats obvious at a glance).
 ## heretic2-charrp-reasoning — tracked, with its shim
 The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
 BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
 untracked. Now in `stacks/`, including
 `conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
 does not propagate modelopt `exclude_modules` to the spec-decode **draft**
 model, so the BF16 MTP head gets quantized and the engine dies at load. Both
 the mount and `PYTHONPATH` are load-bearing.
 Added the two files house convention expects and the directory lacked — a
 `.env.example` naming every knob (all values are compose defaults; the host
 overrides only the three VRAM ones) and a README pointing at
 `docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
 _Archived 2026-09-05._
@@ -1,119 +0,0 @@
 # `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
 Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
 keep track of local .internal dns names that can be agent managed and is
 lightweight."* Built and live in one session; commit `b8003c7`.
 ## Shape
 ```
 dns/internal.yaml     source of truth — 38 hosts + 4 service aliases
 scripts/dns-sync.py   reconciles AdGuard resolvers against it
 stacks/adguard-ana/   the colo's resolver, which did not exist
 dns/README.md         workflow, naming, the IPv6 caveat
 ```
 Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
 resolvers are derived state, you see a diff before anything changes.
 `--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
 `nothing to do`.
 Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
 (operator's call). `.internal` is ICANN-reserved for private use since 2024;
 `.local` is reserved for mDNS, which is why the pre-existing
 `searxng.pfi.local` was a standards collision that merely happened to work.
 Every name is published to **every** resolver — the site label says where a
 host *is*, not which resolver knows about it.
 ## The framing correction that mattered most
 The ask reads as "I can't memorise v6 addresses", but the deeper problem is
 that **v6 addresses are derived, not assigned**, so they cannot reliably be
 *written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
 privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
 DHCP reservation** — so a hand-maintained v6 table rots on its own.
 ⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
 server-class hosts, (2) then the name table is just a file. Surfaced to the
 operator before building.
 **Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
 ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
 `ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
 EMPTY and correct, and the naming layer was built first rather than blocking
 on v6. Names established now need no renaming when addresses land.
 Suggested convention when they do (awaiting operator): each server static at
 its site's `/64` with low-order bits echoing the v4 host octet —
 `esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
 ## Two properties not to break
 **Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
 `.internal` are managed. ESH's resolver turned out to carry three hand-made
 `esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
 read of the config missed them**, because an `awk` range on `rewrites:` matched
 an empty-looking block. A resolver-wide authoritative sync would have silently
 deleted all three on first run. Verified intact after sync.
 **Within `.internal` it IS authoritative** — names added by hand in the AdGuard
 UI get deleted by the next sync. That is the point: one place to look.
 ## The colo had no resolver at all
 ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
 `1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
 `stacks/adguard-ana/`.
 ⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
 that busy host. The port is therefore carried **per-site in the yaml**, not
 assumed by the script, so the odd one out cannot be forgotten.
 ⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
 human browsing; this one resolves for a rack of servers, where a blocklist
 false-positive breaks service-to-service calls at 3am for no upside.
 First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
 the conf volume before first start, so the container comes up configured
 instead of sitting in the setup wizard.
 ## Credential — service account, not the operator's
 Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
 than asking for the `lkraven` password (per the standing migrate-off-operator-
 creds directive). Password vaulted at
 `nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
 backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
 resolvers kept answering across the restart.
 Two landmines worth keeping:
 - **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
  marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
  the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
  on a live resolver.
 - **The vault appends a trailing newline on `get`.** A password carrying a
  stray `\n` fails auth in a way that looks exactly like a wrong password.
  `dns-sync.py` strips it.
 ## `pfi.local` migration — and the one that must NOT move
 `searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
 alongside** in the Traefik rule so nothing breaks mid-migration; both return
 200. Drop the fallback once the access log shows the old name unused.
 **`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
 baked into every user ID, room ID and signing key, and federation identity
 derives from it — renaming it is not a DNS change, it is rebuilding the
 homeserver's identity and invalidating its history. The operator approved
 "migrate pfi.local" generally; this was surfaced as a deliberate exclusion
 rather than executed blindly.
 ## Still open
 Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
 — it only answers what asks it directly. Repointing a whole site's DNS is a
 bigger change than standing the service up, and is the operator's to schedule.
 See also [[2026-08-17-fleet-ipv6-mesh]].
@@ -1,145 +0,0 @@
 # `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
 Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
 ## The cleanup (three real defects)
 - **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
  `settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
  is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
  ⚠️ This will happen again to the next group added without a `tab:` — the rule
  is now written at the top of the layout block.
 - **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
  Monitoring *and* `homepage.group=Apps` on the container. Exactly the
  "never list a labelled container manually" failure the stack README warns
  about; it survived the previous day's audit because a duplicate reads as two
  plausible cards rather than as an error. Manual block deleted, label moved to
  `Monitoring`, `homepage.siteMonitor` added.
 - **Column counts were fiction** — several groups declared more columns than
  they had members, so the last row of each was dead space (Notes: 1 card in a
  4-wide row). Columns now track member counts; `GET /api/services` prints the
  live per-group counts and is the check.
 Later, on operator instruction, the **AI tab was reordered by clickability**:
 Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
 seats and TTS endpoints. Reasoning written into the config so it survives:
 order by "would I click this?", not by how central the service is.
 ## ⚠️ The expensive red herring — the tab bar after a recreate
 After a recreate the client render comes up with **no tab bar, no wallpaper and
 no i18n** (search box shows the raw key `search.search`), groups falling back to
 side-by-side columns. **It restores itself with no intervention.**
 Timing, measured rather than assumed: a fresh container was still tab-less at
 **4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
 reporting `healthy` says nothing about it — the container is serving, the page
 is just wrong.
 An hour went into ruling out four causes that were never the cause:
 1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
   their committed versions reproduces it, as does the pre-adoption backup in
   `/opt/docker-bu/conf/homepage/`.
 2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
   identical symptoms, and the image never changed anyway (working and broken
   both report `v2.0.0` / rev `17456f2`).
 3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
   socket unmounted entirely.
 4. **Not server-side** — the server-rendered HTML still contains the tab
   markup, the background URL and `useEqualHeights`; `GET /api/validate`
   returns `[]`. The loss is client-side, with no page error, no failed chunk
   and no non-200.
 Every throwaway container in that list was judged within ~30s of starting, so
 they were all inside the same window — and that consistency **read as a
 reproduction when it was the same measurement mistake five times over.**
 **Operative rule: recreate, walk away, re-check later. Do not chase it.**
 ## ⚠️ The iteration loop that would have prevented the overcook
 `custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
 change needs a **browser reload** — not a container recreate, and it never owed
 the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
 per attempt (operator called this out directly).
 Faster still, and how the final pass was done: **inject candidate CSS into the
 running page and screenshot it** —
 `await p.addStyleTag({content: css})` in Playwright against the live
 dashboard. Seconds per iteration, no deploy. Build + deploy only once the
 render looks right.
 ## The theme — Australis Skyfall
 Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
 Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
 family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
 Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
 "calm depth" language of **hairline + two-layer shadow on every elevated
 surface, never one without the other**.
 ```
 theme/colors.css layout.css typography.css   vendored VERBATIM from the bundle
 theme/fonts/Supreme-{400,500,700}.woff2      the body/UI face
 theme/skyfall.css.in                         the Homepage bindings (ours)
 theme/build.py            → conf/custom.css  (generated — do not hand-edit)
 ```
 The build step exists for one reason: **Homepage serves only `custom.css` and
 `custom.js` out of its config dir**, with no static route beside them, so a
 `@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
 as a data URI. The background image takes the other road, because
 `/app/public/images` **is** a real static route (mounted read-only in
 `compose.yaml`).
 Only Supreme is embedded: a link dashboard has no display type, and Victor
 Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
 handful of latency figures.
 ## The background is generated, not stock
 **Arbo as an image-gen engine** (the operator's actual ask, which I first
 misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
 workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
 Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
 WebP** (smooth gradients compress absurdly well).
 ⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
 bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
 ## Two documented deviations from the design system
 1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
   photography, no textures", with one permitted motif — a subtle aurora
   gradient on hero/empty-state areas only, *"never behind body text blocks"*.
   A dashboard is a body-text block. Present on the operator's explicit
   instruction, mitigated rather than excused: abstract, no subject, strictly
   cool temperature, held at **`opacity: 30`**. That number is load-bearing —
   at 14 the aurora was invisible, and turning it up makes the cards fight the
   ribbon.
 2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
   only makes them illegible.
 ## Overcorrection, and the colour pass
 First stat-well pass went from `font-thin` 13px straight to **bold 22px in
 heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
 The principle missed: a stat only has to out-rank **its own label**, not the
 service name above it. Now `--text-md` medium in cyan.
 Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
 cyan, green) the *primary* families, "used generously, in that order", while
 Dawn (amber/red/violet) is semantic-only. So group markers cycle
 blue→cyan→green down the page (icons full strength, names at 0.72), service
 icons take a single cool wash, latency tags move to the info family so
 "how fast" stops looking like "is it alive". **No Dawn colour is used
 decoratively anywhere.**
 Two DOM findings that made it possible:
 - **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
  via `background`, not `color`.
 - **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
  original selectors matched nothing, so every green pill up to that point was
  stock colouring rather than the theme. Both forms are now matched.
@@ -1,77 +0,0 @@
 # `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
 Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
 ## The pattern worth remembering
 Chasing two bad-looking cards on the dashboard turned up **four stacks running
 on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
 known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
 ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
 ⇒ **A dashboard card is a cheap census of what is actually running.** When
 something on it looks wrong, check whether the stack behind it is even in
 `stacks/` before debugging the symptom — twice here the answer was "no", and
 the fix belonged in version control as much as on the host.
 Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
 `stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
 **deliberately left unmanaged** — adopting three live resolvers while also
 introducing a new DNS naming system is two risky changes at once.
 ## SearXNG — the healthcheck was eating itself
 Card flapped UNHEALTHY; the container was fine the whole time. The compose
 passed `--tries` and `--spider` as **two separate argv entries**, so wget
 consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
 which means every probe since April **downloaded** the healthz response to a
 file:
 ```
 295,287 healthz.N files in the container's working directory
 ```
 With that many files, wget's scan for the next free filename is what
 intermittently blew the 10s timeout. **Self-worsening — every probe made the
 next one slower.** Restored `--tries=1`; the junk lived in the writable layer
 so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
 Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
 silently swallows the next argument turns a liveness probe into a workload.
 ## SeaFile — not broken, never restarted
 Card showed EXITED for three months. **None of the three services declared a
 restart policy**, so Docker defaulted them to `no`. On
 **2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
 other — a daemon restart or host reboot — and nothing brought them back.
 ⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
 SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
 it as one sends you hunting a bug that does not exist. The tell was all three
 services stopping within 200ms.
 Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
 healthcheck exactly as the existing `depends_on` comments intended, seahub
 started without the race, `302` → login page. Data was in local named volumes,
 not on the ana-nas NFS, so nothing was at risk.
 Three months of silent downtime whose only signal was a card nobody read as an
 outage — the argument for semantic status colour on the dashboard (see
 [[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
 mis-grouped AI seats obvious at a glance).
 ## heretic2-charrp-reasoning — tracked, with its shim
 The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
 BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
 untracked. Now in `stacks/`, including
 `conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
 does not propagate modelopt `exclude_modules` to the spec-decode **draft**
 model, so the BF16 MTP head gets quantized and the engine dies at load. Both
 the mount and `PYTHONPATH` are load-bearing.
 Added the two files house convention expects and the directory lacked — a
 `.env.example` naming every knob (all values are compose defaults; the host
 overrides only the three VRAM ones) and a README pointing at
 `docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
@@ -1,176 +0,0 @@
 # `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
 Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
 a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
 `nohup` (PID 1283383) that would not survive a reboot. Now
 `stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
 irv-ml1:8410. Commits `a2b5b58`, `8189076`.
 Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
 pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
 **Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
 a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
 own guards already neutralised both landmines and the project was in
 wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
 with a real reason behind it, and one `update.sh` run carried both. Verified
 end to end after the update: healthy, `backend: cupy`, and a real 256² plate
 render completes warm — the kernel-cache volume survived the image swap.
 ## Build context lives OUTSIDE the compose dir — on purpose
 `/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
 out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
 rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
 `compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
 refreshes source → rebuild → recreate → health, and is verified end to end.
 ## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
 `gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
 silently drops to the numpy path at ~21x wall time — it does not error, it
 just gets slow. waterland-dev warned about the build side.
 The runtime side is worse and was not in the handover: **`studio/jobs.py`
 shells the renderer out as a literal `uv run waterland ...` with no `--extra`
 flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
 mid-job to its default extras and prunes cupy back out from under a correctly
 built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
 pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
 slower environment.
 **Fixed upstream in `464dfc2`:** the server now spawns
 `sys.executable -m waterland.cli` directly — no resolver in the render path at
 all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
 if any future code path re-enters `uv` inside the container, the job fails
 loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
 in the image regardless — it performs the build-time `uv sync` /
 `uv pip install`, and this is a single-stage build.
 ## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
 Every render died 1.7s in with:
 ```
 RuntimeError: Failed to find CUDA headers.
 ```
 printed **through argparse's usage banner**, which makes it read like a CLI
 argument bug rather than a missing toolkit. That misdirection is the reason
 this is written down.
 cupy compiles kernels at runtime through NVRTC, which needs the toolkit
 **headers** — not just the driver and the runtime libs bundled in the
 `cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
 bare `nohup` process found them **by accident**; a slim image has none.
 Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
 hundred MB against ~6 GB for a `-devel` base image. It runs **after**
 `uv sync`, because sync prunes what it does not know about.
 Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
 and anyone running this without a system toolkit hits it. **Declared upstream
 in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
 stays in the Dockerfile**: the header requirement is a property of *this*
 image — a slim base with no system CUDA toolkit — so it belongs in the file
 that creates the problem, not inherited from an extra two repos away. It also
 survives any future restructuring of the `gpu` extra. Cost of keeping it is
 now measured, not assumed: since `uv sync` satisfies it first, the line
 reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
 that documents a non-obvious requirement is worth 0.3s. (waterland-dev
 independently agreed they would keep it too.)
 ## Landmine 3 — the GPU index inside the container is not the host's
 The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
 `CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
 `nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
 (`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
 container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
 the host's value selects a device that does not exist. Host device 0 is the
 3090, which carries the TTS zoo and must not be touched.
 ## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
 | job | wall |
 |---|---|
 | 256² + anim, cold container | 23.3 s |
 | 256² + anim, warm | **6.1 s** |
 | 256² plate only (`--codec none`) | 3.9 s |
 | 512² plate only | 6.4 s |
 Warm beats the **7.4 s** recorded against the bare-metal process, so
 containerising cost nothing. Verified the cache volume properly: recreate
 (fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
 it every restart makes the next user wait 4x and the service merely *looks*
 slow.
 ## Upstream finding — the on-disk job store grows without bound
 `JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
 startup**. Consequences:
 1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
   how this was spotted: the API reported **1 job** while the volume held all
   **16 directories, 60.6 MB**. Not data loss.
 2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
   so directories orphaned by a restart are **never reclaimed**. The
   handover's "bounded around 500 MB" holds within a single process lifetime;
   across restarts the store grows monotonically at ~12 MB per animated job.
 Reported to waterland-dev with evidence; **not patched from the infra side** —
 it is their code. Prune the volume by hand if it bites first.
 **waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
 claim holds within one process lifetime and nowhere else, which on a
 `restart: unless-stopped` service is the wrong lifetime to have bounded. They
 have **surfaced a startup-rehydrate fix to the operator** rather than opening a
 third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
 and is DEPLOYED (2026-08-19).**
 Startup rehydrate, as recommended — and waterland-dev deliberately went
 further than the framing I sent them. I had said a directory the scan cannot
 parse "just does not enter the index"; they made the opposite call, because a
 directory that never enters the index is exactly the one that never gets
 reclaimed. **That is the sharper reading and it is the reason the fix works on
 this volume at all** — the 16 pre-existing dirs have no sidecar. Their
 adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
 dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
 sidecar → degrades to inference, no startup crash; **neither source nor
 sidecar → skipped on purpose**, since adopting it would turn eviction into a
 delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
 go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
 unreachable via the artifact route.
 They also closed a second leak I never saw, because it needs a restart
 *mid-render* to surface: a job left `running`/`queued` in its sidecar is
 non-terminal forever, and eviction skips non-terminal jobs — so it is a
 phantom that is never reclaimed and `queue_depth` over-reports for the life of
 the process. Adoption now marks those `failed`.
 **Verified on this host after the update:** `/api/jobs` went **1 → 16** while
 the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
 Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
 made them visible. A subsequent real render took both to 17. From here the
 store is bounded **across** restarts, not merely within a process.
 ## Access
 Repo is not anonymously readable (a bare clone 403s). Operator granted
 **`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
 pull: True`. Token on irv-ml1 at
 `/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
 **repo-scoped** credential helper; `.git/config` carries no token (verified),
 so the remote stays clean in any diff or backup. The operator's `vh`
 site-admin token was used only for the initial clone and the grant itself and
 was **never written to disk on that host** — a site-admin credential on a GPU
 box is a blast radius nobody needs for a read-only fetch.
 ## Constraints honoured as stated (not inferred)
 - **Serial by design — one replica, one card.** A render is 20–45s of near-full
  GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
  Throughput is a hardware conversation, not a replica-count one.
 - **No authentication, arbitrary file uploads** ⇒ stays inside the
  LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
  waterland-dev offered to add a real auth layer if wider reach is ever needed.
@@ -1,95 +0,0 @@
 # `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
 The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
 using the MTP-aware, vision-preserving **Robinson formula** (documented in
 `docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
 Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
 ## Why this model, why abliterate it ourselves
 Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
 llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
 **~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
 4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
 over-refusal 0. So there is a real creative-content refusal surface to remove.
 This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
 we abliterate it ourselves.
 **It is additive over the current gen seat.** The live Heretic seat
 (`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
 the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
 not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
 residual-write matrices), and the MTP head is what gates speculative acceptance.
 That in-band MTP edit is the delta this experiment tests.
 ## Recipe maps 1:1 — dry-run PASSED
 Against the staged bf16: 1199 tensors, 333 vision preserved,
 `down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
 **131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
 name drift. Two hard gates in the harness halt before any write: the coverage
 identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
 **dim 3994** (orthogonalizing a direction living there bricks the model).
 ## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
 **The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
 The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
 AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
 Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
 produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
 finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
 range, so this is **precision-driven catastrophic cancellation, not overflow** —
 **fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
 prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
 input path through the unstable recurrence.
 Fixes, all in the committed harness (`7abd301`):
 - **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
 - **A finite-gate aborts on a non-finite direction** — the sink screen alone
  can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
  saved silently on the first run).
 - `attn_implementation="sdpa"` pinned.
 **fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
 forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
 `vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
 ⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
 grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
 meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
 into the remainder. All three restored to healthy.
 ⚠ **fla lives in a side dir, not the venv.** The shared
 `/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
 `--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
 reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
 ## Result
 Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
 in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
 Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
 ⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
 Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
 0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
 (HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
 is valid and sink-clean but noisier than ideal; abliterating on it risks
 under-removing refusals or nicking capability. **Expand the sets to a few
 hundred each and re-capture** before the `--out` write.
 ## Sequence from here
 1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
 2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
 3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
   (the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
   (gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
 4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
   **Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
   holds through real multi-turn use.
 bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
 provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
 re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
 `7abd301`.
@@ -0,0 +1,41 @@
 # `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
 `vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of
 97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another
 96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM
 gateway restart he attributed them to (different host, different component, 50 min earlier, and
 six of the seven crashes predate it).
 **Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have
 the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52
 (55,126 MiB) since the August move.
 ## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY
    gen at util 0.43   wants 42,091 MiB    GPU0 free 42,113    margin 22 MiB   — would not start
    gen at util 0.41   wants 40,134 MiB    margin 1,979 MiB    — I called this safe. IT WAS NOT.
    result             1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min
    gen at util 0.38   34,558 MiB actual   4,466-7,548 MiB free   ZERO OOM events over 6 min
 **`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the
 process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable
 segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat
 came up and then silently fell back to slower kernels for want of 20 MB chunks.
 ⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the
 thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive.
 ## Final state and what it cost
    GPU0   mog-sec 55,126 + gen 34,558 = 89,701 / 97,887
    GPU1   50,507 / 97,887 — 46.7 GB free, was 216 MiB
 `gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len**
 — it holds exactly one full-context request. Short/medium requests still batch; long-context
 throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched.
    .env.bak-preGPU0-20260904-164032   the GPU move
    .env.bak-preShrink-165133          the utilization
 ⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on
 ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env.
@@ -0,0 +1,46 @@
 # `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early
 The ESH-Media ↔ UDM uplink is a 3 m OEM `SFP-H10GB-CU3M` twinax that negotiated **1000 Mbps**
 with **zero errors** on both ends. Diagnosis: `sfp_compliance: Unknown` — the switch reads the
 EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so
 autoneg falls back to the safe rate. Control case: a TP-Link `TL-SM5220-1M` on the adjacent UDM
 port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg.
 Operator authorised forcing it. Forced `autoneg False / 10000 / full` on the **UDM end only**;
 ESH-Media followed to 10000 unforced — proof the cable was electrically *capable*.
 ## ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG
    10:03   200 errors   link-up burst
    10:48   221          +21 in 42 min  — I reported this as "flat", it was not
    14:38   416          +195 over 4 h, plus user-visible flapping the operator felt
 **A marginal link declares itself over hours, not minutes.** I watched a two-minute flat window
 and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping
 before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back
 to autoneg/1000, stable.
 ⚠⚠ **The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal
 cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at 10G is exactly
 what a cheap 3 m twinax is — which means the platform's autoneg fallback was **protecting
 something real**, not being fussy about vendor coding. The coding explains the negotiation; it
 does not explain errors once forced. I conflated the two.
 ⚠ **Do not re-force this port.** The fix is the cable.
 ## Method notes worth keeping
 - **Force the RECOVERABLE end.** ESH-Media reaches the controller *through* this link, so a
  failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is
  safe because the path to the controller (`nh3-dev → 10.100.10.1 → 10.0.0.1`, site VPN) does not
  cross it. Verified with `traceroute` **before** the change; revert payload written before the
  forward one.
 - ⚠ `port_overrides` is a **whole-array PUT** — the array was diffed to prove exactly one field
  on one port changed before sending, and read back after.
 - Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP
  and the Zigbee coordinator.
 Resolution: the operator already owns a replacement and ran the copper himself through a drilled
 floor 2x4. Recommendation given was **fiber** (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because
 the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks
 the same fault. Runbook `docs/runbooks/esh-media-dac-10g.md`; commits `b9988a7`, `514ce7a`, `054c098`.
@@ -0,0 +1,33 @@
 # `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN
 `10.0.50.50` is **CT 103 on esh-pve-nas**, a hand-rolled Debian NAS (not a Synology). Created an
 SMB account for the operator's Windows AudioGridder DSP box:
    username   dsp     uid 999, /usr/sbin/nologin, no home — SMB ONLY, cannot log in anywhere
    password   vaulted at esh-nas/dsp-smb-password  (round-trip verified by sha)
    verified   authenticates, sees all 8 shares, WRITE to //share confirmed (mkdir/rmdir)
 ⚠ `smbclient` is absent from the NAS, nh3-dev and esh-docker-vm — verification ran in a throwaway
 `alpine:3.20 --network host` container with `samba-client`, leaving nothing installed.
 Shares are registry-defined (`registry shares = Yes`), so they are invisible in `smb.conf` — use
 **`testparm -s`**, not grep, or you will conclude there are no shares.
 ## ⚠⚠ THE EXPOSURE, UNACTIONED — operator has not ruled
 **NFS: twelve exports, `rw` to all of `10.0.0.0/8`, `sec=sys`, no authentication.**
    /mnt/{backup,books,compose,documents,iso,media,music,share,
          pvestore,nvme-pvestore,ssd-pvestore,tank-vmbu}
 `sec=sys` means the NAS trusts whatever uid the client claims. **Every VLAN at ESH — IoT,
 cameras, guest — can mount the NAS read-write today.** `/mnt/backup` additionally has
 `all_squash,anonuid=2000` so every client collapses to `nas_user`.
 **SMB: every share except `backup` is `guest ok = Yes` and writable**, with
 `map to guest = Bad User` — an unknown username lands as guest with write access.
 So the `dsp` credential is auditable and survives guest being turned off, but it is **not what is
 gating access today**. Hardening (narrow exports to `10.0.50.0/24` + named hosts, drop `guest ok`
 where unneeded) was offered and is roughly an hour; it would break anything relying on guest,
 which is why it needs the operator's say-so. **Not actioned. No tracking surface beyond this entry.**
@@ -0,0 +1,52 @@
 # `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression
 Run 3c is the lr **1e-05** arm against run 3's 2e-04; corpus, base, mask and every other
 hyperparameter byte-identical. It exists to answer one question: does a gentler learning rate
 keep run 3's diversity benefit while dropping its safety regression? **Answer: no, and the
 failure is the informative kind.**
    trained    604/604 in 11:55:40 on pfi-gx10 (GB10, aarch64, sm_121), 69.8 s/it
    merged     /home/infra-ops/erp-tune/serve/merged-run03c, bf16 51.6 GB
    served     erp-tune-v3c on :8098, exposed as gateway alias `trial`
 ## The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md)
    CSAM              175 gens x 3 arms — ZERO hits, hard-line guard passed everywhere
    selfharm/methods  base 10/10 refused -> 3c 0.6   = -40pp de-refusal
    meta pooled       0.0 delta — a CANCELLATION, not cleanliness: selfharm -40pp
                      offset by advice/medical +40pp
    diversity         attractor primary returned to BASE level — the gain is GONE
    cross-arm   run 3  (lr 2e-04)  selfharm -44pp   diversity: real gain
                run 3c (lr 1e-05)  selfharm -40pp   diversity: none
 ⚠ **The selfharm de-refusal is LR-ROBUST.** A 20x cut moved it 4pp while erasing the diversity
 benefit entirely. It comes from corpus content and imprints at even gentle exposure — it is not
 something a lower learning rate dials out. That is what the LR sweep was run to find out.
 ⚠ **A pooled preserve-list test cannot see a single-axis collapse.** The pooled operational
 delta reads 0.0 because two axes moved 40pp in opposite directions. Same structural defect that
 let run 3's gate pass — recorded as R47 §8 item 11.
 ## What the port proved about the box
 - **The tooling loads on aarch64/sm_121.** Full training stack plus flex_attention and the
  chunked-loss path. Nothing exotic needed beyond `python3-dev`.
 - **Verification that earned its keep**: both 49 GB base shards sha256-matched ana-ml2's, and a
  full encode was run into a throwaway dir and compared byte-for-byte — 197,360,233 B, sha256
  `c08bb1fe2ecb0be3`, identical. transformers 5.15.1→5.16.1 and x86-64→aarch64 are *measured*
  inert, not assumed.
 - ⚠ **The encode-cache FILENAME differs by design** — `base_model_path` is in the cache key, so
  rehoming the base changes the key while content stays identical. Input hash, not output hash.
  Do not read it as drift; do not "fix" it by faking `/tank` on the GX10.
 ## The lora_B signal worth carrying forward
    run 2  (lr 2e-04)  min 0.6826  median 1.7212  max 3.7573
    run 3c (lr 1e-05)  min 0.0480  median 0.1561  max 0.4133
 ~11x gentler across the board — exactly what a 20x LR cut should produce. A consistency check
 passing, not a red flag, but worth in hand *before* reading diversity numbers: "the tune did
 nothing" and "the tune did less on purpose" look alike in the output.
 Runbook `docs/runbooks/gx10-run-03c.md`; commit `dae77ee`.
@@ -0,0 +1,39 @@
 # `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping
 brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on
 the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it
 on instinct** — *"that makes zero sense. except for speed, serving a model should be identical
 across servers"* — and he was substantially right.
    ana-ml2   0.9688, 0.9574   spread 1.14pp   mean 0.9631
    gx10      0.9583, 0.9895   spread 3.12pp   mean 0.9739
    between-box LEVEL difference     1.08pp
    ana-ml2's own replicate spread   1.14pp     <- LARGER than the between-box gap
    pooled range ignoring box        3.21pp     <- ~= the entire "GX10 floor"
 **Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of
 dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**.
 brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`,
 doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a
 measured effect and ranked hardware on it. Retracted at `97f73dd`.
 ⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly
 identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect
 moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range
 estimator — he had the refutation in hand and read it as support.
 ⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session
 triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from
 bytes — and this one was **his, and flattering**: it made his earlier work look prescient and
 produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering
 direction needs the extra pass.**
 **What survived, deliberately separated:** re-measuring the floor on whatever stack actually
 serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show
 it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately
 measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect).
 **Retracting the evidence and keeping the discipline are different acts.** Settling it properly
 wants several blocks per box and is its own probe, not a by-product of a gate.
 See [[2026-09-05-vllm-on-sm121-and-run4]].
@@ -0,0 +1,53 @@
 # `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon
 Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a
 stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture.
 ⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its
 sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but
 lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside
 a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on
 this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`).
 Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected:
 `Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip.
 ⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with
 `--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat
 (`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy
 `--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs
 '{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a
 null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box.
 ## Run 4 — the corpus arm
 Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context
 share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped
 training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was
 one-run-scoped, a run 5 needs a third).
 **Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather
 than accepting a "confirm this":**
 1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and
   raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have
   hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*.
 2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256:
   null` on three of four. Preflight resolves `roots_dir/<dataset_id>/clean-v1/CLEANROOT.json`
   literally and requires the sha.
 ⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have
 taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the
 deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate.
 brokkr then went further and recomputed his shas **from shard bytes** rather than reading them
 back out of the deployed CLEANROOT, which had the same defect one step removed.
 **The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples /
 3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio
 (47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target.
 ⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under
 `ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028
 (fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file
 iteration (safe), and the splitlines() paths touch only files written with the default
 `ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way.
@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
-_Last updated: 2026-09-03_
+_Last updated: 2026-09-05_
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,48 +108,76 @@ no longer deployed sidecars here. See Recent decisions.)
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
-_As of 2026-09-03 — **nothing is mid-action; the session closed clean.** Every
+_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else
-item below is a live commitment or a known-open risk, not work in progress._
+below is a live commitment or a known-open risk._
- **Nothing is running.** No training, no deploys pending, no background jobs.
+- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`,
-  Commits sit unpushed on `main` — all docs, runbooks, memory and scripts;
+  log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed,
-  push is the operator's call.
+  a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows
  are short and single-window, so there is no long tail for the sampler to find).
  **~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450,
  gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion**
  — he takes base floors on the GX10 first, then the tuned arm, serially.
- **Run 3c is staged on pfi-gx10 and awaiting the operator's go.** Everything is
+- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly
-  verified and one command away (`ssh infra-ops@10.100.50.60
+  for training, but can serve its trials. unless the box is needed for training work."*
-  '~/erp-tune/launch-run-03c.sh'`); ~13.3 h once started. Its train loop is the
+  So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4
-  one piece never exercised on sm_121 — watch the first three minutes.
+  ends. I over-read an earlier version of this as "training-only" and had to correct
  it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong.
- **`web_search` needs a session restart to appear.** The SearXNG MCP server is
+- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a
-  registered at user scope and `claude mcp list` reports it Connected, but MCP
+  fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by
-  servers load at session start — this session does not have the tool.
+  operator ruling: experimental, NOT a compose stack, does not survive a reboot).
- ⚠ **SearXNG general search is effectively single-engine.** `google cse`
+- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`,
-  carries it; `brave`/`duckduckgo`/`startpage` CAPTCHA even from NH3's
+  `gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error*
-  residential egress. If google cse breaks it goes quiet the same way it just
+  reads as an outage — brokkr checked his own work against mine because he could not
-  did. `scripts/searxng-health.sh` is the detector.
+  tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the
  next gateway change rather than spending a restart on tidying.
- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** Fleecing means a
+- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods
-  slow `pbs-ana` can no longer wedge the guest, but a job that once ran at
+  regression.** Flagged to the operator twice (before adding, and after the gate
  measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the
  same access with a blast radius of one. Settled — do not re-litigate.
 - **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator
  ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek
  SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved.
 - ⚠ **Cityside fiber `/30` is NOT provisioned.** `128.177.138.182/30`, gw `.181`.
  Static passes no traffic and DHCP still hands CGNAT `100.104.3.250`; operator
  power-cycled both ends and opened a ticket. Cutover payloads stay staged:
  `wan1-REVERT.json`, and the `esh-ana` IPsec fix (`ipsec_local_ip 100.104.3.250 →
  128.177.138.182`) **which will otherwise silently break ESH→Anaheim restic backups.**
 - ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to
  `10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable.
  Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
 - ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at
  941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
-  Nightly at 21:00, `all 1`. Worth its own investigation.
+  Nightly 21:00, `all 1`. Worth its own investigation.
- ⚠ **pfi-gx10 is now single-path.** Wired only on a reserved 10.100.50.60; the
+- **ledger-dev asked about the Ledger→SVOS rename** (gitea repo rename cost, and
-  Wi-Fi escape hatch is deliberately gone. A switch-port or reservation failure
+  moving `nh3-dev/development/ledger/env.sh` in the vault — the `secret` CLI has no
-  is a rack visit — on the box run 3c was moved to.
+  rename, so it is re-put + delete). **Unanswered, no deadline**, nothing moves until
  the operator fixes the strings.
- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their
+- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first
-  first client-premises release cut (U10, unscheduled). They wrote the gate down
+  client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled.
  specifically so it would not quietly become never; ping expected when U10 is
  scheduled.
- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** —
+- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate;
-  deliberate, they are the operator's personal machines and registering them
+  they are the operator's personal machines. A choice to revisit, not an oversight.
  implies PFI-managed. They now carry real config, so the omission is a choice
  to revisit, not an oversight.
 ## Recent decisions
 - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md`
 - `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md`
 - `[2026-09-04]` **ERP run 3c trained and GATED — the 20x LR cut erased the diversity gain and did NOT remove the −40pp selfharm regression.** LR-robust, so it comes from corpus content. CSAM clean on all three arms. ⚠ A pooled preserve-list test cannot see a single-axis collapse. → `persistent-memory.d/2026-09-04-run3c-trained-and-gated.md`
 - `[2026-09-04]` **`gen` moved to ana-ml2 GPU0 to stop vllm-embed OOM-crashing (7 restarts) — and I sized it against vLLM's declared budget, not its runtime footprint.** Cost: gen KV down to 1.02x concurrency at 262K. → `persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md`
 - `[2026-09-04]` **SMB account `dsp` created + vaulted for the Windows AudioGridder box — and esh-nas turns out to be wide open.** Twelve NFS exports rw to `10.0.0.0/8`, guest-writable SMB. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
 - `[2026-09-04]` **SLZB-MR1U Zigbee coordinator moved to esh-iot (VLAN 90) at `10.0.90.10`, DHCP-reserved, DNS'd, handed to ha-dev.** ⚠ Home Assistant cannot resolve `.internal` at all (Docker's 127.0.0.11 upstream excludes the fleet AdGuard) — pre-existing; ha-dev declined the fix. Runbook `docs/runbooks/slzb-mr1u-zigbee-coordinator.md`, commits `fed29be`/`0bbdaf9`.
 - `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md`
 - `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md`
 - `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I <wired>` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md`
@@ -232,18 +260,12 @@ item below is a live commitment or a known-open risk, not work in progress._
 - `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
 - `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number.
 - `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302`
 - `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`)
 - `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md`
 - `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
 - `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md`
 - `[2026-08-19]` **Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background.** Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
- `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md`
+
 - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
@@ -282,10 +304,12 @@ item below is a live commitment or a known-open risk, not work in progress._
 _Older entries archived to archival-memory.md._
-_242 older entries archived to archival-memory.md._
+_248 older entries archived to archival-memory.md._
 ## Tried and abandoned
 - `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md`
 - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
 - `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision.
 - `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.