diff --git a/archival-memory.md b/archival-memory.md index 43028d0..bfe42b0 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -3410,3 +3410,634 @@ strictly better than softdog; treat a real firing as unconfirmed until tested. - **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts that are demonstrably online). `stat/sta` is the live view. _Archived 2026-09-03._ + +- `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`) + +_Archived 2026-09-05._ + +# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight + +The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` +using the MTP-aware, vision-preserving **Robinson formula** (documented in +`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`). +Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2. + +## Why this model, why abliterate it ourselves + +Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on +llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug): +**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**; +4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign +over-refusal 0. So there is a real creative-content refusal surface to remove. +This **supersedes** the earlier "watch for DavidAU's own heretic build" posture — +we abliterate it ourselves. + +**It is additive over the current gen seat.** The live Heretic seat +(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** — +the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could +not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2 +residual-write matrices), and the MTP head is what gates speculative acceptance. +That in-band MTP edit is the delta this experiment tests. + +## Recipe maps 1:1 — dry-run PASSED + +Against the staged bf16: 1199 tensors, 333 vision preserved, +`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly +**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no +name drift. Two hard gates in the harness halt before any write: the coverage +identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on +**dim 3994** (orthogonalizing a direction living there bricks the model). + +## Capture SUCCEEDED — but only after a real environment fight (the durable lessons) + +**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.** +The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine) +AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**). +Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which +produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input: +finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent +range, so this is **precision-driven catastrophic cancellation, not overflow** — +**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain +prompt = 65 layers all finite; chat-template input = NaN; the trigger is the +input path through the unstable recurrence. + +Fixes, all in the committed harness (`7abd301`): +- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN). +- **A finite-gate aborts on a non-finite direction** — the sink screen alone + can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and + saved silently on the first run). +- `attn_implementation="sdpa"` pinned. + +**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the +forward OOM'd against the resident seats. Had to **stop three seats** for VRAM: +`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`. +⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily +grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving +meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen +into the remainder. All three restored to healthy. + +⚠ **fla lives in a side dir, not the venv.** The shared +`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are +`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and +reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers. + +## Result + +Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%** +in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated. +Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`. + +⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.** +Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's +0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless** +(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction +is valid and sink-clean but noisier than ideal; abliterating on it risks +under-removing refusals or nicking capability. **Expand the sets to a few +hundred each and re-capture** before the `--out` write. + +## Sequence from here + +1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down). +2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`. +3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/` + (the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant + (gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence. +4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate. + **Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it + holds through real multi-turn use. + +bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`, +provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch +re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`, +`7abd301`. + +_Archived 2026-09-05._ + +# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers + +Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to +keep track of local .internal dns names that can be agent managed and is +lightweight."* Built and live in one session; commit `b8003c7`. + +## Shape + +``` +dns/internal.yaml source of truth — 38 hosts + 4 service aliases +scripts/dns-sync.py reconciles AdGuard resolvers against it +stacks/adguard-ana/ the colo's resolver, which did not exist +dns/README.md workflow, naming, the IPv6 caveat +``` + +Deliberately the same posture as `deploy-stack.sh`: the file is intent, the +resolvers are derived state, you see a diff before anything changes. +`--dry-run` / `--yes` / `--site `. Verified idempotent — a second run prints +`nothing to do`. + +Naming is `..internal` with sites **`ana` / `esh` / `nh3`** +(operator's call). `.internal` is ICANN-reserved for private use since 2024; +`.local` is reserved for mDNS, which is why the pre-existing +`searxng.pfi.local` was a standards collision that merely happened to work. + +Every name is published to **every** resolver — the site label says where a +host *is*, not which resolver knows about it. + +## The framing correction that mattered most + +The ask reads as "I can't memorise v6 addresses", but the deeper problem is +that **v6 addresses are derived, not assigned**, so they cannot reliably be +*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or +privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a +DHCP reservation** — so a hand-maintained v6 table rots on its own. + +⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on +server-class hosts, (2) then the name table is just a file. Surfaced to the +operator before building. + +**Verified 2026-08-19: no fleet host has a global v6 address at all yet** — +ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to +`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships +EMPTY and correct, and the naming layer was built first rather than blocking +on v6. Names established now need no renaming when addresses land. + +Suggested convention when they do (awaiting operator): each server static at +its site's `/64` with low-order bits echoing the v4 host octet — +`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable. + +## Two properties not to break + +**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in +`.internal` are managed. ESH's resolver turned out to carry three hand-made +`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first +read of the config missed them**, because an `awk` range on `rewrites:` matched +an empty-looking block. A resolver-wide authoritative sync would have silently +deleted all three on first run. Verified intact after sync. + +**Within `.internal` it IS authoritative** — names added by hand in the AdGuard +UI get deleted by the next sync. That is the point: one place to look. + +## The colo had no resolver at all + +ESH and NH3 each ran AdGuard; **ana-docker resolved straight against +`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with +`stacks/adguard-ana/`. + +⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on +that busy host. The port is therefore carried **per-site in the yaml**, not +assumed by the script, so the odd one out cannot be forgotten. + +⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for +human browsing; this one resolves for a rack of servers, where a blocklist +false-positive breaks service-to-service calls at 3am for no upside. + +First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into +the conf volume before first start, so the container comes up configured +instead of sitting in the setup wizard. + +## Credential — service account, not the operator's + +Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather +than asking for the `lkraven` password (per the standing migrate-off-operator- +creds directive). Password vaulted at +`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs +backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing +resolvers kept answering across the restart. + +Two landmines worth keeping: + +- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different + marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise + the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it + on a live resolver. +- **The vault appends a trailing newline on `get`.** A password carrying a + stray `\n` fails auth in a way that looks exactly like a wrong password. + `dns-sync.py` strips it. + +## `pfi.local` migration — and the one that must NOT move + +`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept +alongside** in the Traefik rule so nothing breaks mid-migration; both return +200. Drop the fallback once the access log shows the old name unused. + +**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is +baked into every user ID, room ID and signing key, and federation identity +derives from it — renaming it is not a DNS change, it is rebuilding the +homeserver's identity and invalidating its history. The operator approved +"migrate pfi.local" generally; this was surfaced as a deliberate exclusion +rather than executed blindly. + +## Still open + +Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver +— it only answers what asks it directly. Repointing a whole site's DNS is a +bigger change than standing the service up, and is the operator's to schedule. + +See also [[2026-08-17-fleet-ipv6-mesh]]. + +_Archived 2026-09-05._ + +# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured + +Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`): +a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare +`nohup` (PID 1283383) that would not survive a reboot. Now +`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on +irv-ml1:8410. Commits `a2b5b58`, `8189076`. + +Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the +pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway. + +**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for +a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's +own guards already neutralised both landmines and the project was in +wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild +with a real reason behind it, and one `update.sh` run carried both. Verified +end to end after the update: healthy, `backend: cupy`, and a real 256² plate +render completes warm — the kernel-cache volume survived the image swap. + +## Build context lives OUTSIDE the compose dir — on purpose + +`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed +out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh` +rsyncs `stacks//` with `--delete`**, so a checkout kept beside +`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh` +refreshes source → rebuild → recreate → health, and is verified end to end. + +## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN + +`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer +silently drops to the numpy path at ~21x wall time — it does not error, it +just gets slow. waterland-dev warned about the build side. + +The runtime side is worse and was not in the handover: **`studio/jobs.py` +shells the renderer out as a literal `uv run waterland ...` with no `--extra` +flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project +mid-job to its default extras and prunes cupy back out from under a correctly +built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the +pin ever stops holding the job fails **loudly** instead of quietly rebuilding a +slower environment. + +**Fixed upstream in `464dfc2`:** the server now spawns +`sys.executable -m waterland.cli` directly — no resolver in the render path at +all. **The pins stay anyway.** They cost nothing and are now defence-in-depth: +if any future code path re-enters `uv` inside the container, the job fails +loudly instead of quietly dropping to the numpy backend. `uv` itself must stay +in the image regardless — it performs the build-time `uv sync` / +`uv pip install`, and this is a single-stage build. + +## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare + +Every render died 1.7s in with: + +``` +RuntimeError: Failed to find CUDA headers. +``` + +printed **through argparse's usage banner**, which makes it read like a CLI +argument bug rather than a missing toolkit. That misdirection is the reason +this is written down. + +cupy compiles kernels at runtime through NVRTC, which needs the toolkit +**headers** — not just the driver and the runtime libs bundled in the +`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the +bare `nohup` process found them **by accident**; a slim image has none. + +Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few +hundred MB against ~6 GB for a `-devel` base image. It runs **after** +`uv sync`, because sync prunes what it does not know about. + +Reported upstream: it is an undeclared runtime dependency of the `gpu` extra, +and anyone running this without a system toolkit hits it. **Declared upstream +in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install +stays in the Dockerfile**: the header requirement is a property of *this* +image — a slim base with no system CUDA toolkit — so it belongs in the file +that creates the problem, not inherited from an extra two repos away. It also +survives any future restructuring of the `gpu` extra. Cost of keeping it is +now measured, not assumed: since `uv sync` satisfies it first, the line +reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op +that documents a non-obvious requirement is worth 0.3s. (waterland-dev +independently agreed they would keep it too.) + +## Landmine 3 — the GPU index inside the container is not the host's + +The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects +`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where +`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU +(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the +container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying +the host's value selects a device that does not exist. Host device 0 is the +3090, which carries the TTS zoo and must not be touched. + +## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume + +| job | wall | +|---|---| +| 256² + anim, cold container | 23.3 s | +| 256² + anim, warm | **6.1 s** | +| 256² plate only (`--codec none`) | 3.9 s | +| 512² plate only | 6.4 s | + +Warm beats the **7.4 s** recorded against the bare-metal process, so +containerising cost nothing. Verified the cache volume properly: recreate +(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without +it every restart makes the next user wait 4x and the service merely *looks* +slow. + +## Upstream finding — the on-disk job store grows without bound + +`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at +startup**. Consequences: + +1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and + how this was spotted: the API reported **1 job** while the volume held all + **16 directories, 60.6 MB**. Not data loss. +2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict, + so directories orphaned by a restart are **never reclaimed**. The + handover's "bounded around 500 MB" holds within a single process lifetime; + across restarts the store grows monotonically at ~12 MB per animated job. + +Reported to waterland-dev with evidence; **not patched from the infra side** — +it is their code. Prune the volume by hand if it bites first. + +**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover +claim holds within one process lifetime and nowhere else, which on a +`restart: unless-stopped` service is the wrong lifetime to have bounded. They +have **surfaced a startup-rehydrate fix to the operator** rather than opening a +third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b` +and is DEPLOYED (2026-08-19).** + +Startup rehydrate, as recommended — and waterland-dev deliberately went +further than the framing I sent them. I had said a directory the scan cannot +parse "just does not enter the index"; they made the opposite call, because a +directory that never enters the index is exactly the one that never gets +reclaimed. **That is the sharper reading and it is the reason the fix works on +this volume at all** — the 16 pre-existing dirs have no sidecar. Their +adoption ladder: sidecar → restored verbatim; no sidecar → adopted with +dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt +sidecar → degrades to inference, no startup crash; **neither source nor +sidecar → skipped on purpose**, since adopting it would turn eviction into a +delete-arbitrary-directories primitive pointed at this volume. Sidecar writes +go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is +unreachable via the artifact route. + +They also closed a second leak I never saw, because it needs a restart +*mid-render* to surface: a job left `running`/`queued` in its sidecar is +non-terminal forever, and eviction skips non-terminal jobs — so it is a +phantom that is never reclaimed and `queue_depth` over-reports for the life of +the process. Adoption now marks those `failed`. + +**Verified on this host after the update:** `/api/jobs` went **1 → 16** while +the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time. +Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only +made them visible. A subsequent real render took both to 17. From here the +store is bounded **across** restarts, not merely within a process. + +## Access + +Repo is not anonymously readable (a bare clone 403s). Operator granted +**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False, +pull: True`. Token on irv-ml1 at +`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a +**repo-scoped** credential helper; `.git/config` carries no token (verified), +so the remote stays clean in any diff or backup. The operator's `vh` +site-admin token was used only for the initial clone and the grant itself and +was **never written to disk on that host** — a site-admin credential on a GPU +box is a blast radius nobody needs for a read-only fetch. + +## Constraints honoured as stated (not inferred) + +- **Serial by design — one replica, one card.** A render is 20–45s of near-full + GPU with a single worker thread. Two on the same A6000 would OOM or thrash. + Throughput is a hardware conversation, not a replica-count one. +- **No authentication, arbitrary file uploads** ⇒ stays inside the + LAN/WireGuard boundary. Do **not** paper over it with a proxy password; + waterland-dev offered to add a real auth layer if wider reach is ever needed. + +_Archived 2026-09-05._ + +# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background + +Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`. + +## The cleanup (three real defects) + +- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in + `settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour + is that a group with no `tab:` is shown on **every** tab. Pinned to Main. + ⚠️ This will happen again to the next group added without a `tab:` — the rule + is now written at the top of the layout block. +- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under + Monitoring *and* `homepage.group=Apps` on the container. Exactly the + "never list a labelled container manually" failure the stack README warns + about; it survived the previous day's audit because a duplicate reads as two + plausible cards rather than as an error. Manual block deleted, label moved to + `Monitoring`, `homepage.siteMonitor` added. +- **Column counts were fiction** — several groups declared more columns than + they had members, so the last row of each was dead space (Notes: 1 card in a + 4-wide row). Columns now track member counts; `GET /api/services` prints the + live per-group counts and is the check. + +Later, on operator instruction, the **AI tab was reordered by clickability**: +Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs` +seats and TTS endpoints. Reasoning written into the config so it survives: +order by "would I click this?", not by how central the service is. + +## ⚠️ The expensive red herring — the tab bar after a recreate + +After a recreate the client render comes up with **no tab bar, no wallpaper and +no i18n** (search box shows the raw key `search.search`), groups falling back to +side-by-side columns. **It restores itself with no intervention.** + +Timing, measured rather than assumed: a fresh container was still tab-less at +**4m30s, twice**; it was healthy again after roughly an hour. `docker ps` +reporting `healthy` says nothing about it — the container is serving, the page +is just wrong. + +An hour went into ruling out four causes that were never the cause: + +1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to + their committed versions reproduces it, as does the pre-adoption backup in + `/opt/docker-bu/conf/homepage/`. +2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows + identical symptoms, and the image never changed anyway (working and broken + both report `v2.0.0` / rev `17456f2`). +3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the + socket unmounted entirely. +4. **Not server-side** — the server-rendered HTML still contains the tab + markup, the background URL and `useEqualHeights`; `GET /api/validate` + returns `[]`. The loss is client-side, with no page error, no failed chunk + and no non-200. + +Every throwaway container in that list was judged within ~30s of starting, so +they were all inside the same window — and that consistency **read as a +reproduction when it was the same measurement mistake five times over.** + +**Operative rule: recreate, walk away, re-check later. Do not chase it.** + +## ⚠️ The iteration loop that would have prevented the overcook + +`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS +change needs a **browser reload** — not a container recreate, and it never owed +the layout warm-up above. Conflating the two costs ~10 operator-visible minutes +per attempt (operator called this out directly). + +Faster still, and how the final pass was done: **inject candidate CSS into the +running page and screenshot it** — +`await p.addStyleTag({content: css})` in Playwright against the live +dashboard. Seconds per iteration, no deploy. Build + deploy only once the +render looks right. + +## The theme — Australis Skyfall + +Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`). +Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic +family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a +Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a +"calm depth" language of **hairline + two-layer shadow on every elevated +surface, never one without the other**. + +``` +theme/colors.css layout.css typography.css vendored VERBATIM from the bundle +theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face +theme/skyfall.css.in the Homepage bindings (ours) +theme/build.py → conf/custom.css (generated — do not hand-edit) +``` + +The build step exists for one reason: **Homepage serves only `custom.css` and +`custom.js` out of its config dir**, with no static route beside them, so a +`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive +as a data URI. The background image takes the other road, because +`/app/public/images` **is** a real static route (mounted read-only in +`compose.yaml`). + +Only Supreme is embedded: a link dashboard has no display type, and Victor +Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a +handful of latency figures. + +## The background is generated, not stock + +**Arbo as an image-gen engine** (the operator's actual ask, which I first +misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background` +workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*. +Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB +WebP** (smooth gradients compress absurdly well). + +⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a +bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape. + +## Two documented deviations from the design system + +1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no + photography, no textures", with one permitted motif — a subtle aurora + gradient on hero/empty-state areas only, *"never behind body text blocks"*. + A dashboard is a body-text block. Present on the operator's explicit + instruction, mitigated rather than excused: abstract, no subject, strictly + cool temperature, held at **`opacity: 30`**. That number is load-bearing — + at 14 the aurora was invisible, and turning it up makes the cards fight the + ribbon. +2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS + only makes them illegible. + +## Overcorrection, and the colour pass + +First stat-well pass went from `font-thin` 13px straight to **bold 22px in +heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."* +The principle missed: a stat only has to out-rank **its own label**, not the +service name above it. Now `--text-md` medium in cyan. + +Colour was then lifted **from inside the system**: Skyfall names Aurora (blue, +cyan, green) the *primary* families, "used generously, in that order", while +Dawn (amber/red/violet) is semantic-only. So group markers cycle +blue→cyan→green down the page (icons full strength, names at 0.72), service +icons take a single cool wash, latency tags move to the info family so +"how fast" stops looking like "is it alive". **No Dawn colour is used +decoratively anywhere.** + +Two DOM findings that made it possible: + +- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour + via `background`, not `color`. +- **Homepage emits `docker-status-`, not `status-`.** The + original selectors matched nothing, so every green pill up to that point was + stock colouring rather than the theme. Both forms are now matched. + +_Archived 2026-09-05._ + +# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken + +Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`. + +## The pattern worth remembering + +Chasing two bad-looking cards on the dashboard turned up **four stacks running +on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already +known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on +ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git). + +⇒ **A dashboard card is a cheap census of what is actually running.** When +something on it looks wrong, check whether the stack behind it is even in +`stacks/` before debugging the symptom — twice here the answer was "no", and +the fix belonged in version control as much as on the host. + +Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`, +`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were +**deliberately left unmanaged** — adopting three live resolvers while also +introducing a new DNS naming system is two risky changes at once. + +## SearXNG — the healthcheck was eating itself + +Card flapped UNHEALTHY; the container was fine the whole time. The compose +passed `--tries` and `--spider` as **two separate argv entries**, so wget +consumed `--spider` as the *value* of `--tries`. Spider mode never engaged, +which means every probe since April **downloaded** the healthz response to a +file: + +``` +295,287 healthz.N files in the container's working directory +``` + +With that many files, wget's scan for the next free filename is what +intermittently blew the 10s timeout. **Self-worsening — every probe made the +next one slower.** Restored `--tries=1`; the junk lived in the writable layer +so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s. + +Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that +silently swallows the next argument turns a liveness probe into a workload. + +## SeaFile — not broken, never restarted + +Card showed EXITED for three months. **None of the three services declared a +restart policy**, so Docker defaulted them to `no`. On +**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each +other — a daemon restart or host reboot — and nothing brought them back. + +⚠️ **Exit code `255` is a red herring**: it is what a container that ignores +SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading +it as one sends you hunting a bug that does not exist. The tell was all three +services stopping within 200ms. + +Added `restart: unless-stopped` to all three; brought up; mariadb gated on its +healthcheck exactly as the existing `depends_on` comments intended, seahub +started without the race, `302` → login page. Data was in local named volumes, +not on the ana-nas NFS, so nothing was at risk. + +Three months of silent downtime whose only signal was a card nobody read as an +outage — the argument for semantic status colour on the dashboard (see +[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six +mis-grouped AI seats obvious at a glance). + +## heretic2-charrp-reasoning — tracked, with its shim + +The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted +BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running +untracked. Now in `stacks/`, including +`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0 +does not propagate modelopt `exclude_modules` to the spec-decode **draft** +model, so the BF16 MTP head gets quantized and the engine dies at load. Both +the mount and `PYTHONPATH` are load-bearing. + +Added the two files house convention expects and the directory lacked — a +`.env.example` naming every knob (all values are compose defaults; the host +overrides only the three VRAM ones) and a README pointing at +`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it. + +_Archived 2026-09-05._ diff --git a/persistent-memory.d/2026-08-19-fleet-internal-dns.md b/persistent-memory.d/2026-08-19-fleet-internal-dns.md deleted file mode 100644 index fb9baeb..0000000 --- a/persistent-memory.d/2026-08-19-fleet-internal-dns.md +++ /dev/null @@ -1,119 +0,0 @@ -# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers - -Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to -keep track of local .internal dns names that can be agent managed and is -lightweight."* Built and live in one session; commit `b8003c7`. - -## Shape - -``` -dns/internal.yaml source of truth — 38 hosts + 4 service aliases -scripts/dns-sync.py reconciles AdGuard resolvers against it -stacks/adguard-ana/ the colo's resolver, which did not exist -dns/README.md workflow, naming, the IPv6 caveat -``` - -Deliberately the same posture as `deploy-stack.sh`: the file is intent, the -resolvers are derived state, you see a diff before anything changes. -`--dry-run` / `--yes` / `--site `. Verified idempotent — a second run prints -`nothing to do`. - -Naming is `..internal` with sites **`ana` / `esh` / `nh3`** -(operator's call). `.internal` is ICANN-reserved for private use since 2024; -`.local` is reserved for mDNS, which is why the pre-existing -`searxng.pfi.local` was a standards collision that merely happened to work. - -Every name is published to **every** resolver — the site label says where a -host *is*, not which resolver knows about it. - -## The framing correction that mattered most - -The ask reads as "I can't memorise v6 addresses", but the deeper problem is -that **v6 addresses are derived, not assigned**, so they cannot reliably be -*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or -privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a -DHCP reservation** — so a hand-maintained v6 table rots on its own. - -⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on -server-class hosts, (2) then the name table is just a file. Surfaced to the -operator before building. - -**Verified 2026-08-19: no fleet host has a global v6 address at all yet** — -ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to -`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships -EMPTY and correct, and the naming layer was built first rather than blocking -on v6. Names established now need no renaming when addresses land. - -Suggested convention when they do (awaiting operator): each server static at -its site's `/64` with low-order bits echoing the v4 host octet — -`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable. - -## Two properties not to break - -**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in -`.internal` are managed. ESH's resolver turned out to carry three hand-made -`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first -read of the config missed them**, because an `awk` range on `rewrites:` matched -an empty-looking block. A resolver-wide authoritative sync would have silently -deleted all three on first run. Verified intact after sync. - -**Within `.internal` it IS authoritative** — names added by hand in the AdGuard -UI get deleted by the next sync. That is the point: one place to look. - -## The colo had no resolver at all - -ESH and NH3 each ran AdGuard; **ana-docker resolved straight against -`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with -`stacks/adguard-ana/`. - -⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on -that busy host. The port is therefore carried **per-site in the yaml**, not -assumed by the script, so the odd one out cannot be forgotten. - -⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for -human browsing; this one resolves for a rack of servers, where a blocklist -false-positive breaks service-to-service calls at 3am for no upside. - -First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into -the conf volume before first start, so the container comes up configured -instead of sitting in the setup wizard. - -## Credential — service account, not the operator's - -Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather -than asking for the `lkraven` password (per the standing migrate-off-operator- -creds directive). Password vaulted at -`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs -backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing -resolvers kept answering across the restart. - -Two landmines worth keeping: - -- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different - marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise - the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it - on a live resolver. -- **The vault appends a trailing newline on `get`.** A password carrying a - stray `\n` fails auth in a way that looks exactly like a wrong password. - `dns-sync.py` strips it. - -## `pfi.local` migration — and the one that must NOT move - -`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept -alongside** in the Traefik rule so nothing breaks mid-migration; both return -200. Drop the fallback once the access log shows the old name unused. - -**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is -baked into every user ID, room ID and signing key, and federation identity -derives from it — renaming it is not a DNS change, it is rebuilding the -homeserver's identity and invalidating its history. The operator approved -"migrate pfi.local" generally; this was surfaced as a deliberate exclusion -rather than executed blindly. - -## Still open - -Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver -— it only answers what asks it directly. Repointing a whole site's DNS is a -bigger change than standing the service up, and is the operator's to schedule. - -See also [[2026-08-17-fleet-ipv6-mesh]]. diff --git a/persistent-memory.d/2026-08-19-homepage-skyfall-theme.md b/persistent-memory.d/2026-08-19-homepage-skyfall-theme.md deleted file mode 100644 index 682d1ac..0000000 --- a/persistent-memory.d/2026-08-19-homepage-skyfall-theme.md +++ /dev/null @@ -1,145 +0,0 @@ -# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background - -Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`. - -## The cleanup (three real defects) - -- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in - `settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour - is that a group with no `tab:` is shown on **every** tab. Pinned to Main. - ⚠️ This will happen again to the next group added without a `tab:` — the rule - is now written at the top of the layout block. -- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under - Monitoring *and* `homepage.group=Apps` on the container. Exactly the - "never list a labelled container manually" failure the stack README warns - about; it survived the previous day's audit because a duplicate reads as two - plausible cards rather than as an error. Manual block deleted, label moved to - `Monitoring`, `homepage.siteMonitor` added. -- **Column counts were fiction** — several groups declared more columns than - they had members, so the last row of each was dead space (Notes: 1 card in a - 4-wide row). Columns now track member counts; `GET /api/services` prints the - live per-group counts and is the check. - -Later, on operator instruction, the **AI tab was reordered by clickability**: -Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs` -seats and TTS endpoints. Reasoning written into the config so it survives: -order by "would I click this?", not by how central the service is. - -## ⚠️ The expensive red herring — the tab bar after a recreate - -After a recreate the client render comes up with **no tab bar, no wallpaper and -no i18n** (search box shows the raw key `search.search`), groups falling back to -side-by-side columns. **It restores itself with no intervention.** - -Timing, measured rather than assumed: a fresh container was still tab-less at -**4m30s, twice**; it was healthy again after roughly an hour. `docker ps` -reporting `healthy` says nothing about it — the container is serving, the page -is just wrong. - -An hour went into ruling out four causes that were never the cause: - -1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to - their committed versions reproduces it, as does the pre-adoption backup in - `/opt/docker-bu/conf/homepage/`. -2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows - identical symptoms, and the image never changed anyway (working and broken - both report `v2.0.0` / rev `17456f2`). -3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the - socket unmounted entirely. -4. **Not server-side** — the server-rendered HTML still contains the tab - markup, the background URL and `useEqualHeights`; `GET /api/validate` - returns `[]`. The loss is client-side, with no page error, no failed chunk - and no non-200. - -Every throwaway container in that list was judged within ~30s of starting, so -they were all inside the same window — and that consistency **read as a -reproduction when it was the same measurement mistake five times over.** - -**Operative rule: recreate, walk away, re-check later. Do not chase it.** - -## ⚠️ The iteration loop that would have prevented the overcook - -`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS -change needs a **browser reload** — not a container recreate, and it never owed -the layout warm-up above. Conflating the two costs ~10 operator-visible minutes -per attempt (operator called this out directly). - -Faster still, and how the final pass was done: **inject candidate CSS into the -running page and screenshot it** — -`await p.addStyleTag({content: css})` in Playwright against the live -dashboard. Seconds per iteration, no deploy. Build + deploy only once the -render looks right. - -## The theme — Australis Skyfall - -Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`). -Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic -family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a -Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a -"calm depth" language of **hairline + two-layer shadow on every elevated -surface, never one without the other**. - -``` -theme/colors.css layout.css typography.css vendored VERBATIM from the bundle -theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face -theme/skyfall.css.in the Homepage bindings (ours) -theme/build.py → conf/custom.css (generated — do not hand-edit) -``` - -The build step exists for one reason: **Homepage serves only `custom.css` and -`custom.js` out of its config dir**, with no static route beside them, so a -`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive -as a data URI. The background image takes the other road, because -`/app/public/images` **is** a real static route (mounted read-only in -`compose.yaml`). - -Only Supreme is embedded: a link dashboard has no display type, and Victor -Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a -handful of latency figures. - -## The background is generated, not stock - -**Arbo as an image-gen engine** (the operator's actual ask, which I first -misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background` -workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*. -Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB -WebP** (smooth gradients compress absurdly well). - -⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a -bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape. - -## Two documented deviations from the design system - -1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no - photography, no textures", with one permitted motif — a subtle aurora - gradient on hero/empty-state areas only, *"never behind body text blocks"*. - A dashboard is a body-text block. Present on the operator's explicit - instruction, mitigated rather than excused: abstract, no subject, strictly - cool temperature, held at **`opacity: 30`**. That number is load-bearing — - at 14 the aurora was invisible, and turning it up makes the cards fight the - ribbon. -2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS - only makes them illegible. - -## Overcorrection, and the colour pass - -First stat-well pass went from `font-thin` 13px straight to **bold 22px in -heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."* -The principle missed: a stat only has to out-rank **its own label**, not the -service name above it. Now `--text-md` medium in cyan. - -Colour was then lifted **from inside the system**: Skyfall names Aurora (blue, -cyan, green) the *primary* families, "used generously, in that order", while -Dawn (amber/red/violet) is semantic-only. So group markers cycle -blue→cyan→green down the page (icons full strength, names at 0.72), service -icons take a single cool wash, latency tags move to the info family so -"how fast" stops looking like "is it alive". **No Dawn colour is used -decoratively anywhere.** - -Two DOM findings that made it possible: - -- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour - via `background`, not `color`. -- **Homepage emits `docker-status-`, not `status-`.** The - original selectors matched nothing, so every green pill up to that point was - stock colouring rather than the theme. Both forms are now matched. diff --git a/persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md b/persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md deleted file mode 100644 index 1f33da0..0000000 --- a/persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md +++ /dev/null @@ -1,77 +0,0 @@ -# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken - -Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`. - -## The pattern worth remembering - -Chasing two bad-looking cards on the dashboard turned up **four stacks running -on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already -known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on -ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git). - -⇒ **A dashboard card is a cheap census of what is actually running.** When -something on it looks wrong, check whether the stack behind it is even in -`stacks/` before debugging the symptom — twice here the answer was "no", and -the fix belonged in version control as much as on the host. - -Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`, -`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were -**deliberately left unmanaged** — adopting three live resolvers while also -introducing a new DNS naming system is two risky changes at once. - -## SearXNG — the healthcheck was eating itself - -Card flapped UNHEALTHY; the container was fine the whole time. The compose -passed `--tries` and `--spider` as **two separate argv entries**, so wget -consumed `--spider` as the *value* of `--tries`. Spider mode never engaged, -which means every probe since April **downloaded** the healthz response to a -file: - -``` -295,287 healthz.N files in the container's working directory -``` - -With that many files, wget's scan for the next free filename is what -intermittently blew the 10s timeout. **Self-worsening — every probe made the -next one slower.** Restored `--tries=1`; the junk lived in the writable layer -so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s. - -Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that -silently swallows the next argument turns a liveness probe into a workload. - -## SeaFile — not broken, never restarted - -Card showed EXITED for three months. **None of the three services declared a -restart policy**, so Docker defaulted them to `no`. On -**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each -other — a daemon restart or host reboot — and nothing brought them back. - -⚠️ **Exit code `255` is a red herring**: it is what a container that ignores -SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading -it as one sends you hunting a bug that does not exist. The tell was all three -services stopping within 200ms. - -Added `restart: unless-stopped` to all three; brought up; mariadb gated on its -healthcheck exactly as the existing `depends_on` comments intended, seahub -started without the race, `302` → login page. Data was in local named volumes, -not on the ana-nas NFS, so nothing was at risk. - -Three months of silent downtime whose only signal was a card nobody read as an -outage — the argument for semantic status colour on the dashboard (see -[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six -mis-grouped AI seats obvious at a glance). - -## heretic2-charrp-reasoning — tracked, with its shim - -The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted -BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running -untracked. Now in `stacks/`, including -`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0 -does not propagate modelopt `exclude_modules` to the spec-decode **draft** -model, so the BF16 MTP head gets quantized and the engine dies at load. Both -the mount and `PYTHONPATH` are load-bearing. - -Added the two files house convention expects and the directory lacked — a -`.env.example` naming every knob (all values are compose defaults; the host -overrides only the three VRAM ones) and a README pointing at -`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it. diff --git a/persistent-memory.d/2026-08-19-waterland-studio-containerised.md b/persistent-memory.d/2026-08-19-waterland-studio-containerised.md deleted file mode 100644 index e3de018..0000000 --- a/persistent-memory.d/2026-08-19-waterland-studio-containerised.md +++ /dev/null @@ -1,176 +0,0 @@ -# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured - -Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`): -a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare -`nohup` (PID 1283383) that would not survive a reboot. Now -`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on -irv-ml1:8410. Commits `a2b5b58`, `8189076`. - -Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the -pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway. - -**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for -a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's -own guards already neutralised both landmines and the project was in -wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild -with a real reason behind it, and one `update.sh` run carried both. Verified -end to end after the update: healthy, `backend: cupy`, and a real 256² plate -render completes warm — the kernel-cache volume survived the image swap. - -## Build context lives OUTSIDE the compose dir — on purpose - -`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed -out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh` -rsyncs `stacks//` with `--delete`**, so a checkout kept beside -`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh` -refreshes source → rebuild → recreate → health, and is verified end to end. - -## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN - -`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer -silently drops to the numpy path at ~21x wall time — it does not error, it -just gets slow. waterland-dev warned about the build side. - -The runtime side is worse and was not in the handover: **`studio/jobs.py` -shells the renderer out as a literal `uv run waterland ...` with no `--extra` -flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project -mid-job to its default extras and prunes cupy back out from under a correctly -built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the -pin ever stops holding the job fails **loudly** instead of quietly rebuilding a -slower environment. - -**Fixed upstream in `464dfc2`:** the server now spawns -`sys.executable -m waterland.cli` directly — no resolver in the render path at -all. **The pins stay anyway.** They cost nothing and are now defence-in-depth: -if any future code path re-enters `uv` inside the container, the job fails -loudly instead of quietly dropping to the numpy backend. `uv` itself must stay -in the image regardless — it performs the build-time `uv sync` / -`uv pip install`, and this is a single-stage build. - -## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare - -Every render died 1.7s in with: - -``` -RuntimeError: Failed to find CUDA headers. -``` - -printed **through argparse's usage banner**, which makes it read like a CLI -argument bug rather than a missing toolkit. That misdirection is the reason -this is written down. - -cupy compiles kernels at runtime through NVRTC, which needs the toolkit -**headers** — not just the driver and the runtime libs bundled in the -`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the -bare `nohup` process found them **by accident**; a slim image has none. - -Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few -hundred MB against ~6 GB for a `-devel` base image. It runs **after** -`uv sync`, because sync prunes what it does not know about. - -Reported upstream: it is an undeclared runtime dependency of the `gpu` extra, -and anyone running this without a system toolkit hits it. **Declared upstream -in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install -stays in the Dockerfile**: the header requirement is a property of *this* -image — a slim base with no system CUDA toolkit — so it belongs in the file -that creates the problem, not inherited from an extra two repos away. It also -survives any future restructuring of the `gpu` extra. Cost of keeping it is -now measured, not assumed: since `uv sync` satisfies it first, the line -reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op -that documents a non-obvious requirement is worth 0.3s. (waterland-dev -independently agreed they would keep it too.) - -## Landmine 3 — the GPU index inside the container is not the host's - -The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects -`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where -`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU -(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the -container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying -the host's value selects a device that does not exist. Host device 0 is the -3090, which carries the TTS zoo and must not be touched. - -## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume - -| job | wall | -|---|---| -| 256² + anim, cold container | 23.3 s | -| 256² + anim, warm | **6.1 s** | -| 256² plate only (`--codec none`) | 3.9 s | -| 512² plate only | 6.4 s | - -Warm beats the **7.4 s** recorded against the bare-metal process, so -containerising cost nothing. Verified the cache volume properly: recreate -(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without -it every restart makes the next user wait 4x and the service merely *looks* -slow. - -## Upstream finding — the on-disk job store grows without bound - -`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at -startup**. Consequences: - -1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and - how this was spotted: the API reported **1 job** while the volume held all - **16 directories, 60.6 MB**. Not data loss. -2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict, - so directories orphaned by a restart are **never reclaimed**. The - handover's "bounded around 500 MB" holds within a single process lifetime; - across restarts the store grows monotonically at ~12 MB per animated job. - -Reported to waterland-dev with evidence; **not patched from the infra side** — -it is their code. Prune the volume by hand if it bites first. - -**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover -claim holds within one process lifetime and nowhere else, which on a -`restart: unless-stopped` service is the wrong lifetime to have bounded. They -have **surfaced a startup-rehydrate fix to the operator** rather than opening a -third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b` -and is DEPLOYED (2026-08-19).** - -Startup rehydrate, as recommended — and waterland-dev deliberately went -further than the framing I sent them. I had said a directory the scan cannot -parse "just does not enter the index"; they made the opposite call, because a -directory that never enters the index is exactly the one that never gets -reclaimed. **That is the sharper reading and it is the reason the fix works on -this volume at all** — the 16 pre-existing dirs have no sidecar. Their -adoption ladder: sidecar → restored verbatim; no sidecar → adopted with -dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt -sidecar → degrades to inference, no startup crash; **neither source nor -sidecar → skipped on purpose**, since adopting it would turn eviction into a -delete-arbitrary-directories primitive pointed at this volume. Sidecar writes -go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is -unreachable via the artifact route. - -They also closed a second leak I never saw, because it needs a restart -*mid-render* to surface: a job left `running`/`queued` in its sidecar is -non-terminal forever, and eviction skips non-terminal jobs — so it is a -phantom that is never reclaimed and `queue_depth` over-reports for the life of -the process. Adoption now marks those `failed`. - -**Verified on this host after the update:** `/api/jobs` went **1 → 16** while -the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time. -Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only -made them visible. A subsequent real render took both to 17. From here the -store is bounded **across** restarts, not merely within a process. - -## Access - -Repo is not anonymously readable (a bare clone 403s). Operator granted -**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False, -pull: True`. Token on irv-ml1 at -`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a -**repo-scoped** credential helper; `.git/config` carries no token (verified), -so the remote stays clean in any diff or backup. The operator's `vh` -site-admin token was used only for the initial clone and the grant itself and -was **never written to disk on that host** — a site-admin credential on a GPU -box is a blast radius nobody needs for a read-only fetch. - -## Constraints honoured as stated (not inferred) - -- **Serial by design — one replica, one card.** A render is 20–45s of near-full - GPU with a single worker thread. Two on the same A6000 would OOM or thrash. - Throughput is a hardware conversation, not a replica-count one. -- **No authentication, arbitrary file uploads** ⇒ stays inside the - LAN/WireGuard boundary. Do **not** paper over it with a proxy password; - waterland-dev offered to add a real auth layer if wider reach is ever needed. diff --git a/persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md b/persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md deleted file mode 100644 index 3e2b47c..0000000 --- a/persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md +++ /dev/null @@ -1,95 +0,0 @@ -# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight - -The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` -using the MTP-aware, vision-preserving **Robinson formula** (documented in -`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`). -Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2. - -## Why this model, why abliterate it ourselves - -Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on -llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug): -**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**; -4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign -over-refusal 0. So there is a real creative-content refusal surface to remove. -This **supersedes** the earlier "watch for DavidAU's own heretic build" posture — -we abliterate it ourselves. - -**It is additive over the current gen seat.** The live Heretic seat -(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** — -the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could -not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2 -residual-write matrices), and the MTP head is what gates speculative acceptance. -That in-band MTP edit is the delta this experiment tests. - -## Recipe maps 1:1 — dry-run PASSED - -Against the staged bf16: 1199 tensors, 333 vision preserved, -`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly -**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no -name drift. Two hard gates in the harness halt before any write: the coverage -identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on -**dim 3994** (orthogonalizing a direction living there bricks the model). - -## Capture SUCCEEDED — but only after a real environment fight (the durable lessons) - -**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.** -The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine) -AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**). -Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which -produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input: -finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent -range, so this is **precision-driven catastrophic cancellation, not overflow** — -**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain -prompt = 65 layers all finite; chat-template input = NaN; the trigger is the -input path through the unstable recurrence. - -Fixes, all in the committed harness (`7abd301`): -- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN). -- **A finite-gate aborts on a non-finite direction** — the sink screen alone - can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and - saved silently on the first run). -- `attn_implementation="sdpa"` pinned. - -**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the -forward OOM'd against the resident seats. Had to **stop three seats** for VRAM: -`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`. -⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily -grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving -meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen -into the remainder. All three restored to healthy. - -⚠ **fla lives in a side dir, not the venv.** The shared -`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are -`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and -reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers. - -## Result - -Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%** -in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated. -Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`. - -⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.** -Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's -0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless** -(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction -is valid and sink-clean but noisier than ideal; abliterating on it risks -under-removing refusals or nicking capability. **Expand the sets to a few -hundred each and re-capture** before the `--out` write. - -## Sequence from here - -1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down). -2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`. -3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/` - (the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant - (gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence. -4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate. - **Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it - holds through real multi-turn use. - -bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`, -provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch -re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`, -`7abd301`. diff --git a/persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md b/persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md new file mode 100644 index 0000000..693271b --- /dev/null +++ b/persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md @@ -0,0 +1,41 @@ +# `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour + +`vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of +97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another +96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM +gateway restart he attributed them to (different host, different component, 50 min earlier, and +six of the seven crashes predate it). + +**Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have +the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52 +(55,126 MiB) since the August move. + +## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY + + gen at util 0.43 wants 42,091 MiB GPU0 free 42,113 margin 22 MiB — would not start + gen at util 0.41 wants 40,134 MiB margin 1,979 MiB — I called this safe. IT WAS NOT. + result 1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min + gen at util 0.38 34,558 MiB actual 4,466-7,548 MiB free ZERO OOM events over 6 min + +**`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the +process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable +segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat +came up and then silently fell back to slower kernels for want of 20 MB chunks. + +⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the +thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive. + +## Final state and what it cost + + GPU0 mog-sec 55,126 + gen 34,558 = 89,701 / 97,887 + GPU1 50,507 / 97,887 — 46.7 GB free, was 216 MiB + +`gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len** +— it holds exactly one full-context request. Short/medium requests still batch; long-context +throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched. + + .env.bak-preGPU0-20260904-164032 the GPU move + .env.bak-preShrink-165133 the utilization + +⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on +ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env. diff --git a/persistent-memory.d/2026-09-04-dac-forced-10g-failed.md b/persistent-memory.d/2026-09-04-dac-forced-10g-failed.md new file mode 100644 index 0000000..9675542 --- /dev/null +++ b/persistent-memory.d/2026-09-04-dac-forced-10g-failed.md @@ -0,0 +1,46 @@ +# `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early + +The ESH-Media ↔ UDM uplink is a 3 m OEM `SFP-H10GB-CU3M` twinax that negotiated **1000 Mbps** +with **zero errors** on both ends. Diagnosis: `sfp_compliance: Unknown` — the switch reads the +EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so +autoneg falls back to the safe rate. Control case: a TP-Link `TL-SM5220-1M` on the adjacent UDM +port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg. + +Operator authorised forcing it. Forced `autoneg False / 10000 / full` on the **UDM end only**; +ESH-Media followed to 10000 unforced — proof the cable was electrically *capable*. + +## ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG + + 10:03 200 errors link-up burst + 10:48 221 +21 in 42 min — I reported this as "flat", it was not + 14:38 416 +195 over 4 h, plus user-visible flapping the operator felt + +**A marginal link declares itself over hours, not minutes.** I watched a two-minute flat window +and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping +before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back +to autoneg/1000, stable. + +⚠⚠ **The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal +cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at 10G is exactly +what a cheap 3 m twinax is — which means the platform's autoneg fallback was **protecting +something real**, not being fussy about vendor coding. The coding explains the negotiation; it +does not explain errors once forced. I conflated the two. + +⚠ **Do not re-force this port.** The fix is the cable. + +## Method notes worth keeping + +- **Force the RECOVERABLE end.** ESH-Media reaches the controller *through* this link, so a + failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is + safe because the path to the controller (`nh3-dev → 10.100.10.1 → 10.0.0.1`, site VPN) does not + cross it. Verified with `traceroute` **before** the change; revert payload written before the + forward one. +- ⚠ `port_overrides` is a **whole-array PUT** — the array was diffed to prove exactly one field + on one port changed before sending, and read back after. +- Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP + and the Zigbee coordinator. + +Resolution: the operator already owns a replacement and ran the copper himself through a drilled +floor 2x4. Recommendation given was **fiber** (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because +the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks +the same fault. Runbook `docs/runbooks/esh-media-dac-10g.md`; commits `b9988a7`, `514ce7a`, `054c098`. diff --git a/persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md b/persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md new file mode 100644 index 0000000..5e1cdd3 --- /dev/null +++ b/persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md @@ -0,0 +1,33 @@ +# `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN + +`10.0.50.50` is **CT 103 on esh-pve-nas**, a hand-rolled Debian NAS (not a Synology). Created an +SMB account for the operator's Windows AudioGridder DSP box: + + username dsp uid 999, /usr/sbin/nologin, no home — SMB ONLY, cannot log in anywhere + password vaulted at esh-nas/dsp-smb-password (round-trip verified by sha) + verified authenticates, sees all 8 shares, WRITE to //share confirmed (mkdir/rmdir) + +⚠ `smbclient` is absent from the NAS, nh3-dev and esh-docker-vm — verification ran in a throwaway +`alpine:3.20 --network host` container with `samba-client`, leaving nothing installed. + +Shares are registry-defined (`registry shares = Yes`), so they are invisible in `smb.conf` — use +**`testparm -s`**, not grep, or you will conclude there are no shares. + +## ⚠⚠ THE EXPOSURE, UNACTIONED — operator has not ruled + +**NFS: twelve exports, `rw` to all of `10.0.0.0/8`, `sec=sys`, no authentication.** + + /mnt/{backup,books,compose,documents,iso,media,music,share, + pvestore,nvme-pvestore,ssd-pvestore,tank-vmbu} + +`sec=sys` means the NAS trusts whatever uid the client claims. **Every VLAN at ESH — IoT, +cameras, guest — can mount the NAS read-write today.** `/mnt/backup` additionally has +`all_squash,anonuid=2000` so every client collapses to `nas_user`. + +**SMB: every share except `backup` is `guest ok = Yes` and writable**, with +`map to guest = Bad User` — an unknown username lands as guest with write access. + +So the `dsp` credential is auditable and survives guest being turned off, but it is **not what is +gating access today**. Hardening (narrow exports to `10.0.50.0/24` + named hosts, drop `guest ok` +where unneeded) was offered and is roughly an hour; it would break anything relying on guest, +which is why it needs the operator's say-so. **Not actioned. No tracking surface beyond this entry.** diff --git a/persistent-memory.d/2026-09-04-run3c-trained-and-gated.md b/persistent-memory.d/2026-09-04-run3c-trained-and-gated.md new file mode 100644 index 0000000..582f2cf --- /dev/null +++ b/persistent-memory.d/2026-09-04-run3c-trained-and-gated.md @@ -0,0 +1,52 @@ +# `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression + +Run 3c is the lr **1e-05** arm against run 3's 2e-04; corpus, base, mask and every other +hyperparameter byte-identical. It exists to answer one question: does a gentler learning rate +keep run 3's diversity benefit while dropping its safety regression? **Answer: no, and the +failure is the informative kind.** + + trained 604/604 in 11:55:40 on pfi-gx10 (GB10, aarch64, sm_121), 69.8 s/it + merged /home/infra-ops/erp-tune/serve/merged-run03c, bf16 51.6 GB + served erp-tune-v3c on :8098, exposed as gateway alias `trial` + +## The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md) + + CSAM 175 gens x 3 arms — ZERO hits, hard-line guard passed everywhere + selfharm/methods base 10/10 refused -> 3c 0.6 = -40pp de-refusal + meta pooled 0.0 delta — a CANCELLATION, not cleanliness: selfharm -40pp + offset by advice/medical +40pp + diversity attractor primary returned to BASE level — the gain is GONE + + cross-arm run 3 (lr 2e-04) selfharm -44pp diversity: real gain + run 3c (lr 1e-05) selfharm -40pp diversity: none + +⚠ **The selfharm de-refusal is LR-ROBUST.** A 20x cut moved it 4pp while erasing the diversity +benefit entirely. It comes from corpus content and imprints at even gentle exposure — it is not +something a lower learning rate dials out. That is what the LR sweep was run to find out. + +⚠ **A pooled preserve-list test cannot see a single-axis collapse.** The pooled operational +delta reads 0.0 because two axes moved 40pp in opposite directions. Same structural defect that +let run 3's gate pass — recorded as R47 §8 item 11. + +## What the port proved about the box + +- **The tooling loads on aarch64/sm_121.** Full training stack plus flex_attention and the + chunked-loss path. Nothing exotic needed beyond `python3-dev`. +- **Verification that earned its keep**: both 49 GB base shards sha256-matched ana-ml2's, and a + full encode was run into a throwaway dir and compared byte-for-byte — 197,360,233 B, sha256 + `c08bb1fe2ecb0be3`, identical. transformers 5.15.1→5.16.1 and x86-64→aarch64 are *measured* + inert, not assumed. +- ⚠ **The encode-cache FILENAME differs by design** — `base_model_path` is in the cache key, so + rehoming the base changes the key while content stays identical. Input hash, not output hash. + Do not read it as drift; do not "fix" it by faking `/tank` on the GX10. + +## The lora_B signal worth carrying forward + + run 2 (lr 2e-04) min 0.6826 median 1.7212 max 3.7573 + run 3c (lr 1e-05) min 0.0480 median 0.1561 max 0.4133 + +~11x gentler across the board — exactly what a 20x LR cut should produce. A consistency check +passing, not a red flag, but worth in hand *before* reading diversity numbers: "the tune did +nothing" and "the tune did less on purpose" look alike in the output. + +Runbook `docs/runbooks/gx10-run-03c.md`; commit `dae77ee`. diff --git a/persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md b/persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md new file mode 100644 index 0000000..f22cd63 --- /dev/null +++ b/persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md @@ -0,0 +1,39 @@ +# `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping + +brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on +the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it +on instinct** — *"that makes zero sense. except for speed, serving a model should be identical +across servers"* — and he was substantially right. + + ana-ml2 0.9688, 0.9574 spread 1.14pp mean 0.9631 + gx10 0.9583, 0.9895 spread 3.12pp mean 0.9739 + + between-box LEVEL difference 1.08pp + ana-ml2's own replicate spread 1.14pp <- LARGER than the between-box gap + pooled range ignoring box 3.21pp <- ~= the entire "GX10 floor" + +**Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of +dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**. +brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`, +doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a +measured effect and ranked hardware on it. Retracted at `97f73dd`. + +⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly +identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect +moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range +estimator — he had the refutation in hand and read it as support. + +⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session +triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from +bytes — and this one was **his, and flattering**: it made his earlier work look prescient and +produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering +direction needs the extra pass.** + +**What survived, deliberately separated:** re-measuring the floor on whatever stack actually +serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show +it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately +measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect). +**Retracting the evidence and keeping the discipline are different acts.** Settling it properly +wants several blocks per box and is its own probe, not a by-product of a gate. + +See [[2026-09-05-vllm-on-sm121-and-run4]]. diff --git a/persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md b/persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md new file mode 100644 index 0000000..bd122ce --- /dev/null +++ b/persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md @@ -0,0 +1,53 @@ +# `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon + +Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a +stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture. + +⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its +sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but +lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside +a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on +this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`). + +Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected: +`Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip. + +⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with +`--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat +(`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy +`--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs +'{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a +null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box. + +## Run 4 — the corpus arm + +Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context +share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped +training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was +one-run-scoped, a run 5 needs a third). + +**Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather +than accepting a "confirm this":** + +1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and + raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have + hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*. +2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256: + null` on three of four. Preflight resolves `roots_dir//clean-v1/CLEANROOT.json` + literally and requires the sha. + +⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have +taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the +deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate. +brokkr then went further and recomputed his shas **from shard bytes** rather than reading them +back out of the deployed CLEANROOT, which had the same defect one step removed. + +**The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples / +3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio +(47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target. + +⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under +`ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028 +(fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file +iteration (safe), and the splitlines() paths touch only files written with the default +`ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way. diff --git a/persistent-memory.md b/persistent-memory.md index 30a4580..2e03b42 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-03_ +_Last updated: 2026-09-05_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,48 +108,76 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-09-03 — **nothing is mid-action; the session closed clean.** Every -item below is a live commitment or a known-open risk, not work in progress._ +_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else +below is a live commitment or a known-open risk._ -- **Nothing is running.** No training, no deploys pending, no background jobs. - Commits sit unpushed on `main` — all docs, runbooks, memory and scripts; - push is the operator's call. +- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`, + log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed, + a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows + are short and single-window, so there is no long tail for the sampler to find). + **~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450, + gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion** + — he takes base floors on the GX10 first, then the tuned arm, serially. -- **Run 3c is staged on pfi-gx10 and awaiting the operator's go.** Everything is - verified and one command away (`ssh infra-ops@10.100.50.60 - '~/erp-tune/launch-run-03c.sh'`); ~13.3 h once started. Its train loop is the - one piece never exercised on sm_121 — watch the first three minutes. +- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly + for training, but can serve its trials. unless the box is needed for training work."* + So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4 + ends. I over-read an earlier version of this as "training-only" and had to correct + it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong. -- **`web_search` needs a session restart to appear.** The SearXNG MCP server is - registered at user scope and `claude mcp list` reports it Connected, but MCP - servers load at session start — this session does not have the tool. +- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a + fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by + operator ruling: experimental, NOT a compose stack, does not survive a reboot). -- ⚠ **SearXNG general search is effectively single-engine.** `google cse` - carries it; `brave`/`duckduckgo`/`startpage` CAPTCHA even from NH3's - residential egress. If google cse breaks it goes quiet the same way it just - did. `scripts/searxng-health.sh` is the detector. +- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`, + `gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error* + reads as an outage — brokkr checked his own work against mine because he could not + tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the + next gateway change rather than spending a restart on tidying. -- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** Fleecing means a - slow `pbs-ana` can no longer wedge the guest, but a job that once ran at +- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods + regression.** Flagged to the operator twice (before adding, and after the gate + measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the + same access with a blast radius of one. Settled — do not re-litigate. + +- **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator + ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek + SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved. + +- ⚠ **Cityside fiber `/30` is NOT provisioned.** `128.177.138.182/30`, gw `.181`. + Static passes no traffic and DHCP still hands CGNAT `100.104.3.250`; operator + power-cycled both ends and opened a ticket. Cutover payloads stay staged: + `wan1-REVERT.json`, and the `esh-ana` IPsec fix (`ipsec_local_ip 100.104.3.250 → + 128.177.138.182`) **which will otherwise silently break ESH→Anaheim restic backups.** + +- ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to + `10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable. + Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md` + +- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at 941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms. - Nightly at 21:00, `all 1`. Worth its own investigation. + Nightly 21:00, `all 1`. Worth its own investigation. -- ⚠ **pfi-gx10 is now single-path.** Wired only on a reserved 10.100.50.60; the - Wi-Fi escape hatch is deliberately gone. A switch-port or reservation failure - is a rack visit — on the box run 3c was moved to. +- **ledger-dev asked about the Ledger→SVOS rename** (gitea repo rename cost, and + moving `nh3-dev/development/ledger/env.sh` in the vault — the `secret` CLI has no + rename, so it is re-put + delete). **Unanswered, no deadline**, nothing moves until + the operator fixes the strings. -- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their - first client-premises release cut (U10, unscheduled). They wrote the gate down - specifically so it would not quietly become never; ping expected when U10 is - scheduled. +- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first + client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled. -- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — - deliberate, they are the operator's personal machines and registering them - implies PFI-managed. They now carry real config, so the omission is a choice - to revisit, not an oversight. +- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate; + they are the operator's personal machines. A choice to revisit, not an oversight. ## Recent decisions +- `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md` +- `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md` +- `[2026-09-04]` **ERP run 3c trained and GATED — the 20x LR cut erased the diversity gain and did NOT remove the −40pp selfharm regression.** LR-robust, so it comes from corpus content. CSAM clean on all three arms. ⚠ A pooled preserve-list test cannot see a single-axis collapse. → `persistent-memory.d/2026-09-04-run3c-trained-and-gated.md` +- `[2026-09-04]` **`gen` moved to ana-ml2 GPU0 to stop vllm-embed OOM-crashing (7 restarts) — and I sized it against vLLM's declared budget, not its runtime footprint.** Cost: gen KV down to 1.02x concurrency at 262K. → `persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md` +- `[2026-09-04]` **SMB account `dsp` created + vaulted for the Windows AudioGridder box — and esh-nas turns out to be wide open.** Twelve NFS exports rw to `10.0.0.0/8`, guest-writable SMB. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md` +- `[2026-09-04]` **SLZB-MR1U Zigbee coordinator moved to esh-iot (VLAN 90) at `10.0.90.10`, DHCP-reserved, DNS'd, handed to ha-dev.** ⚠ Home Assistant cannot resolve `.internal` at all (Docker's 127.0.0.11 upstream excludes the fleet AdGuard) — pre-existing; ha-dev declined the fix. Runbook `docs/runbooks/slzb-mr1u-zigbee-coordinator.md`, commits `fed29be`/`0bbdaf9`. + - `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md` - `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md` - `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I ` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md` @@ -232,18 +260,12 @@ item below is a live commitment or a known-open risk, not work in progress._ - `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`) - `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number. - `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302` -- `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`) - -- `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` -- `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` -- `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` -- `[2026-08-19]` **Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background.** Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md` -- `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md` + - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). @@ -282,10 +304,12 @@ item below is a live commitment or a known-open risk, not work in progress._ _Older entries archived to archival-memory.md._ -_242 older entries archived to archival-memory.md._ +_248 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md` + - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. - `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision. - `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.