diff --git a/persistent-memory.md b/persistent-memory.md index 2dfef86..1341cc7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -120,7 +120,7 @@ _As of 2026-08-24 (late) — a very long ops session. The homepage arc and the c - **🟢 Both Anaheim IPsec tunnels run AES-128.** NH3 245→**270 Mbit/s**, ESH 268→**304**. Ceiling is **the UDM's software AES-CBC, not the FortiGate**. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - **🟢 Scriberr LIVE** — ana-ml2 **GPU1** :8080, built locally, uid **10001**, needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` - **🟢 ESH DNS fixed at the IPv6 layer**; RDNSS **redirected** to AdGuard. ⚠ Proxmox bridges need `accept_ra=2`. Naming scheme lives in `docs/pfi/ipv6-naming-scheme.md` — **a convention, not memory state; never let a memory line be the only copy again.** → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` -- **🟢 SEAT MAP.** **`gen`** = Qwen3.8-27B-Uncensored NVFP4-mixed, GPU0 :8015, 7 aliases, UP. **`char-rp`** = Gemma-4 MoE NVFP4, GPU0 :8016, DOWN deliberately. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019, sharing GPU1 with Scriberr. +- **🟢 SEAT MAP.** ⚠ **ana-ml2 runs a vLLM VERSION SPREAD, not one version** — do not say "ana-ml2 runs X". Measured 2026-08-24: `gen` **0.27.2rc1.dev150** (`nightly-311b3513`), `mog-sec` **0.26.1rc1.dev1102** (`nightly-e9d1398d`), `rerank-a3`/`coder`/`reward`/`embed` **0.24.0**, char-rp + trainee-bench pinned **v0.26.0**. `v0.27.1` (tagged) and three nightlies sit on disk unused. **`gen`** = Qwen3.8-27B-Uncensored NVFP4-mixed, GPU0 :8015, 7 aliases, UP. **`char-rp`** = Gemma-4 MoE NVFP4, GPU0 :8016, DOWN deliberately. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019, sharing GPU1 with Scriberr. - **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Isolating experiment is **MTP k=3 on `e9d1398d`** — still not run. Operator ruling: degeneration lives in the **un-fixed vLLM**, not the weights; MTP-head hypothesis **retracted**. Both sightings n=1. - **🟢 ana-ml2 mounts `/mnt/smithy`** ro + soft, **NOT in fstab** — manual remount after reboot. `nconnect=8` approved but deliberately not applied. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` - **🟢 ESH IS DUAL-STACK**; v4 static is an unprovisioned Cityside ticket. **NH3 stays v6-off by explicit ruling.** @@ -130,7 +130,7 @@ _As of 2026-08-24 (late) — a very long ops session. The homepage arc and the c ## Recent decisions - `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md` -- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then; **ana-ml2 now runs vLLM 0.26.0, two minor versions on**. So: RETEST hot-swap LoRA on 0.26.0 before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation. +- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation. - `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`