memory: snapshot — session stand-down; GX10 baselined, althing at 3.2.4
Rewrote the in-flight section to reflect that nothing is running and the operator stood the session down. The GX10 work is recorded as what it was asked to be: a baseline for the box and a check that the tooling loads, with the run-3c port scoped but explicitly declined. Added index lines for four decisions that had detail files but no pointer: the GX10 baseline at 79.35 s/it, the costing error the operator overruled, the althing four-surface deploy finding, and the irv-ml1 GPU resident map. Auto-archival fired at the soft cap and moved 17 entries dated on or before 2026-08-18 to archival-memory.md, holding back 4 that carry open deferred-work pointers. The index went from 410 lines to 296, mostly by rewriting in-flight rather than by archiving -- the dated log was not what made it long.
This commit is contained in:
@@ -3,6 +3,511 @@
|
||||
_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._
|
||||
|
||||
## Recent decisions (archived)
|
||||
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
|
||||
|
||||
## The finding
|
||||
|
||||
`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
|
||||
Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
|
||||
carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
|
||||
(571 MB free)**.
|
||||
|
||||
⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
|
||||
SLC/pSLC with a real controller, so the **284 GB written since boot is
|
||||
unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
|
||||
that was wrong and the correction matters. What actually justifies the work:
|
||||
|
||||
1. **It is on the USB bus** — a reset or re-enumeration drops the *root
|
||||
filesystem* out from under a running hypervisor whose guests keep executing.
|
||||
NAND quality is irrelevant to that.
|
||||
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
|
||||
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
|
||||
4. **It has blocked patching for months** — the operator-visible symptom and the
|
||||
real urgency.
|
||||
|
||||
## The patching blockage (measured)
|
||||
|
||||
`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
|
||||
Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
|
||||
`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
|
||||
because it cannot take a kernel.
|
||||
|
||||
⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
|
||||
includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
|
||||
landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
|
||||
glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
|
||||
and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
|
||||
patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
|
||||
keeps downloads off root, but the kernel still lands in `/boot`.
|
||||
|
||||
## Mitigation applied 2026-08-17 — root 90% → 76%
|
||||
|
||||
| step | effect |
|
||||
|---|---|
|
||||
| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
|
||||
| vacuumed the journal | **freed 446 MB** |
|
||||
| `apt-get clean` | 79 MB |
|
||||
| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
|
||||
| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
|
||||
|
||||
571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
|
||||
through the ZFS-backed journal.
|
||||
|
||||
⚠ **Stopping journald over SSH kills your own session** — it takes the
|
||||
connection's logging path with it. The first attempt died mid-swap, leaving the
|
||||
dataset staged and the move incomplete (host was never at risk; journald
|
||||
socket-activated straight back). Redo as a detached `systemd-run` transient unit.
|
||||
Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
|
||||
|
||||
Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
|
||||
the wear argument no longer justifies touching a service `pvestatd` depends on.
|
||||
|
||||
## The plan — split boot from root (operator's proposal, strictly better)
|
||||
|
||||
My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
|
||||
proposed keeping boot on the DOM with a fallback image and putting all its files
|
||||
on ZFS. That is better and I should have gotten there myself** — I had assumed
|
||||
boot and root must share a device.
|
||||
|
||||
| | device | contents | written when |
|
||||
|---|---|---|---|
|
||||
| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
|
||||
| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
|
||||
|
||||
Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
|
||||
ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
|
||||
`zstd_compress` enabled and **GRUB cannot read those**.
|
||||
|
||||
**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
|
||||
`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
|
||||
day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
|
||||
the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
|
||||
mid-run no longer kills the running system. Free upside: boot environments
|
||||
(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
|
||||
|
||||
**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
|
||||
**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
|
||||
initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
|
||||
(leave it on the DOM LV — **never** swap on a zvol).
|
||||
|
||||
**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
|
||||
root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
|
||||
import by scan today, which is a coin-flip when the initramfs must find root.
|
||||
Set the cachefile before rebuilding the initramfs.
|
||||
|
||||
Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
|
||||
mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
|
||||
updates.
|
||||
|
||||
## ⚠ Blast radius — the gating constraint, invisible from the host itself
|
||||
|
||||
**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
|
||||
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
|
||||
|
||||
- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
|
||||
- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
|
||||
|
||||
Known incident shape — the only remedy for esh-docker-vm's D-state is a host
|
||||
reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
|
||||
risks corruption under `soft`. Quiesce both before any reboot of this host.
|
||||
Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
|
||||
|
||||
## Also identified
|
||||
|
||||
- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
|
||||
being CT 109 on pfi-pve.
|
||||
- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
|
||||
passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
|
||||
and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
|
||||
That resolves the dashboard's long-standing "role TBC". Retire rather than
|
||||
migrate.
|
||||
- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
|
||||
|
||||
## Staging executed 2026-08-18 — everything but the reboot
|
||||
|
||||
Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
|
||||
`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
|
||||
4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
|
||||
(ZFS initramfs, grub.cfg, both menu entries, grubenv).
|
||||
|
||||
**`grub-install` is deliberately NOT run.** The ESP stub still points at the old
|
||||
`/boot` inside the ext4 root, so the host's boot path is byte-identical to the
|
||||
last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
|
||||
`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
|
||||
|
||||
Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
|
||||
(new) + `pve-swap` 256 M (was 768 M).
|
||||
|
||||
### The three landmines staging found
|
||||
|
||||
1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
|
||||
mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
|
||||
which costs the "one reboot" property the design rests on. Only live source
|
||||
was the swap LV. Operator chose shrink-to-256M over drop-entirely.
|
||||
2. **The one-pool cachefile would have broken the NAS.** `zpool set
|
||||
cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
|
||||
cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
|
||||
(verified: scan active, cache inactive beforehand), so a cache holding only
|
||||
`nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
|
||||
bind mounts spanning all three pools. Every export would come up empty and
|
||||
both `hard` NFS clients would hang.
|
||||
3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
|
||||
Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
|
||||
`grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
|
||||
cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
|
||||
probe failure is swallowed by `2>/dev/null || true`. The same feature set
|
||||
that forced `/boot` to stay ext4 also corrupts the kernel command line, which
|
||||
the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
|
||||
drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
|
||||
`pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
|
||||
derived from pool member device paths and would shift if the mirror changed.
|
||||
|
||||
**The transferable lesson from (3):** the original verify grepped for
|
||||
`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
|
||||
was added that grep passes — while pool-less entries sit in the menu untouched.
|
||||
The check that holds walks every `linux` line, takes the **last** `root=`, and
|
||||
asserts it against a known-good set. **Assert the effective value, not the
|
||||
presence of a substring.**
|
||||
|
||||
### One-shot boot, not a new default
|
||||
|
||||
`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
|
||||
`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
|
||||
returns to ext4 by itself on the next reboot — no console, no hands. That matters
|
||||
more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
|
||||
and the NFS clients hang rather than fail. Only after a second clean ZFS boot
|
||||
should the saved default move.
|
||||
|
||||
### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
|
||||
|
||||
- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
|
||||
compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
|
||||
LV was live during the read, so a restore replays the ext4 journal. Not
|
||||
fixable with an LVM snapshot: the VG has no free extents.
|
||||
- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
|
||||
644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
|
||||
boot-chain artifact; the dd image is the belt-and-braces full-device restore.
|
||||
- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
|
||||
grub.cfg, fstab, LVM/ZFS/blkid state.
|
||||
⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
|
||||
file` redirects only the last command, so `pvs`/`vgs` output leaked into the
|
||||
tar stream on stdout. Group with `{ …; } > file`.
|
||||
|
||||
Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
|
||||
`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
|
||||
sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
|
||||
captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
|
||||
every guest config. **The newest on-disk copy before this was June 2024.**
|
||||
Commits `2275e11`, `3e31175`, `8ddc87c`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
|
||||
|
||||
Two unrelated housekeeping jobs from the same session, both with durable lessons.
|
||||
|
||||
## irv-ml1 — 782 GB reclaimed
|
||||
|
||||
Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
|
||||
**74%** (477 G).
|
||||
|
||||
**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
|
||||
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
|
||||
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
|
||||
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
|
||||
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
|
||||
nobody opened that shed for 15 months. Re-verified the consumer check immediately
|
||||
before deleting, not just during the audit.
|
||||
|
||||
**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
|
||||
`modelscope` 29 G (mtime **2024-04-23**).
|
||||
|
||||
**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
|
||||
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
|
||||
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
|
||||
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
|
||||
|
||||
### The lesson: one dead-looking app pinned three delete targets
|
||||
|
||||
`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
|
||||
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
|
||||
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
|
||||
would have half-broken a live listener in a way that surfaces only at its next
|
||||
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
|
||||
verified :7860 released and handles at zero, *then* deleted.
|
||||
|
||||
⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
|
||||
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
|
||||
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
|
||||
from uv. Checking open handles **per path** is what separated them — the earlier
|
||||
"not running" read would have deleted the wrong thing.
|
||||
|
||||
⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
|
||||
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
|
||||
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
|
||||
|
||||
All 16 containers and both systemd services verified healthy afterward.
|
||||
|
||||
## Homepage under version control
|
||||
|
||||
`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
|
||||
config lived only on the host. Its version history was **six hand-rolled
|
||||
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
|
||||
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
|
||||
105 cards across 19 groups, no empty groups.
|
||||
|
||||
⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
|
||||
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
|
||||
the top of the file. All five engines were discovering correctly the whole time.
|
||||
|
||||
**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
|
||||
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
|
||||
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
|
||||
dropped the `UltraSeedbox` layout group (nothing provides it).
|
||||
|
||||
⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
|
||||
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
|
||||
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
|
||||
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
|
||||
|
||||
⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
|
||||
change gave two false readings (a stale prerender, then an empty page).
|
||||
`GET /api/services` is the honest instrument, and config changes need a
|
||||
**recreate**, not a restart (a restart keeps the cached render in the writable
|
||||
layer).
|
||||
|
||||
⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
|
||||
also removed a host-side `README.md` in the conf dir. Content survived (it is now
|
||||
in the repo README) but that was a side effect, not a plan.
|
||||
|
||||
Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
|
||||
|
||||
Written because the operator expects to reference this "before too long" — the
|
||||
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
|
||||
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
|
||||
a nice-to-have.
|
||||
|
||||
## Why IPv6 suddenly matters: CGNAT at ESH
|
||||
|
||||
New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
|
||||
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
|
||||
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
|
||||
a global v6 address on each UDM restores a routable endpoint pair without
|
||||
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
|
||||
most likely first consumer of fleet IPv6.
|
||||
|
||||
Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
|
||||
|
||||
## The topology — as VERIFIED, not as assumed
|
||||
|
||||
Three transports, three different technologies. Do not describe this as "a
|
||||
WireGuard mesh"; a prior session did and was corrected.
|
||||
|
||||
| Link | Transport | Evidence |
|
||||
|---|---|---|
|
||||
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
|
||||
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
|
||||
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
|
||||
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
|
||||
|
||||
**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
|
||||
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
|
||||
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
|
||||
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
|
||||
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
|
||||
|
||||
**The FortiGate never terminates WireGuard — it port-forwards to the host that
|
||||
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
|
||||
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
|
||||
chasing a non-problem. Do not re-derive this.
|
||||
|
||||
## Per-site IPv6 state (2026-08-17)
|
||||
|
||||
| Site | Edge | IPv6 |
|
||||
|---|---|---|
|
||||
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
|
||||
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
|
||||
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
|
||||
|
||||
## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
|
||||
|
||||
`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
|
||||
|
||||
The proof matters because the naive read is wrong: the WAN sits at `c110::48`
|
||||
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
|
||||
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
|
||||
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
|
||||
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
|
||||
settings yielding the same /64 is the signature of a single-/64 delegation.
|
||||
|
||||
**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
|
||||
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
|
||||
routable address on the *WAN*, not a LAN prefix.
|
||||
|
||||
The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
|
||||
with no size field alongside), so the prefix-ID test is the only read-only-ish way
|
||||
to establish it from the API.
|
||||
|
||||
## What a v6 mesh actually requires (and what it does NOT)
|
||||
|
||||
**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
|
||||
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
|
||||
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
|
||||
then reverted.
|
||||
|
||||
**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
|
||||
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
|
||||
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
|
||||
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
|
||||
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
|
||||
|
||||
**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
|
||||
one field to flip when wanted.
|
||||
|
||||
**Inbound v6 is default-deny and that held without intervention.** The UDM runs
|
||||
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
|
||||
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
|
||||
rules exist," which is wrong and alarming. Use
|
||||
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
|
||||
for both families with `Allow Return Traffic`; the only v6-specific allows are
|
||||
link-local plumbing (ND solicit/advert, RA, DHCPv6).
|
||||
|
||||
## The stability problem — design around it up front
|
||||
|
||||
All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
|
||||
not a static assignment). A three-way mesh where every node can move is fragile;
|
||||
WireGuard tolerates one roaming end, not all of them.
|
||||
|
||||
The fleet already solves this on the v4 side — IPsec peers use **hostnames**
|
||||
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
|
||||
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
|
||||
Cloudflare DNS-edit token, so this is self-serve.
|
||||
|
||||
## Access recipes (cost a prior session real time)
|
||||
|
||||
- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
|
||||
`unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
|
||||
Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
|
||||
Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
|
||||
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
|
||||
(`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
|
||||
`root@10.250.50.252`; believe it.
|
||||
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
|
||||
password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
|
||||
`drain()` hangs the session**; read until the `ana-gw #` prompt and answer
|
||||
`--More--` with a space. Two invocations timed out at 3 min before this was fixed.
|
||||
|
||||
## Changes made and reverted this session
|
||||
|
||||
- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
|
||||
on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
|
||||
verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
|
||||
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
|
||||
private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
|
||||
(client configs carry private keys) were all mode **644**. Now **600**, and
|
||||
`keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
|
||||
WireGuard holds keys in kernel memory, so no restart was needed. The parent
|
||||
`/etc/wireguard` was already 700, which capped the real exposure to root-capable
|
||||
contexts inside the LXC — but the modes were still wrong.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
|
||||
## per-REQUEST truth, not the total — eight /64s exist and are unclaimed
|
||||
|
||||
The section above concludes AT&T hands out a single `/64` and that the
|
||||
`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
|
||||
itself superseded later in the same session, and the correction never made it
|
||||
into memory** — it survived only in the session transcript, and was recovered
|
||||
2026-08-24 while assessing a proposal to grab more prefixes.
|
||||
|
||||
Reading the **BGW's own LAN statistics page** gave the whole picture:
|
||||
|
||||
```
|
||||
BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix
|
||||
BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself
|
||||
Delegated 2600:1700:b25:c11f::/64 <- what the UDM got
|
||||
```
|
||||
|
||||
**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
|
||||
delegating from the top down — the UDM got `c11f`, the last one. So
|
||||
`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
|
||||
sitting unclaimed.
|
||||
|
||||
Both observations are compatible, which is why the first one looked conclusive:
|
||||
the prefix-ID test only carves *within* a delegation already held, so a UDM
|
||||
holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
|
||||
issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.
|
||||
|
||||
**Consequence — the ceiling is the requester, not the carrier.** More prefixes
|
||||
need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
|
||||
UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
|
||||
is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
|
||||
looks like BGW rationing rather than a provisioning-profile limit.
|
||||
|
||||
Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
|
||||
2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.
|
||||
|
||||
### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise
|
||||
|
||||
The seven unclaimed `/64`s stay unclaimed. Two facts close it:
|
||||
|
||||
- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
|
||||
on it — so the cheap path (let the UDM take the `/60` directly and carve it
|
||||
natively, as it already does at ESH) does not exist here.
|
||||
- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
|
||||
requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
|
||||
(UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
|
||||
firewall policy in nftables on that VM**, because routing v6 around the UDM
|
||||
bypasses its zone firewall entirely and would leave every LAN host globally
|
||||
reachable.
|
||||
|
||||
Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
|
||||
Do not re-propose on the strength of "there are seven free prefixes" — the
|
||||
prefixes are real, the firewall rebuild is why nobody wants them.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
|
||||
_Archived 2026-09-02._
|
||||
|
||||
- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
|
||||
_Archived 2026-09-02._
|
||||
|
||||
|
||||
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
|
||||
_Archived 2026-09-01._
|
||||
@@ -914,6 +1419,60 @@ _Archived 2026-08-27._
|
||||
_Archived 2026-08-24._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
|
||||
|
||||
**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
|
||||
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
|
||||
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
|
||||
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
|
||||
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
|
||||
|
||||
**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
|
||||
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
|
||||
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
|
||||
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
|
||||
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
|
||||
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
|
||||
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
|
||||
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
|
||||
`reference_abliteration_mtp_lessons`.
|
||||
|
||||
**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
|
||||
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
|
||||
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
|
||||
(`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
|
||||
+ `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
|
||||
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
|
||||
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
|
||||
sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
|
||||
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
|
||||
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
|
||||
`--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
|
||||
|
||||
**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
|
||||
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
|
||||
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
|
||||
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
|
||||
no cap OOM'd — cudagraph captured to batch-512).
|
||||
|
||||
**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
|
||||
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
|
||||
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
|
||||
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
|
||||
|
||||
**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
|
||||
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
|
||||
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
|
||||
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
|
||||
|
||||
**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
|
||||
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
|
||||
isolate before deleting.**
|
||||
|
||||
Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
|
||||
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
|
||||
_Archived 2026-09-02._
|
||||
|
||||
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
`container: image: debian:bookworm-slim` — fails:
|
||||
|
||||
Reference in New Issue
Block a user