vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — session stand-down; GX10 baselined, althing at 3.2.4

Browse Source 
Rewrote the in-flight section to reflect that nothing is running and the
operator stood the session down. The GX10 work is recorded as what it was
asked to be: a baseline for the box and a check that the tooling loads,
with the run-3c port scoped but explicitly declined.

Added index lines for four decisions that had detail files but no pointer:
the GX10 baseline at 79.35 s/it, the costing error the operator overruled,
the althing four-surface deploy finding, and the irv-ml1 GPU resident map.

Auto-archival fired at the soft cap and moved 17 entries dated on or
before 2026-08-18 to archival-memory.md, holding back 4 that carry open
deferred-work pointers. The index went from 410 lines to 296, mostly by
rewriting in-flight rather than by archiving -- the dated log was not what
made it long.
This commit is contained in:
vh
2026-09-02 01:23:19 -07:00
parent b039aa19e8
commit e39106bd03
6 changed files with 603 additions and 669 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+44 -157
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-01_
_Last updated: 2026-09-02_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,150 +108,53 @@ no longer deployed sidecars here. See Recent decisions.)
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
## Current state / in-flight
_As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._
_As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is
baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has
not arrived._
- **✅ pfi-gx10 BASELINED AND OUTFITTED (2026-09-02) — bare metal, training stack verified.**
`uv` + torch 2.14.0+cu130 + transformers/peft/trl/datasets/bitsandbytes all clean on aarch64;
nvcc, NVIDIA Container Toolkit, docker group, NOPASSWD sudo. **Baseline: ~79.35 s/it** on the
run-3c shape (seq 16384, batch 2, LoRA r64, flex_attention, harness chunked loss) -> 604 steps
≈ 13.3 h vs ana-ml2's 2.2-2.7 h. ⚠ **6x slower where compute predicts 2.7x** — likely memory
bandwidth; **capacity box, not throughput box.** Port NOT done (operator stood it down; the
baseline was the goal). Still on Wi-Fi at `10.100.10.226`, not racked.
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **▶ (superseded) pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered.
**The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and*
**run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight.
**First move is a throughput probe, not a port**: ARM64 torch, load the base, ten steps,
report s/it — that decides whether 604 steps is an overnight run or unusable. Racking is
one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`).
⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns.
✅ **Liveness confirmed 2026-09-01 23:09** — up 11h48m, idle, 118/121 GB free, 822 GB disk
free, still bare (no torch/nvcc/uv), still on Wi-Fi at `10.100.10.226`, wired NIC has no
cable. ⚠ **Racking is NOT a prerequisite** — the Wi-Fi 7 link measures **67 MB/s over SSH**
(49 GB base in ~12 min), so the throughput probe can run from the desk today.
⚠ `nvidia-smi` FB Memory `N/A` is CORRECT for GB10 unified memory, not a fault.
✅ **OUTFITTED 2026-09-01 (userspace)** — **RULING: BARE METAL, not Proxmox** (no aarch64 PVE
build; the GPU is on-package and cache-coherent over NVLink-C2C so passthrough would partition
the unified memory that is the whole point; fleet GPU hosts are all bare metal).
Installed `uv` 0.12.9 + `~/ml/.venv` with torch **2.14.0+cu130** and the full training stack
(transformers/accelerate/peft/trl/datasets/bitsandbytes) — **all import clean on aarch64**.
**CUDA WORKS: sm_121, 121.6 GiB addressable, ~93 TFLOP/s bf16** (A6000-class speed, 2.5x the
memory). ⚠ **sm_121 is NOT in torch's arch list** — everything JITs from sm_120 PTX, so
**first use of every kernel pays a JIT compile** and cubin-only libraries will fail.
⚠⚠ **ALWAYS WARM UP BEFORE TIMING** — an unwarmed benchmark read 27 TFLOP/s (it was timing the
JIT) and nearly became a phantom "tensor cores are broken" report.
✅ **PRIVILEGED HALF DONE 2026-09-01** — nvcc, NVIDIA Container Toolkit, docker group; playbook
`gx10-outfit-privileged.yaml` green 5/5 (a CUDA container sees the GB10). **Bare metal, fully
outfitted.** ⚠⚠ **I was NEVER blocked** — infra-ops had NOPASSWD sudo all along; I probed with
**`sudo -n -v`, which is WRONG on sudo >= 1.9.15** (it refreshes a timestamp NOPASSWD never
creates) and asked the operator for a password that did not exist to need. **Always
`sudo -n true`.** Fixed in `scripts/elway` (same probe, would have broken fleet-wide as hosts
pass sudo 1.9.13) and in auto-memory. Password rotated + vaulted at
`pfi-gx10/infra-ops-password` (kept not locked: **no iDRAC on this box**, so it is the only
console break-glass). ⚠ Docker's buildkit DB was **already corrupt** — the restart surfaced it,
did not cause it; cleared `/var/lib/docker/buildkit` (cache, 0 images/containers).
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
(the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it
receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout
can). Everything else waits
on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
⚠ ~1 kW loaded — same class as the box that tripped the breaker; pick the NH3 circuit before
racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json`
validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
without deciding that first. Exactly TWO 3c launches, only one died.
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- **🟢 ALTHING 3.2.0 DEPLOYED + MIGRATION COMPLETE on nh3-dev (2026-09-01 22:13-22:26).**
forseti authored, infra-ops deployed; **5/5 pane routes pinned**, all four affected seats
re-declared within ~12 min of individual notice, log clean. Pane-poking replaces detached
waiters because **CC 2.1.257 reaps detached tasks when the seat sits in `shell`**.
⚠ **`uv tool install .` is a SILENT NO-OP on reinstall** — matches the source *spec*, not
contents; exits 0 having done nothing. **Always `--force` from a local path.**
⚠ **The herald's exclusion reason ("process gone, or pid reused") is FALSE for the migration
case** — all four declaring processes were alive 4 days, no pid wrap; real cause is the route
predating the guard fields. Don't debug a dead agent that isn't.
⚠ **My miss:** I called non-Claude pane delivery an "open risk" when forseti had personally
measured it on Aug 28 — didn't check whether it was already settled before calling it open.
Delivery now confirmed on `claude`, `pi` and `grok` seats.
⚠ One-tick latency after `declare` reads as failure; `--discover-pid` on a non-Claude process
tree remains untested.
🔴 **3.2.1 SAME NIGHT (22:40)** — 3.2.0 wrote pokes into panes people were **typing in**
(append + Enter submits their half-written message; hit the operator within the hour). Guard 4
pins `idle_cursor` and stays silent when the column moved. ⚠ **2 of 5 pane seats CANNOT be
guarded** — `kimi` and `pi` report no cursor at all, so bil- and regin-smithy-dev stay
permanently exposed; "3.2.1 fixes it" is true only where the cursor is legible. Mitigation for
those = `althing-route release` (pull-only). ⚠ Declare with an EMPTY input line or the pin
captures a composing cursor. **Outcome:** delivery verified on **four TUI families**
(claude/codex 3, grok 7, kimi/pi none). The two unguardable seats split — **regin-smithy-dev
RELEASED (now pull-only, operator composes in that pane routinely); bil-smithy-dev KEPT**
(pane is poke-driven, and pull-only had already cost them a notice unread for days).
⚠ **Deciding variable is WHO COMPOSES in the pane, not risk appetite** — a flat rule either
way would be wrong for one of them. ⚠ **Path to closing the hole (regin's):** a moment-check
reading pane CONTENT (input-line length) rather than cursor COLUMN would cover pi/kimi —
column is a proxy, empty-input-line is the real predicate.
✅ **3.2.2 deployed 22:51** — closes 4 of 5: `--discover-pid` was a **BUG not an unknown**
(matched `comm=="claude"`, so 4 of 5 pane seats could never have used it; now matches
`pane_command`), exclusion message split rather than softened, `--force` in the runbook,
latency + status-not-durable in the skill. 🔵 **PENDING OPERATOR:** the kimi/pi seats guard 4
cannot cover — poke unguarded / write-without-Enter / pull-only. **Recommended per-seat
status quo**; write-without-Enter beats it for neither seat.
✅ **DEPLOY IS NOW ONE COMMAND: `scripts/deploy-althing.sh`** (`--check` reports drift),
documented in `docs/runbooks/althing-deploy.md`. **FOUR surfaces**: tool install (⚠ `--force`
or it is a silent no-op), herald restart, skill sync, **plugin reinstall** — the fourth had no
runbook step and drifted 5 days, costing the SessionStart auto-declare hook entirely.
⚠ **eshpfi owns the plugin deploy, NOT the althing repo** — it targets per-machine paths, and
althing's `sync_skill.sh` deliberately reaches into no other tree; that boundary is correct.
→ `persistent-memory.d/2026-09-01-althing-320-deploy.md`
- **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone
stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid,
HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** —
ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against
FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask
to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB
baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two
incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind
a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both
are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose
nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are
**512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count
from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking.
→ `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
`/storetank`, so (b) was unavailable during the transition, not structurally).
**TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order
or the pool lands at ~84%; (2) retain vs reclaim irv-ml1's `/storetank` after the move —
RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the
stack; and under a no-spend striped destination it is the only redundancy). comfy-dev also
found ~215 GiB of unreferenced weights and offers a per-file prune audit — **explicitly NOT
gating**, because pruning all of it still lands the as-bought mirror at 72%. ✅ **H3 encoder
pin RESOLVED** — infra-ops' nvfp4-vs-int8 question had the direction backwards (pinned on
Ampere sm_86; Ada sm_89 is a strict superset, so the move cannot break it). Pin is about
**VRAM not architecture**; stands. The 26 GB int8 is a **spare, not an orphan** — do not
delete.
→ `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
**TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills
all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB
at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim
irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a
from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered
and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps.
⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive
each way** with **13 hosts dark including three SureFire CLIENT machines**, which no
`save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not
executed** — operator stood it down.
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
"trust the HTTP status, not `error_code`" still applies on running instances.
- **🟢 Matrix upgraded and hardened.** Synapse v1.159.0, Element v1.12.27, `/_synapse/admin`
LAN-only, appservice namespace opened, alias convention ratified. Miranda provisioned; her
summons reached the operator's watch. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
- **🟢 phasefinal.com LIVE** behind Cloudflare with edge caching + Always Online; apex 301s to
www. `inquiry@` alias added by the operator. Stack `stacks/phasefinal-web/`.
- **🟢 char-rp restored** on ana-ml2 **GPU1** (was dead 7 days on a quant-flag/checkpoint
mismatch crash-loop). ⚠ **GPU1 is now at 96,384 / 97,887 MiB — ~1.5 GB free, six tenants.**
GPU0's ~44 GB free is a DELIBERATE scratch reserve, not headroom to reclaim.
- **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD`
from a `.env` — until then `stacks/synapse/` and the live file have DIVERGED and
`deploy-stack.sh` must not be used on it (README says so).
- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design.
→ `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
from the vault rather than an inline value.
- **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart
hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether
their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because
they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render
agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that
evaporates the moment ComfyUI reloads.
## Recent decisions
- `[2026-09-02]` **pfi-gx10 BASELINED: ~79.35 s/it on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 7 timed steps, 0.19% spread, 86/121 GB. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md`
- `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
@@ -339,40 +242,24 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
- `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
- `[2026-08-17]` **irv-ml1 cleared of 782 GB, and Homepage brought under version control.** One dead-looking Gradio app pinned three delete targets at once; `/opt/ComfyUI` is NOT the ComfyUI that serves. → `persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md`
- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates one /64 PER REQUEST** (`2600:1700:b25:c11f::/64`) — and the BGW holds the whole `/60`, rationing `c118`–`c11f` one at a time while keeping `c110`–`c117`. So eight /64s exist; UniFi just solicits once. ⛔ **CLOSED 2026-08-24 — operator ruling, do not re-raise:** the BGW has **no IP-passthrough** (operator confirmed, and we have admin on it), so the only route to the other seven is a multi-DUID DHCPv6 client on a VM — which means split-stack routing and rebuilding the entire v6 firewall policy off the UDM. Juice not worth the squeeze. NH3 LANs stay v6-off. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
- `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
- `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
- `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`.
- `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
- `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
- `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
- `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
- `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
- `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
@@ -388,9 +275,10 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
_223 older entries archived to archival-memory.md._
_Older entries archived to archival-memory.md._
_239 older entries archived to archival-memory.md._
## Tried and abandoned
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
@@ -402,8 +290,7 @@ _Older entries archived to archival-memory.md._
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
- `[2026-08-15]` **Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless `re:^mtp.*` is in the quant-config `ignore`; and W4A16=Marlin (not native FP4) costs ~20% even on decode.** Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
_143 older entries archived to archival-memory.md._
_144 older entries archived to archival-memory.md._