memory: snapshot — Cold-Fusion abliteration in flight, capture done
Captures the session's real work as the in-flight focus: abliterating DavidAU Cold-Fusion with the Robinson formula. fp32 capture succeeded (finite direction, layer 22, sink-clean) but two-template agreement is 0.59 vs Robinson's 0.99 — calibration-set expansion is the next step. New detail file records the full saga including the transformers/DeltaNet bf16-NaN fight (fp32 fix, the causal-conv1d kernel gap, the seat-restart VRAM-greed gotcha). Supersedes the earlier "watch for DavidAU's heretic build" posture — we abliterate it ourselves. Auto-archived 4 closed entries (Recent decisions: Booth-3-features 08-05, worldtree-sdk 07-31; Tried and abandoned: containerd-race 08-03, mv-rename 08-02) to archival-memory.md; the rest of the over-cap entries are held back by the <14-day and open-deferred guards. Index 331 -> 327.
This commit is contained in:
@@ -2495,3 +2495,20 @@ Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/l
|
||||
|
||||
- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
|
||||
_Archived 2026-08-19._
|
||||
|
||||
## Recent decisions (archived 2026-08-20 batch)
|
||||
|
||||
- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).
|
||||
_Archived 2026-08-20._
|
||||
|
||||
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||
_Archived 2026-08-20._
|
||||
|
||||
## Tried and abandoned (archived 2026-08-20 batch)
|
||||
|
||||
- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
|
||||
_Archived 2026-08-20._
|
||||
|
||||
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
|
||||
_Archived 2026-08-20._
|
||||
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
|
||||
|
||||
The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
|
||||
using the MTP-aware, vision-preserving **Robinson formula** (documented in
|
||||
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
|
||||
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
|
||||
|
||||
## Why this model, why abliterate it ourselves
|
||||
|
||||
Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
|
||||
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
|
||||
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
|
||||
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
|
||||
over-refusal 0. So there is a real creative-content refusal surface to remove.
|
||||
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
|
||||
we abliterate it ourselves.
|
||||
|
||||
**It is additive over the current gen seat.** The live Heretic seat
|
||||
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
|
||||
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
|
||||
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
|
||||
residual-write matrices), and the MTP head is what gates speculative acceptance.
|
||||
That in-band MTP edit is the delta this experiment tests.
|
||||
|
||||
## Recipe maps 1:1 — dry-run PASSED
|
||||
|
||||
Against the staged bf16: 1199 tensors, 333 vision preserved,
|
||||
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
|
||||
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
|
||||
name drift. Two hard gates in the harness halt before any write: the coverage
|
||||
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
|
||||
**dim 3994** (orthogonalizing a direction living there bricks the model).
|
||||
|
||||
## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
|
||||
|
||||
**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
|
||||
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
|
||||
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
|
||||
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
|
||||
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
|
||||
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
|
||||
range, so this is **precision-driven catastrophic cancellation, not overflow** —
|
||||
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
|
||||
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
|
||||
input path through the unstable recurrence.
|
||||
|
||||
Fixes, all in the committed harness (`7abd301`):
|
||||
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
|
||||
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
|
||||
can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
|
||||
saved silently on the first run).
|
||||
- `attn_implementation="sdpa"` pinned.
|
||||
|
||||
**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
|
||||
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
|
||||
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
|
||||
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
|
||||
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
|
||||
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
|
||||
into the remainder. All three restored to healthy.
|
||||
|
||||
⚠ **fla lives in a side dir, not the venv.** The shared
|
||||
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
|
||||
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
|
||||
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
|
||||
|
||||
## Result
|
||||
|
||||
Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
|
||||
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
|
||||
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
|
||||
|
||||
⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
|
||||
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
|
||||
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
|
||||
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
|
||||
is valid and sink-clean but noisier than ideal; abliterating on it risks
|
||||
under-removing refusals or nicking capability. **Expand the sets to a few
|
||||
hundred each and re-capture** before the `--out` write.
|
||||
|
||||
## Sequence from here
|
||||
|
||||
1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
|
||||
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
|
||||
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
|
||||
(the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
|
||||
(gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
|
||||
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
|
||||
**Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
|
||||
holds through real multi-turn use.
|
||||
|
||||
bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
|
||||
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
|
||||
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
|
||||
`7abd301`.
|
||||
+11
-13
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-08-19_
|
||||
_Last updated: 2026-08-20_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -114,7 +114,7 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
- **🔴→🟢 ESH OUTAGE 2026-08-19 — esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS.** Presented as "wifi/routing issues"; internet was healthy throughout. Cause: `esh-userland` (VLAN 10, the `PVC` SSID) handed out **one** resolver, `10.0.50.45` (AdGuard on `esh-docker-vm`) — cross-VLAN, no secondary — and its hypervisor died. **Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored.** Two fixes landed: gateway `10.0.10.1` added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and **`softdog` → `iTCO_wdt` hardware watchdog owned by systemd** (`playbooks/esh-pve-hardware-watchdog.yaml`, idempotent, verified armed) so a repeat self-recovers in 60s. **VM 102 pinned off** (`onboot: 0`) as the passthrough suspect. **⏳ OPEN:** (a) the watchdog is armed but **has not been proven to fire** — needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard **RJ45** is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel `6.8.12-16` rollback held in reserve if it freezes again. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md`
|
||||
|
||||
_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ~~⚠ 2 unread althing messages from waterland-dev left deliberately undrained~~ — **drained 2026-08-19 post-`/clear`; thread closed, see below.**_
|
||||
_As of 2026-08-20 — continued from the 08-19 infra session (DNS/.internal, waterland, homepage-theme, unmanaged-stacks — all landed). This session's adds, all committed: **waterland upstream fixes deployed** (`b72425b`, all 3 findings closed); **esh-pve froze 4.5h → hardware watchdog + secondary DNS** (SPOF closed); **Homepage theme REBUILT** on canonical Australis tokens (the Skyfall/Arbo version was ugly — pills/cards realigned, wallpaper killed, live at 10.0.50.45:5100); **Booth gained kept-boards (`.forever`) + a standing agent link board + inline .md/.txt rendering**; **fleet IPv6 plan settled** (endpoints not internal numbering; ESH has a /56); and **the Cold-Fusion abliteration** (above) — capture done, calibration expansion next. ⚠ MANY commits unpushed (through `7abd301`)._
|
||||
|
||||
- **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `<host>.<site>.internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
|
||||
@@ -150,7 +150,7 @@ _As of 2026-08-19 (late) — a long infra session, all of it landed and committe
|
||||
|
||||
- **🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor — **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** — `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`.
|
||||
|
||||
- **⏳ WATCHING: DavidAU's HERETIC build of Qwen3.8-27B — the one worth waiting for.** `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` examined 2026-08-17 and **NOT adopted**: it is a capability/efficiency finetune of **stock** Qwen3.8 and every bench row on its card is labelled **`[non heretic]`** — adopting it would reintroduce base refusals the current seat does not have. ⚠ **Easy to misread as uncensored** (operator did, and it is a fair mistake): DavidAU's back catalog is almost all `Uncensored-Heretic` builds — Fable-Fusion 711, Qwen3.5-9B Cold-Fusion — so the naming pattern implies it. This one simply has not had that stage run yet; the card's roadmap says the HERETIC version is **IN PROGRESS from base**. **That** is the release to watch, not this one. What makes it worth watching: third-party benches (Nightmedia, mxfp8) beat stock Qwen3.8 by **+0.064 arc/c, +0.056 arc/e, +0.056 obkqa**; claimed **MTP acceptance 55.7% (record 59.9%)** vs our measured 47.2%; **thinking tokens cut to 1/10–1/2**; PPL *dropped* vs base. Same GAIN/Cold-Fusion pipeline that produced Fable-Fusion 711, which we already serve on `char-rp-reasoning` — proven in-fleet, not just claimed. Structurally clean (1199 tensors, 15 mtp in shard 18, 333 visual). ⚠ MTP/speed figures are **GGUF/llama.cpp on a 5090, not vLLM** — may not transfer; and because its MTP head was likely *trained*, the free CPU-hash shortcut would NOT apply (it won't match base) so a real acceptance gate would be needed.
|
||||
- **🟣 COLD-FUSION ABLITERATION — IN PROGRESS (the real work; capture done, calibration-set expansion is next).** We are abliterating `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` ourselves with the **Robinson formula** (`docs/pfi/abliteration-recipe-qwen38.md`; harness `services/coldfusion-abliteration/`), rather than waiting for DavidAU's own heretic build — this **supersedes** the earlier "watch for the heretic version" posture. Justified by a **measured** stock refusal profile (~33% creative refusal, explicit-sexual + graphic-torture specifically; self-harm guardrails intact; 0 benign over-refusal). **Additive over the gen seat:** the live Heretic seat's MTP head is a byte-identical base graft (wrapper never loads it); Robinson abliterates the MTP head **in-band** — the delta this tests. **STATE:** bf16 staged on `ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`); dry-run PASSED 1:1 (131 tensors); **fp32 `--capture` PASSED** → finite, unit-normed direction at **layer 22**, sink-clean (0.0008% in dim 3994). ⚠ **Quality caveat: two-template agreement is 0.59 vs Robinson's 0.99 — the small 8/8 calibration set is the suspect, which is why NEXT = expand HARMFUL/HARMLESS to a few hundred each and re-capture** before the `--out` write. ⚠ The write re-stops three ana-ml2 seats (meromero, fablefusion, gen) for fp32 VRAM — batch re-capture + write in one window. Full saga incl. the transformers/DeltaNet bf16-NaN fight (fp32 fix, seat restart-order gotcha) → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md`
|
||||
|
||||
- **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; `GRUB_DEFAULT=0` means entry 0 is already the -42 entry with the correct `root=ZFS=nvme/ROOT/pve-1`, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (409M) — `zfs rollback -r <snap> && reboot`. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since `/boot` changed.** esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate).
|
||||
⚠⚠ **THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration.** The staging chroot did `mount --rbind /dev` + `/sys` with **no `--make-rslave`**; on systemd `/` is *shared*, so the cutover's `umount -R` **propagated back into the live host** and stripped the real `/sys/fs/cgroup`, `/dev/pts`, `/dev/shm`. logind could then create no sessions: ping fine, TCP fine, **SSH authenticates**, resident daemons keep serving (pveproxy returned clean 401s) — and **every new exec hangs, including `/sbin/reboot`**, so the reboot never ran. **It is a near-perfect impostor of failing root-disk I/O**, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. **Operator caught it** — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in `dmesg` all along: `[16.00] [sdq] Attached SCSI removable disk` (clean, no errors) and a last-line timestamp of **`[12114881]` = 140 days = the ORIGINAL boot** — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than *no* reboot. Recovered with **no console access** by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. **RULES: (1) always `--make-rslave` after `--rbind` (playbook now guards on `PROPAGATION != shared`); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get `dmesg` and check the boot timestamp.**
|
||||
@@ -165,10 +165,12 @@ _As of 2026-08-19 (late) — a long infra session, all of it landed and committe
|
||||
|
||||
- **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims).
|
||||
|
||||
- **eshpfi push state:** operator pushes **manually**; last push `50d13f5..8be8a51`. ⚠ **17 commits unpushed as of 2026-08-19** (`707a8cb`..`b8003c7`, plus this snapshot commit) — nothing is on the remote from this session.
|
||||
- **eshpfi push state:** operator pushes **manually**; last push `50d13f5..8be8a51`. ⚠ **MANY commits unpushed as of 2026-08-20** (`707a8cb`..`7abd301`, plus this snapshot commit) — nothing from the 08-19 or 08-20 sessions is on the remote.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md`
|
||||
|
||||
- `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md`
|
||||
|
||||
- `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
@@ -280,7 +282,6 @@ _As of 2026-08-19 (late) — a long infra session, all of it landed and committe
|
||||
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
|
||||
|
||||
|
||||
- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).
|
||||
|
||||
|
||||
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
|
||||
@@ -293,14 +294,13 @@ _As of 2026-08-19 (late) — a long infra session, all of it landed and committe
|
||||
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
|
||||
|
||||
|
||||
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
_206 older entries archived to archival-memory.md._
|
||||
_208 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -309,11 +309,6 @@ _206 older entries archived to archival-memory.md._
|
||||
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
|
||||
|
||||
|
||||
- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
|
||||
|
||||
|
||||
|
||||
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
|
||||
|
||||
|
||||
|
||||
@@ -326,4 +321,7 @@ _206 older entries archived to archival-memory.md._
|
||||
|
||||
|
||||
|
||||
_141 older entries archived to archival-memory.md._
|
||||
|
||||
|
||||
|
||||
_143 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user