diff --git a/archival-memory.md b/archival-memory.md index aee5892..a6ff2ad 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -4,6 +4,181 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re ## Recent decisions (archived) +- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.** + +**The failure.** The ana-docker instance answered `/healthz` every 30s, showed +`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while +returning **0 results for every query tested**. It was running **2026.4.17 +against a current 2026.9.3**: 4.5 months of engine scrapers rotting against +sites that had changed their markup. SearXNG ships near-daily releases for +exactly this reason. + +⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A +container created in April on `:latest` is pinned to April forever. + +⚠ **`/healthz` proves the web app answers and says NOTHING about whether search +works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that +was a stalled backup, the runner audit that trusted liveness for identity, and +the statusline bell that measured a mechanism. + +**Proven before acting**: the new image, same settings file, same host, same +query, in a throwaway container → **20 results where the running one returned +0**. Network was ruled out first — from inside the container DNS resolved and +mojeek/wikipedia were reachable, so engines were reachable and the parsers were +the broken part. + +**Why NH3 and not an in-place update** (operator's call, and the measurement +backs it): + + ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA + nh3-docker egress 70.230.226.88 residential -> not gated the same way + +Search engines gate datacenter ranges. Same reason the fleet keeps a residential +SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied +around. `outgoing.proxies` has the fallback commented in place if NH3's egress +ever changes. + +⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from +NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`, +`github`, `stackoverflow`, `marginalia` work. **General search is effectively +single-engine** — if google cse breaks, it goes quiet again. + +**Two config defects, both silent:** `base_url` still named +`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins, +so nothing broke and the file lied to every reader); and the +`karmasearch.videos` removal key never matched because the engine's real name +has a space in it. + +**`scripts/searxng-health.sh` asserts results > 0** across three unrelated +queries. That is the only check that could have caught this — the mechanism was +healthy throughout. + +**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered +`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets +`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an +empty list is indistinguishable from a broken aggregator, which is precisely how +this hid. Same principle as althing's "unreachable post office is an OUTAGE, +never an empty inbox". + +⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1 +`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install +--force` served a CACHED build** and silently reinstalled the old code — the +installed file still had the v1 API after the source no longer did. +`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what +caught it. + +Old instance stopped and removed; DNS alias repointed to +`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at +`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]] +for the other UniFi-side change the same day. + _Archived 2026-09-18._ + +- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.** + +`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only. + +**Operator ruling, and the better design:** put the address on the +**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A +host-side static works until the box moves, and then it is a stale netplan file +on a machine whose address you no longer know. A reservation moves with the MAC. + + UniFi switch port 22 native network -> nh3-servers (VLAN 50) + UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60 + host unchanged, still DHCP + +`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and +was NOT used — annotated as retired at its top. Its safety *ordering* was +followed and is still right. + +⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd +`10.100.0.111` from `nh3-default`. The switch port had to be repointed before +anything else could work. Do not assume a racked port is on the VLAN you asked +for. + +⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two +unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and +written back. + +⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still +up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits +directly on the userland subnet — so "I can reach it on the new address" proved +NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on +inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what +settled it: + + ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10 + ping -c2 -I enP7s7 1.1.1.1 0% loss egress + +Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to +`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot — +verified by reading `/var/lib/NetworkManager/NetworkManager.state` back. + +⚠ **The box now has exactly ONE path.** If that switch port or the reservation +breaks it is a rack visit; the escape hatch is deliberately gone. Correct end +state for a racked server, but a posture change from the desk setup — and this +is the box run 3c moved to. + +Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`. + _Archived 2026-09-18._ + +- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.** + + vuongs-mac-mini 10.100.79.2 infra-ops + lkraven + esh-macbook-air 10.0.10.83 infra-ops + lkraven + esh-mac-studio 10.0.10.10 infra-ops + vhpfi + +Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32 +random chars and vaulted at `/infra-ops-password`. `dsh` runs in the +operator's own account on each, on a **device-scoped** LiteLLM key +(`-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than +trusted from the mint) — a laptop travels, and losing one should be one +revocation, not a fleet key rotation. + +`scripts/provision-mac-dsh.sh [name]` carries every trap; the +operator-run half is `docs/runbooks/mac-provisioning.md`. + +⚠ **`sudo -u ` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit +`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at +a **working install in another account**. Only filesystem permissions stopped +it. The script now refuses to run unless `$HOME` matches the target. + +⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl` +leaves `/Users/` root-owned; the account authenticates, gets a shell, +reports the right `$HOME`, and cannot write to it — surfacing as a bare +"Permission denied" hours later. + +⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers +`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a +bad password, AND a user outside `com.apple.access_ssh`. This produced a false +diagnosis twice in one session — once where the password was a typo +(`no-password` vs `nopassword`) and I blamed the access group, once where the +Studio's operator account is **`vhpfi`, not `lkraven`**. Check +`dscl . -list /Users` FIRST. + +⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and +no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with +"Operation is not permitted without secure token unlock". `dscl` works precisely +because there is no token to desync. True on all three Macs. + +⚠ **FileVault kills remote access across reboots** — the machine sits at the +pre-boot unlock screen with no network. Nothing unattended should depend on a +Mac being reachable after a restart. + +⚠ macOS has no `adduser`, `useradd`, or `timeout`. + +Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook; +`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a +two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the +provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of +`agent-default-model` — without it the picker offers three models the gateway +does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`. + _Archived 2026-09-18._ + +# `[2026-09-03]` nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue + +**nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md` + _Archived 2026-09-18._ + - `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md` _Archived 2026-09-17._ diff --git a/persistent-memory.d/2026-09-03-gx10-rack-network.md b/persistent-memory.d/2026-09-03-gx10-rack-network.md deleted file mode 100644 index 006deed..0000000 --- a/persistent-memory.d/2026-09-03-gx10-rack-network.md +++ /dev/null @@ -1,46 +0,0 @@ -- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.** - -`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only. - -**Operator ruling, and the better design:** put the address on the -**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A -host-side static works until the box moves, and then it is a stale netplan file -on a machine whose address you no longer know. A reservation moves with the MAC. - - UniFi switch port 22 native network -> nh3-servers (VLAN 50) - UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60 - host unchanged, still DHCP - -`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and -was NOT used — annotated as retired at its top. Its safety *ordering* was -followed and is still right. - -⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd -`10.100.0.111` from `nh3-default`. The switch port had to be repointed before -anything else could work. Do not assume a racked port is on the VLAN you asked -for. - -⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two -unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and -written back. - -⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still -up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits -directly on the userland subnet — so "I can reach it on the new address" proved -NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on -inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what -settled it: - - ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10 - ping -c2 -I enP7s7 1.1.1.1 0% loss egress - -Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to -`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot — -verified by reading `/var/lib/NetworkManager/NetworkManager.state` back. - -⚠ **The box now has exactly ONE path.** If that switch port or the reservation -breaks it is a rack visit; the escape hatch is deliberately gone. Correct end -state for a racked server, but a posture change from the desk setup — and this -is the box run 3c moved to. - -Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`. diff --git a/persistent-memory.d/2026-09-03-mac-fleet-dsh.md b/persistent-memory.d/2026-09-03-mac-fleet-dsh.md deleted file mode 100644 index dd0600a..0000000 --- a/persistent-memory.d/2026-09-03-mac-fleet-dsh.md +++ /dev/null @@ -1,51 +0,0 @@ -- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.** - - vuongs-mac-mini 10.100.79.2 infra-ops + lkraven - esh-macbook-air 10.0.10.83 infra-ops + lkraven - esh-mac-studio 10.0.10.10 infra-ops + vhpfi - -Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32 -random chars and vaulted at `/infra-ops-password`. `dsh` runs in the -operator's own account on each, on a **device-scoped** LiteLLM key -(`-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than -trusted from the mint) — a laptop travels, and losing one should be one -revocation, not a fleet key rotation. - -`scripts/provision-mac-dsh.sh [name]` carries every trap; the -operator-run half is `docs/runbooks/mac-provisioning.md`. - -⚠ **`sudo -u ` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit -`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at -a **working install in another account**. Only filesystem permissions stopped -it. The script now refuses to run unless `$HOME` matches the target. - -⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl` -leaves `/Users/` root-owned; the account authenticates, gets a shell, -reports the right `$HOME`, and cannot write to it — surfacing as a bare -"Permission denied" hours later. - -⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers -`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a -bad password, AND a user outside `com.apple.access_ssh`. This produced a false -diagnosis twice in one session — once where the password was a typo -(`no-password` vs `nopassword`) and I blamed the access group, once where the -Studio's operator account is **`vhpfi`, not `lkraven`**. Check -`dscl . -list /Users` FIRST. - -⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and -no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with -"Operation is not permitted without secure token unlock". `dscl` works precisely -because there is no token to desync. True on all three Macs. - -⚠ **FileVault kills remote access across reboots** — the machine sits at the -pre-boot unlock screen with no network. Nothing unattended should depend on a -Mac being reachable after a restart. - -⚠ macOS has no `adduser`, `useradd`, or `timeout`. - -Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook; -`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a -two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the -provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of -`agent-default-model` — without it the picker offers three models the gateway -does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`. diff --git a/persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md b/persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md deleted file mode 100644 index a61ea89..0000000 --- a/persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-03]` nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue - -**nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md` diff --git a/persistent-memory.d/2026-09-03-searxng-nh3-move.md b/persistent-memory.d/2026-09-03-searxng-nh3-move.md deleted file mode 100644 index 0892f0a..0000000 --- a/persistent-memory.d/2026-09-03-searxng-nh3-move.md +++ /dev/null @@ -1,67 +0,0 @@ -- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.** - -**The failure.** The ana-docker instance answered `/healthz` every 30s, showed -`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while -returning **0 results for every query tested**. It was running **2026.4.17 -against a current 2026.9.3**: 4.5 months of engine scrapers rotting against -sites that had changed their markup. SearXNG ships near-daily releases for -exactly this reason. - -⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A -container created in April on `:latest` is pinned to April forever. - -⚠ **`/healthz` proves the web app answers and says NOTHING about whether search -works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that -was a stalled backup, the runner audit that trusted liveness for identity, and -the statusline bell that measured a mechanism. - -**Proven before acting**: the new image, same settings file, same host, same -query, in a throwaway container → **20 results where the running one returned -0**. Network was ruled out first — from inside the container DNS resolved and -mojeek/wikipedia were reachable, so engines were reachable and the parsers were -the broken part. - -**Why NH3 and not an in-place update** (operator's call, and the measurement -backs it): - - ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA - nh3-docker egress 70.230.226.88 residential -> not gated the same way - -Search engines gate datacenter ranges. Same reason the fleet keeps a residential -SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied -around. `outgoing.proxies` has the fallback commented in place if NH3's egress -ever changes. - -⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from -NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`, -`github`, `stackoverflow`, `marginalia` work. **General search is effectively -single-engine** — if google cse breaks, it goes quiet again. - -**Two config defects, both silent:** `base_url` still named -`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins, -so nothing broke and the file lied to every reader); and the -`karmasearch.videos` removal key never matched because the engine's real name -has a space in it. - -**`scripts/searxng-health.sh` asserts results > 0** across three unrelated -queries. That is the only check that could have caught this — the mechanism was -healthy throughout. - -**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered -`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets -`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an -empty list is indistinguishable from a broken aggregator, which is precisely how -this hid. Same principle as althing's "unreachable post office is an OUTAGE, -never an empty inbox". - -⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1 -`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install ---force` served a CACHED build** and silently reinstalled the old code — the -installed file still had the v1 API after the source no longer did. -`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what -caught it. - -Old instance stopped and removed; DNS alias repointed to -`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at -`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]] -for the other UniFi-side change the same day. diff --git a/persistent-memory.d/2026-09-17-next-voice-seats.md b/persistent-memory.d/2026-09-17-next-voice-seats.md new file mode 100644 index 0000000..003da11 --- /dev/null +++ b/persistent-memory.d/2026-09-17-next-voice-seats.md @@ -0,0 +1,69 @@ +# `[2026-09-17]` Which voices earn a training seat next — measured against the catalogue, not chosen by taste + +Method: rank every author in the kvasir catalogue by **usable extracted** works, then apply the +selection criterion the lv-krakauer parking established — *does the author have a voice*, asked +before any corpus work, and specifically **does that voice live where the instrument looks**. +`voice_distance.py` is Burrows's Delta over CHARACTER BIGRAMS, so it sees function-word morphology, +punctuation and sentence rhythm. A writer whose distinction is plot, research or subject matter is +invisible to it — an adapter cannot carry that, and the gate cannot measure it. + +⚠ `triage.length` is in **CHARACTERS**, ~5.2 chars/word calibrated against builds we did ourselves +(The Crossing mobi 777,420 chars = our measured 149,985 words). Dedup by title taking the max across +formats, and floor at 100,000 chars — that is what excludes the `accepted`-but-truncated rows +(Blood Meridian epub at 6,031 chars beside the mobi's 623,849). + +## ⭐ The size ranking INVERTS the voice ranking at the top + +``` +Stephen King 76 works 12,133,529 w <- biggest, and NOT a candidate +Agatha Christie 72 5,451,377 <- second biggest, the Krakauer case exactly +Terry Pratchett 52 4,821,474 +Georgette Heyer 30 3,416,867 +Graham Greene 45 3,037,425 +William Faulkner 25 2,981,183 <- the pick +``` + +Christie is the whole lesson in one row: a superb writer whose genius is plot architecture, in prose +deliberately kept transparent. Nothing for a char-bigram Delta to grip. King is the softer version — +distinctive in pacing and brand-name texture, not in syntax. + +## The three that clear both bars + +**1. William Faulkner — 25 catalogue rows, ~15 pure novels, ~1.6M words.** +*The voice in one sentence:* sentences that defer their main clause through stacked subordination +and coined compounds until the reader is held inside a single unbroken perception. +About as char-bigram-legible as English gets — the voice IS the clause-joining morphology and the +`and`/`which`/`that` density. ⭐ **And he is McCarthy's stylistic ancestor, which is the real +argument:** the Brontë gate record states the frozen adjudication needs "a control-author panel (to +place an absolute band and a hard-negative sister)" and notes we have none. Faulkner beside McCarthy +makes each the other's hard negative — a METHOD upgrade, not just another roster entry. +⚠ Messiest corpus of the three: a 446k-word `Snopes: The Hamlet, The Town, The Mansion` omnibus +duplicates novels also present individually, and `Three Famous Short Novels` overlaps it again. That +is the Hemingway trap (169,759 words of measured 90-96% collection duplication) — containment pass +before anything else. + +**2. Toni Morrison — 13 rows, 11 novels after pruning, ~818k words.** +*The voice:* free-indirect discourse sliding between narrator and character mid-sentence, carried on +incantatory repetition and deliberate fragments. +Cleanest corpus shape on the list: **11 novels → 11 val units, beating Hemingway's 10.** Val units +scale with WORK COUNT, which is the structural reason Brontë's voice axis came back underpowered at +4 with no cheap fix. ⚠ Drop `Burn This Book` (anthology she edited) and `Playing in the Dark` +(criticism) — same reason Krakauer's reporting does not transfer. + +**3. Raymond Chandler — 9 rows, 7 novels + a 409k short-story omnibus, ~970k words.** +*The voice:* clipped first-person declaratives that periodically detonate into one baroque simile, +with dialogue carrying most of the scene. +Fills the register gap nobody else fills — **first-person hardboiled**; the line has no first-person +male narrator at all. Corpus is almost exactly Hemingway-sized (997k vs 958k), which was the +decisive gate. ⚠ Drop `Essays and Reviews` — non-fiction. + +## Held, and why + +**Conrad** (31 works, 2.4M) is a genuine tier-1.5 if a fourth is wanted. **Melville** (10, 1.9M) has +a superb voice but a mixed-register corpus — the cetology chapters are a different book from the +narrative. **Austen** (12, 1.18M) is worth noting because Burrows's Delta was developed on her, so +the instrument is known to resolve her. The romantasy cluster is a separate question entirely — +see [[2026-09-17-romantasy-register-measured]]. + +Related: [[2026-09-17-mccarthy-split-name-leak]], [[2026-09-17-lv-bronte-gate]], +[[2026-09-17-lv-hemingway-gate]]. diff --git a/persistent-memory.d/2026-09-17-romantasy-register-measured.md b/persistent-memory.d/2026-09-17-romantasy-register-measured.md new file mode 100644 index 0000000..519a91a --- /dev/null +++ b/persistent-memory.d/2026-09-17-romantasy-register-measured.md @@ -0,0 +1,71 @@ +# `[2026-09-17]` Romantasy measured as a register — it is real, we already took its best voice, and the obvious next pick is its worst + +Prompted by the operator pushing back on a one-clause dismissal of the lane as "depth behind +Yarros". The dismissal was taste; this is a measurement, on the gate's own instrument. + +**Method.** Char-bigram Burrows's Delta, the same measure `voice_distance.py` gates on. ~120k words +per author, sampled from the MIDDLE quartile of each author's largest works (front and back matter +are not the voice), equalised so a bigger sample is not a different measurement. 400 most-frequent +bigrams as the feature set, z-scored over 4,000-word chunks pooled across all authors. + +**Controls first, because a between-author number without a within-author floor is unfalsifiable.** + +``` +A-vs-A floor (two halves of the SAME author) + Yarros 0.285 Maas 0.298 Armentrout 0.314 St. Clair 0.322 Cole 0.338 + Kenyon 0.375 Reyne 0.391 + McCarthy 0.303 Morrison 0.327 Brontë 0.209 Hemingway 0.454 <- worst, used as the bar + +positive controls (known-distinct pairs — the instrument must separate these) + Yarros vs McCarthy 0.862 1.9x + Hemingway vs Brontë 0.773 1.7x + McCarthy vs Morrison 0.675 1.5x + Hemingway vs McCarthy 0.655 1.4x + +romantasy, all 21 pairs median 0.537 1.2x floor (range 0.465 - 0.674) +``` + +**The register is real but tight.** 1.2x floor against controls at 1.4-1.9x. Only one pair falls to +1.0x, so it is not seven names for one voice. + +⚠ **Sensitivity floor, stated because a result without one is unfalsifiable.** The 0.454 bar is +Hemingway's, inflated by his own heterogeneous corpus (1920s-1960s, novels + stories + posthumous). +Against the romantasy authors' OWN floors (~0.34) the same pairs read ~1.6x — control-grade. The +truth sits between those readings and **this method cannot split it finer**. One sample per pair, no +repeat draws: read the rank ordering as indicative, do not read small gaps at all. + +## Two findings that survive either floor reading + +⭐ **Yarros is the cluster OUTLIER, not a typical member.** Four of the five largest distances in the +matrix involve her — Yarros-Kenyon 0.674, Yarros-St. Clair 0.644, Yarros-Reyne 0.637, Yarros-Maas +0.567. **We already trained the most distinctive romantasy voice we hold**, so a second seat in the +lane buys measurably less than the first did. That is the actual answer to "what about romantasy". + +⭐ **Maas is the centroid, so the obvious commercial pick is the least distinctive.** Maas-Reyne +0.465 and Maas-Cole 0.470 are the two SMALLEST distances in the whole matrix. She is the biggest +name available (922k words) and measurably the most generic of the seven in char-bigram terms. +Picking by sales rank picks the worst adapter. + +## If the lane gets a second seat it is Kenyon + +Furthest from the shipped Yarros (0.674), so it adds the most new signal — **and 27 works means 27 +val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4, where 4 +is the documented structural cause of an underpowered voice axis with no cheap fix). + +⚠ Two costs: the 27 are one series (Dark-Hunter), so the shared proper-noun space makes +`--scope corpus` mandatory rather than optional; and a "Dark Hunter - The Dark Hunter Complete" +omnibus sits in the catalogue rows, so the containment pass runs first. + +**Corpus shapes for the lane** (works ≥100k chars, deduped by title): +``` +Sherrilyn Kenyon 27 2,368,396 w Scarlett St. Clair 11 1,133,067 +Opal Reyne 14 2,466,307 Kresley Cole 10 1,037,914 +Jennifer Armentrout 6 1,179,953 Sarah J. Maas 5 922,711 +Rebecca Yarros 5 820,425 <- SHIPPED on this +``` +⭐ Worth noting for any future bar-setting: **Yarros shipped on 5 works / 820k words.** The corpus +bar is lower than it looks. + +Instrument: `scratchpad/regdist.py` (screening tool, not the gate). + +Related: [[2026-09-17-next-voice-seats]], [[2026-09-16-lv-voices-line]], [[2026-09-17-lv-bronte-gate]]. diff --git a/persistent-memory.md b/persistent-memory.md index e3bfafa..11adbd0 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-17 ~12:15 PT (lv-mccarthy: the leak gate had PASSED with five protagonist names still in every copy — hidden from `\bName\b` by a drop-cap space or a line-break hyphen. Corpus rebuilt, gate taught to see the class, D4 pairs generating on gx10. lv-hemingway keeps its 3 — operator ruled leave it, so the gate now exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static. lv-krakauer parked.)_ +_Last updated: 2026-09-17 ~23:20 PT (lv-mccarthy TRAINING on gx10, ~150/1380 steps. The leak gate had PASSED with five protagonist names still in every copy — corpus rebuilt, gate taught to see the class. lv-hemingway keeps its 3 by operator ruling, so the gate exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static; headscale split-DNSes nh3.phasefinal.com. Next voice seat measured, not guessed. lv-krakauer parked.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,7 +115,30 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-17 ~09:00 PT._ +_As of 2026-09-17 ~23:20 PT._ + +### RUNNING — lv-mccarthy LoRA on pfi-gx10, launched 2026-09-17 ~23:05 PT + +`~/r49-runs/mccarthy-4b-pairs-3ep/`, launched by `~/lv-mccarthy/launch-train.sh` (committed at +`scripts/mccarthy-corpus/launch-train.sh`). At the snapshot: **~150 of 1380 steps, ~4.2 s/it, +ETA ~1h25m**, so it lands around **00:45 PT**. A deliberate mirror of the Hemingway recipe — +Qwen3-4B-Instruct, seq-len 1536, rank 32 / alpha 64, lr 1e-4, 3 epochs, batch 1 x accum 8, +seed 4919 — with pairs sha `6a04fcd21800d85c`, 3673 train / 269 val, 0 dropped. + +⚠⚠ **`--save-total-limit` defaults to 12 and that would have silently destroyed the gate.** At the +25-step save cadence over 1380 steps that keeps only epochs ~2.35-3.0 and DELETES the epoch-1 and +epoch-2 checkpoints — which is exactly what the last two gates were decided on (Hemingway's ckpt850 +at epoch 0.96 TIED his eval-loss minimum ckpt1750; on Brontë the earlier epoch-1 checkpoint won the +tiebreak on the axes that resolve). Raised to 60 so all ~55 survive, ~42 GB against 442 GB free. +Eval/save cadence is 25 rather than Hemingway's 50 because McCarthy has 460 steps/epoch against his +887 — matching EPOCH resolution (0.054 vs 0.056), which is the unit the tiebreak is expressed in. + +**Next, in order:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) → +ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read: D1 pre-registered a +punctuation-normalised secondary read, and the `mccarthy` register now states the punctuation tics +explicitly, so `--system-from` hands them to the BASE control arm too. That was done deliberately to +deny the adapter a cheap char-bigram win, but it changes what the primary number means and must be +settled BEFORE a number exists. ### OPEN LOOP — `ravenpen.com`, and a reply hamr-dev is waiting on @@ -234,6 +257,8 @@ fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, i ## Recent decisions +- `[2026-09-17]` ⭐⭐ **The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top.** Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: **Faulkner** (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the **hard-negative sister the gate has lacked since the Brontë record named it missing**); **Morrison** (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); **Chandler** (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → `persistent-memory.d/2026-09-17-next-voice-seats.md` +- `[2026-09-17]` ⭐⭐ **Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst.** Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair **0.537 = 1.2x the worst floor** against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: **Yarros is the cluster OUTLIER** (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and **Maas is the centroid** (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is **Kenyon** — furthest from Yarros at 0.674 and **27 works = 27 val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → `persistent-memory.d/2026-09-17-romantasy-register-measured.md` - `[2026-09-17]` ⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched. ⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24). - `[2026-09-17]` **`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched. @@ -491,13 +516,9 @@ fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, i - `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md` -- `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md` -- `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I ` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md` -- `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md` -- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md` - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312. @@ -522,4 +543,4 @@ _5 older entries archived to archival-memory.md._ - `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md` -_110 older entries archived to archival-memory.md._ +_114 older entries archived to archival-memory.md._ diff --git a/scripts/mccarthy-corpus/launch-train.sh b/scripts/mccarthy-corpus/launch-train.sh new file mode 100755 index 0000000..24d4f86 --- /dev/null +++ b/scripts/mccarthy-corpus/launch-train.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# lv-mccarthy D5 — pairs LoRA. Run on pfi-gx10 as infra-ops. +# +# Mirrors the lv-hemingway recipe deliberately: same carrier, seq-len 1536, rank 32 (alpha 64), +# lr 1e-4, 3 epochs, batch 1 x accum 8, seed 4919. Those are the script defaults; they are +# written out explicitly anyway so this file IS the record of what ran. +# +# ⚠ TWO DELIBERATE DEVIATIONS FROM THE DEFAULTS, both about the checkpoint tiebreak: +# +# --save-total-limit 60 The default is 12. At a 25-step cadence over ~1,377 planned steps +# that is 55 checkpoints, so a limit of 12 keeps only epochs ~2.35-3.0 +# and DELETES the epoch-1 and epoch-2 checkpoints. Both lv-bronte and +# lv-hemingway were decided on exactly those: Hemingway's ckpt850 +# (epoch 0.96) tied its eval-loss minimum ckpt1750 (epoch 1.97) to +# within the neighbour jitter, and on Brontë the earlier epoch-1 +# checkpoint won the tiebreak on the axes that resolve. Losing them +# would remove the choice before it could be made. 55 x 757 MB is +# ~42 GB against 442 GB free. +# +# --eval-steps 25 Hemingway ran 50 on 887 steps/epoch = 0.056 epoch granularity. +# --save-steps 25 McCarthy has 459 steps/epoch, so 25 gives 0.054 -- matching the +# EPOCH resolution rather than the step number, which is what the +# tiebreak is actually expressed in. +set -e +cd ~/lv-mccarthy +PY=/home/infra-ops/ml/.venv/bin/python +OUT=~/r49-runs/mccarthy-4b-pairs-3ep +mkdir -p "$OUT" +"$PY" scripts/yarros-corpus/train_pairs_lora.py \ + --pairs pairs/pairs-full.jsonl \ + --val-pairs pairs/pairs-val.jsonl \ + --base /home/infra-ops/carriers/Qwen3-4B-Instruct \ + --out "$OUT" \ + --seq-len 1536 --rank 32 --lr 1e-4 --epochs 3 --batch 1 --accum 8 --seed 4919 \ + --eval-steps 25 --save-steps 25 --save-total-limit 60 \ + > "$OUT/train.log" 2>&1 +echo DONE > "$OUT/.complete"