memory: snapshot — lv-mccarthy training launched on gx10, and the next voice seat is measured rather than chosen
In-flight rewritten to the live training run (~150/1380, ETA ~00:45 PT) with the --save-total-limit finding that would otherwise have deleted the epoch-1/epoch-2 checkpoints both prior gates were decided on. Two decisions added: the next-seat ranking (Faulkner, Morrison, Chandler -- and the finding that the corpus size ranking inverts the voice ranking, with King and Christie as the two biggest non-candidates), and the romantasy register measured on the gate's own char-bigram instrument (Yarros is the cluster outlier we already shipped; Maas is the centroid and so the worst pick; Kenyon at 27 val units if the lane gets a seat). Auto-archival: 4 entries moved to archival-memory.md; 4 held back by the open-deferred guard.
This commit is contained in:
@@ -4,6 +4,181 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.**
|
||||
|
||||
**The failure.** The ana-docker instance answered `/healthz` every 30s, showed
|
||||
`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while
|
||||
returning **0 results for every query tested**. It was running **2026.4.17
|
||||
against a current 2026.9.3**: 4.5 months of engine scrapers rotting against
|
||||
sites that had changed their markup. SearXNG ships near-daily releases for
|
||||
exactly this reason.
|
||||
|
||||
⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A
|
||||
container created in April on `:latest` is pinned to April forever.
|
||||
|
||||
⚠ **`/healthz` proves the web app answers and says NOTHING about whether search
|
||||
works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that
|
||||
was a stalled backup, the runner audit that trusted liveness for identity, and
|
||||
the statusline bell that measured a mechanism.
|
||||
|
||||
**Proven before acting**: the new image, same settings file, same host, same
|
||||
query, in a throwaway container → **20 results where the running one returned
|
||||
0**. Network was ruled out first — from inside the container DNS resolved and
|
||||
mojeek/wikipedia were reachable, so engines were reachable and the parsers were
|
||||
the broken part.
|
||||
|
||||
**Why NH3 and not an in-place update** (operator's call, and the measurement
|
||||
backs it):
|
||||
|
||||
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
||||
nh3-docker egress 70.230.226.88 residential -> not gated the same way
|
||||
|
||||
Search engines gate datacenter ranges. Same reason the fleet keeps a residential
|
||||
SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied
|
||||
around. `outgoing.proxies` has the fallback commented in place if NH3's egress
|
||||
ever changes.
|
||||
|
||||
⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from
|
||||
NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`,
|
||||
`github`, `stackoverflow`, `marginalia` work. **General search is effectively
|
||||
single-engine** — if google cse breaks, it goes quiet again.
|
||||
|
||||
**Two config defects, both silent:** `base_url` still named
|
||||
`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins,
|
||||
so nothing broke and the file lied to every reader); and the
|
||||
`karmasearch.videos` removal key never matched because the engine's real name
|
||||
has a space in it.
|
||||
|
||||
**`scripts/searxng-health.sh` asserts results > 0** across three unrelated
|
||||
queries. That is the only check that could have caught this — the mechanism was
|
||||
healthy throughout.
|
||||
|
||||
**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered
|
||||
`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets
|
||||
`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an
|
||||
empty list is indistinguishable from a broken aggregator, which is precisely how
|
||||
this hid. Same principle as althing's "unreachable post office is an OUTAGE,
|
||||
never an empty inbox".
|
||||
|
||||
⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1
|
||||
`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install
|
||||
--force` served a CACHED build** and silently reinstalled the old code — the
|
||||
installed file still had the v1 API after the source no longer did.
|
||||
`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what
|
||||
caught it.
|
||||
|
||||
Old instance stopped and removed; DNS alias repointed to
|
||||
`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at
|
||||
`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]]
|
||||
for the other UniFi-side change the same day.
|
||||
_Archived 2026-09-18._
|
||||
|
||||
- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.**
|
||||
|
||||
`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
|
||||
|
||||
**Operator ruling, and the better design:** put the address on the
|
||||
**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A
|
||||
host-side static works until the box moves, and then it is a stale netplan file
|
||||
on a machine whose address you no longer know. A reservation moves with the MAC.
|
||||
|
||||
UniFi switch port 22 native network -> nh3-servers (VLAN 50)
|
||||
UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60
|
||||
host unchanged, still DHCP
|
||||
|
||||
`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and
|
||||
was NOT used — annotated as retired at its top. Its safety *ordering* was
|
||||
followed and is still right.
|
||||
|
||||
⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd
|
||||
`10.100.0.111` from `nh3-default`. The switch port had to be repointed before
|
||||
anything else could work. Do not assume a racked port is on the VLAN you asked
|
||||
for.
|
||||
|
||||
⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two
|
||||
unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and
|
||||
written back.
|
||||
|
||||
⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still
|
||||
up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits
|
||||
directly on the userland subnet — so "I can reach it on the new address" proved
|
||||
NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on
|
||||
inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what
|
||||
settled it:
|
||||
|
||||
ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10
|
||||
ping -c2 -I enP7s7 1.1.1.1 0% loss egress
|
||||
|
||||
Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to
|
||||
`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot —
|
||||
verified by reading `/var/lib/NetworkManager/NetworkManager.state` back.
|
||||
|
||||
⚠ **The box now has exactly ONE path.** If that switch port or the reservation
|
||||
breaks it is a rack visit; the escape hatch is deliberately gone. Correct end
|
||||
state for a racked server, but a posture change from the desk setup — and this
|
||||
is the box run 3c moved to.
|
||||
|
||||
Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`.
|
||||
_Archived 2026-09-18._
|
||||
|
||||
- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.**
|
||||
|
||||
vuongs-mac-mini 10.100.79.2 infra-ops + lkraven
|
||||
esh-macbook-air 10.0.10.83 infra-ops + lkraven
|
||||
esh-mac-studio 10.0.10.10 infra-ops + vhpfi
|
||||
|
||||
Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32
|
||||
random chars and vaulted at `<name>/infra-ops-password`. `dsh` runs in the
|
||||
operator's own account on each, on a **device-scoped** LiteLLM key
|
||||
(`<name>-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than
|
||||
trusted from the mint) — a laptop travels, and losing one should be one
|
||||
revocation, not a fleet key rotation.
|
||||
|
||||
`scripts/provision-mac-dsh.sh <host> <account> [name]` carries every trap; the
|
||||
operator-run half is `docs/runbooks/mac-provisioning.md`.
|
||||
|
||||
⚠ **`sudo -u <user>` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit
|
||||
`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at
|
||||
a **working install in another account**. Only filesystem permissions stopped
|
||||
it. The script now refuses to run unless `$HOME` matches the target.
|
||||
|
||||
⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl`
|
||||
leaves `/Users/<account>` root-owned; the account authenticates, gets a shell,
|
||||
reports the right `$HOME`, and cannot write to it — surfacing as a bare
|
||||
"Permission denied" hours later.
|
||||
|
||||
⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers
|
||||
`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
|
||||
bad password, AND a user outside `com.apple.access_ssh`. This produced a false
|
||||
diagnosis twice in one session — once where the password was a typo
|
||||
(`no-password` vs `nopassword`) and I blamed the access group, once where the
|
||||
Studio's operator account is **`vhpfi`, not `lkraven`**. Check
|
||||
`dscl . -list /Users` FIRST.
|
||||
|
||||
⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and
|
||||
no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
|
||||
"Operation is not permitted without secure token unlock". `dscl` works precisely
|
||||
because there is no token to desync. True on all three Macs.
|
||||
|
||||
⚠ **FileVault kills remote access across reboots** — the machine sits at the
|
||||
pre-boot unlock screen with no network. Nothing unattended should depend on a
|
||||
Mac being reachable after a restart.
|
||||
|
||||
⚠ macOS has no `adduser`, `useradd`, or `timeout`.
|
||||
|
||||
Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook;
|
||||
`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a
|
||||
two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the
|
||||
provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of
|
||||
`agent-default-model` — without it the picker offers three models the gateway
|
||||
does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`.
|
||||
_Archived 2026-09-18._
|
||||
|
||||
# `[2026-09-03]` nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue
|
||||
|
||||
**nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/<UPID>`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md`
|
||||
_Archived 2026-09-18._
|
||||
|
||||
- `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
|
||||
_Archived 2026-09-17._
|
||||
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.**
|
||||
|
||||
`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
|
||||
|
||||
**Operator ruling, and the better design:** put the address on the
|
||||
**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A
|
||||
host-side static works until the box moves, and then it is a stale netplan file
|
||||
on a machine whose address you no longer know. A reservation moves with the MAC.
|
||||
|
||||
UniFi switch port 22 native network -> nh3-servers (VLAN 50)
|
||||
UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60
|
||||
host unchanged, still DHCP
|
||||
|
||||
`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and
|
||||
was NOT used — annotated as retired at its top. Its safety *ordering* was
|
||||
followed and is still right.
|
||||
|
||||
⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd
|
||||
`10.100.0.111` from `nh3-default`. The switch port had to be repointed before
|
||||
anything else could work. Do not assume a racked port is on the VLAN you asked
|
||||
for.
|
||||
|
||||
⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two
|
||||
unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and
|
||||
written back.
|
||||
|
||||
⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still
|
||||
up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits
|
||||
directly on the userland subnet — so "I can reach it on the new address" proved
|
||||
NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on
|
||||
inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what
|
||||
settled it:
|
||||
|
||||
ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10
|
||||
ping -c2 -I enP7s7 1.1.1.1 0% loss egress
|
||||
|
||||
Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to
|
||||
`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot —
|
||||
verified by reading `/var/lib/NetworkManager/NetworkManager.state` back.
|
||||
|
||||
⚠ **The box now has exactly ONE path.** If that switch port or the reservation
|
||||
breaks it is a rack visit; the escape hatch is deliberately gone. Correct end
|
||||
state for a racked server, but a posture change from the desk setup — and this
|
||||
is the box run 3c moved to.
|
||||
|
||||
Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`.
|
||||
@@ -1,51 +0,0 @@
|
||||
- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.**
|
||||
|
||||
vuongs-mac-mini 10.100.79.2 infra-ops + lkraven
|
||||
esh-macbook-air 10.0.10.83 infra-ops + lkraven
|
||||
esh-mac-studio 10.0.10.10 infra-ops + vhpfi
|
||||
|
||||
Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32
|
||||
random chars and vaulted at `<name>/infra-ops-password`. `dsh` runs in the
|
||||
operator's own account on each, on a **device-scoped** LiteLLM key
|
||||
(`<name>-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than
|
||||
trusted from the mint) — a laptop travels, and losing one should be one
|
||||
revocation, not a fleet key rotation.
|
||||
|
||||
`scripts/provision-mac-dsh.sh <host> <account> [name]` carries every trap; the
|
||||
operator-run half is `docs/runbooks/mac-provisioning.md`.
|
||||
|
||||
⚠ **`sudo -u <user>` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit
|
||||
`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at
|
||||
a **working install in another account**. Only filesystem permissions stopped
|
||||
it. The script now refuses to run unless `$HOME` matches the target.
|
||||
|
||||
⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl`
|
||||
leaves `/Users/<account>` root-owned; the account authenticates, gets a shell,
|
||||
reports the right `$HOME`, and cannot write to it — surfacing as a bare
|
||||
"Permission denied" hours later.
|
||||
|
||||
⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers
|
||||
`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
|
||||
bad password, AND a user outside `com.apple.access_ssh`. This produced a false
|
||||
diagnosis twice in one session — once where the password was a typo
|
||||
(`no-password` vs `nopassword`) and I blamed the access group, once where the
|
||||
Studio's operator account is **`vhpfi`, not `lkraven`**. Check
|
||||
`dscl . -list /Users` FIRST.
|
||||
|
||||
⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and
|
||||
no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
|
||||
"Operation is not permitted without secure token unlock". `dscl` works precisely
|
||||
because there is no token to desync. True on all three Macs.
|
||||
|
||||
⚠ **FileVault kills remote access across reboots** — the machine sits at the
|
||||
pre-boot unlock screen with no network. Nothing unattended should depend on a
|
||||
Mac being reachable after a restart.
|
||||
|
||||
⚠ macOS has no `adduser`, `useradd`, or `timeout`.
|
||||
|
||||
Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook;
|
||||
`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a
|
||||
two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the
|
||||
provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of
|
||||
`agent-default-model` — without it the picker offers three models the gateway
|
||||
does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-03]` nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue
|
||||
|
||||
**nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/<UPID>`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md`
|
||||
@@ -1,67 +0,0 @@
|
||||
- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.**
|
||||
|
||||
**The failure.** The ana-docker instance answered `/healthz` every 30s, showed
|
||||
`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while
|
||||
returning **0 results for every query tested**. It was running **2026.4.17
|
||||
against a current 2026.9.3**: 4.5 months of engine scrapers rotting against
|
||||
sites that had changed their markup. SearXNG ships near-daily releases for
|
||||
exactly this reason.
|
||||
|
||||
⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A
|
||||
container created in April on `:latest` is pinned to April forever.
|
||||
|
||||
⚠ **`/healthz` proves the web app answers and says NOTHING about whether search
|
||||
works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that
|
||||
was a stalled backup, the runner audit that trusted liveness for identity, and
|
||||
the statusline bell that measured a mechanism.
|
||||
|
||||
**Proven before acting**: the new image, same settings file, same host, same
|
||||
query, in a throwaway container → **20 results where the running one returned
|
||||
0**. Network was ruled out first — from inside the container DNS resolved and
|
||||
mojeek/wikipedia were reachable, so engines were reachable and the parsers were
|
||||
the broken part.
|
||||
|
||||
**Why NH3 and not an in-place update** (operator's call, and the measurement
|
||||
backs it):
|
||||
|
||||
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
||||
nh3-docker egress 70.230.226.88 residential -> not gated the same way
|
||||
|
||||
Search engines gate datacenter ranges. Same reason the fleet keeps a residential
|
||||
SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied
|
||||
around. `outgoing.proxies` has the fallback commented in place if NH3's egress
|
||||
ever changes.
|
||||
|
||||
⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from
|
||||
NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`,
|
||||
`github`, `stackoverflow`, `marginalia` work. **General search is effectively
|
||||
single-engine** — if google cse breaks, it goes quiet again.
|
||||
|
||||
**Two config defects, both silent:** `base_url` still named
|
||||
`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins,
|
||||
so nothing broke and the file lied to every reader); and the
|
||||
`karmasearch.videos` removal key never matched because the engine's real name
|
||||
has a space in it.
|
||||
|
||||
**`scripts/searxng-health.sh` asserts results > 0** across three unrelated
|
||||
queries. That is the only check that could have caught this — the mechanism was
|
||||
healthy throughout.
|
||||
|
||||
**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered
|
||||
`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets
|
||||
`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an
|
||||
empty list is indistinguishable from a broken aggregator, which is precisely how
|
||||
this hid. Same principle as althing's "unreachable post office is an OUTAGE,
|
||||
never an empty inbox".
|
||||
|
||||
⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1
|
||||
`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install
|
||||
--force` served a CACHED build** and silently reinstalled the old code — the
|
||||
installed file still had the v1 API after the source no longer did.
|
||||
`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what
|
||||
caught it.
|
||||
|
||||
Old instance stopped and removed; DNS alias repointed to
|
||||
`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at
|
||||
`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]]
|
||||
for the other UniFi-side change the same day.
|
||||
@@ -0,0 +1,69 @@
|
||||
# `[2026-09-17]` Which voices earn a training seat next — measured against the catalogue, not chosen by taste
|
||||
|
||||
Method: rank every author in the kvasir catalogue by **usable extracted** works, then apply the
|
||||
selection criterion the lv-krakauer parking established — *does the author have a voice*, asked
|
||||
before any corpus work, and specifically **does that voice live where the instrument looks**.
|
||||
`voice_distance.py` is Burrows's Delta over CHARACTER BIGRAMS, so it sees function-word morphology,
|
||||
punctuation and sentence rhythm. A writer whose distinction is plot, research or subject matter is
|
||||
invisible to it — an adapter cannot carry that, and the gate cannot measure it.
|
||||
|
||||
⚠ `triage.length` is in **CHARACTERS**, ~5.2 chars/word calibrated against builds we did ourselves
|
||||
(The Crossing mobi 777,420 chars = our measured 149,985 words). Dedup by title taking the max across
|
||||
formats, and floor at 100,000 chars — that is what excludes the `accepted`-but-truncated rows
|
||||
(Blood Meridian epub at 6,031 chars beside the mobi's 623,849).
|
||||
|
||||
## ⭐ The size ranking INVERTS the voice ranking at the top
|
||||
|
||||
```
|
||||
Stephen King 76 works 12,133,529 w <- biggest, and NOT a candidate
|
||||
Agatha Christie 72 5,451,377 <- second biggest, the Krakauer case exactly
|
||||
Terry Pratchett 52 4,821,474
|
||||
Georgette Heyer 30 3,416,867
|
||||
Graham Greene 45 3,037,425
|
||||
William Faulkner 25 2,981,183 <- the pick
|
||||
```
|
||||
|
||||
Christie is the whole lesson in one row: a superb writer whose genius is plot architecture, in prose
|
||||
deliberately kept transparent. Nothing for a char-bigram Delta to grip. King is the softer version —
|
||||
distinctive in pacing and brand-name texture, not in syntax.
|
||||
|
||||
## The three that clear both bars
|
||||
|
||||
**1. William Faulkner — 25 catalogue rows, ~15 pure novels, ~1.6M words.**
|
||||
*The voice in one sentence:* sentences that defer their main clause through stacked subordination
|
||||
and coined compounds until the reader is held inside a single unbroken perception.
|
||||
About as char-bigram-legible as English gets — the voice IS the clause-joining morphology and the
|
||||
`and`/`which`/`that` density. ⭐ **And he is McCarthy's stylistic ancestor, which is the real
|
||||
argument:** the Brontë gate record states the frozen adjudication needs "a control-author panel (to
|
||||
place an absolute band and a hard-negative sister)" and notes we have none. Faulkner beside McCarthy
|
||||
makes each the other's hard negative — a METHOD upgrade, not just another roster entry.
|
||||
⚠ Messiest corpus of the three: a 446k-word `Snopes: The Hamlet, The Town, The Mansion` omnibus
|
||||
duplicates novels also present individually, and `Three Famous Short Novels` overlaps it again. That
|
||||
is the Hemingway trap (169,759 words of measured 90-96% collection duplication) — containment pass
|
||||
before anything else.
|
||||
|
||||
**2. Toni Morrison — 13 rows, 11 novels after pruning, ~818k words.**
|
||||
*The voice:* free-indirect discourse sliding between narrator and character mid-sentence, carried on
|
||||
incantatory repetition and deliberate fragments.
|
||||
Cleanest corpus shape on the list: **11 novels → 11 val units, beating Hemingway's 10.** Val units
|
||||
scale with WORK COUNT, which is the structural reason Brontë's voice axis came back underpowered at
|
||||
4 with no cheap fix. ⚠ Drop `Burn This Book` (anthology she edited) and `Playing in the Dark`
|
||||
(criticism) — same reason Krakauer's reporting does not transfer.
|
||||
|
||||
**3. Raymond Chandler — 9 rows, 7 novels + a 409k short-story omnibus, ~970k words.**
|
||||
*The voice:* clipped first-person declaratives that periodically detonate into one baroque simile,
|
||||
with dialogue carrying most of the scene.
|
||||
Fills the register gap nobody else fills — **first-person hardboiled**; the line has no first-person
|
||||
male narrator at all. Corpus is almost exactly Hemingway-sized (997k vs 958k), which was the
|
||||
decisive gate. ⚠ Drop `Essays and Reviews` — non-fiction.
|
||||
|
||||
## Held, and why
|
||||
|
||||
**Conrad** (31 works, 2.4M) is a genuine tier-1.5 if a fourth is wanted. **Melville** (10, 1.9M) has
|
||||
a superb voice but a mixed-register corpus — the cetology chapters are a different book from the
|
||||
narrative. **Austen** (12, 1.18M) is worth noting because Burrows's Delta was developed on her, so
|
||||
the instrument is known to resolve her. The romantasy cluster is a separate question entirely —
|
||||
see [[2026-09-17-romantasy-register-measured]].
|
||||
|
||||
Related: [[2026-09-17-mccarthy-split-name-leak]], [[2026-09-17-lv-bronte-gate]],
|
||||
[[2026-09-17-lv-hemingway-gate]].
|
||||
@@ -0,0 +1,71 @@
|
||||
# `[2026-09-17]` Romantasy measured as a register — it is real, we already took its best voice, and the obvious next pick is its worst
|
||||
|
||||
Prompted by the operator pushing back on a one-clause dismissal of the lane as "depth behind
|
||||
Yarros". The dismissal was taste; this is a measurement, on the gate's own instrument.
|
||||
|
||||
**Method.** Char-bigram Burrows's Delta, the same measure `voice_distance.py` gates on. ~120k words
|
||||
per author, sampled from the MIDDLE quartile of each author's largest works (front and back matter
|
||||
are not the voice), equalised so a bigger sample is not a different measurement. 400 most-frequent
|
||||
bigrams as the feature set, z-scored over 4,000-word chunks pooled across all authors.
|
||||
|
||||
**Controls first, because a between-author number without a within-author floor is unfalsifiable.**
|
||||
|
||||
```
|
||||
A-vs-A floor (two halves of the SAME author)
|
||||
Yarros 0.285 Maas 0.298 Armentrout 0.314 St. Clair 0.322 Cole 0.338
|
||||
Kenyon 0.375 Reyne 0.391
|
||||
McCarthy 0.303 Morrison 0.327 Brontë 0.209 Hemingway 0.454 <- worst, used as the bar
|
||||
|
||||
positive controls (known-distinct pairs — the instrument must separate these)
|
||||
Yarros vs McCarthy 0.862 1.9x
|
||||
Hemingway vs Brontë 0.773 1.7x
|
||||
McCarthy vs Morrison 0.675 1.5x
|
||||
Hemingway vs McCarthy 0.655 1.4x
|
||||
|
||||
romantasy, all 21 pairs median 0.537 1.2x floor (range 0.465 - 0.674)
|
||||
```
|
||||
|
||||
**The register is real but tight.** 1.2x floor against controls at 1.4-1.9x. Only one pair falls to
|
||||
1.0x, so it is not seven names for one voice.
|
||||
|
||||
⚠ **Sensitivity floor, stated because a result without one is unfalsifiable.** The 0.454 bar is
|
||||
Hemingway's, inflated by his own heterogeneous corpus (1920s-1960s, novels + stories + posthumous).
|
||||
Against the romantasy authors' OWN floors (~0.34) the same pairs read ~1.6x — control-grade. The
|
||||
truth sits between those readings and **this method cannot split it finer**. One sample per pair, no
|
||||
repeat draws: read the rank ordering as indicative, do not read small gaps at all.
|
||||
|
||||
## Two findings that survive either floor reading
|
||||
|
||||
⭐ **Yarros is the cluster OUTLIER, not a typical member.** Four of the five largest distances in the
|
||||
matrix involve her — Yarros-Kenyon 0.674, Yarros-St. Clair 0.644, Yarros-Reyne 0.637, Yarros-Maas
|
||||
0.567. **We already trained the most distinctive romantasy voice we hold**, so a second seat in the
|
||||
lane buys measurably less than the first did. That is the actual answer to "what about romantasy".
|
||||
|
||||
⭐ **Maas is the centroid, so the obvious commercial pick is the least distinctive.** Maas-Reyne
|
||||
0.465 and Maas-Cole 0.470 are the two SMALLEST distances in the whole matrix. She is the biggest
|
||||
name available (922k words) and measurably the most generic of the seven in char-bigram terms.
|
||||
Picking by sales rank picks the worst adapter.
|
||||
|
||||
## If the lane gets a second seat it is Kenyon
|
||||
|
||||
Furthest from the shipped Yarros (0.674), so it adds the most new signal — **and 27 works means 27
|
||||
val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4, where 4
|
||||
is the documented structural cause of an underpowered voice axis with no cheap fix).
|
||||
|
||||
⚠ Two costs: the 27 are one series (Dark-Hunter), so the shared proper-noun space makes
|
||||
`--scope corpus` mandatory rather than optional; and a "Dark Hunter - The Dark Hunter Complete"
|
||||
omnibus sits in the catalogue rows, so the containment pass runs first.
|
||||
|
||||
**Corpus shapes for the lane** (works ≥100k chars, deduped by title):
|
||||
```
|
||||
Sherrilyn Kenyon 27 2,368,396 w Scarlett St. Clair 11 1,133,067
|
||||
Opal Reyne 14 2,466,307 Kresley Cole 10 1,037,914
|
||||
Jennifer Armentrout 6 1,179,953 Sarah J. Maas 5 922,711
|
||||
Rebecca Yarros 5 820,425 <- SHIPPED on this
|
||||
```
|
||||
⭐ Worth noting for any future bar-setting: **Yarros shipped on 5 works / 820k words.** The corpus
|
||||
bar is lower than it looks.
|
||||
|
||||
Instrument: `scratchpad/regdist.py` (screening tool, not the gate).
|
||||
|
||||
Related: [[2026-09-17-next-voice-seats]], [[2026-09-16-lv-voices-line]], [[2026-09-17-lv-bronte-gate]].
|
||||
+28
-7
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-17 ~12:15 PT (lv-mccarthy: the leak gate had PASSED with five protagonist names still in every copy — hidden from `\bName\b` by a drop-cap space or a line-break hyphen. Corpus rebuilt, gate taught to see the class, D4 pairs generating on gx10. lv-hemingway keeps its 3 — operator ruled leave it, so the gate now exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static. lv-krakauer parked.)_
|
||||
_Last updated: 2026-09-17 ~23:20 PT (lv-mccarthy TRAINING on gx10, ~150/1380 steps. The leak gate had PASSED with five protagonist names still in every copy — corpus rebuilt, gate taught to see the class. lv-hemingway keeps its 3 by operator ruling, so the gate exits 1 on a shipped tree BY DESIGN. ESH back on the Cityside static; headscale split-DNSes nh3.phasefinal.com. Next voice seat measured, not guessed. lv-krakauer parked.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,7 +115,30 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-17 ~09:00 PT._
|
||||
_As of 2026-09-17 ~23:20 PT._
|
||||
|
||||
### RUNNING — lv-mccarthy LoRA on pfi-gx10, launched 2026-09-17 ~23:05 PT
|
||||
|
||||
`~/r49-runs/mccarthy-4b-pairs-3ep/`, launched by `~/lv-mccarthy/launch-train.sh` (committed at
|
||||
`scripts/mccarthy-corpus/launch-train.sh`). At the snapshot: **~150 of 1380 steps, ~4.2 s/it,
|
||||
ETA ~1h25m**, so it lands around **00:45 PT**. A deliberate mirror of the Hemingway recipe —
|
||||
Qwen3-4B-Instruct, seq-len 1536, rank 32 / alpha 64, lr 1e-4, 3 epochs, batch 1 x accum 8,
|
||||
seed 4919 — with pairs sha `6a04fcd21800d85c`, 3673 train / 269 val, 0 dropped.
|
||||
|
||||
⚠⚠ **`--save-total-limit` defaults to 12 and that would have silently destroyed the gate.** At the
|
||||
25-step save cadence over 1380 steps that keeps only epochs ~2.35-3.0 and DELETES the epoch-1 and
|
||||
epoch-2 checkpoints — which is exactly what the last two gates were decided on (Hemingway's ckpt850
|
||||
at epoch 0.96 TIED his eval-loss minimum ckpt1750; on Brontë the earlier epoch-1 checkpoint won the
|
||||
tiebreak on the axes that resolve). Raised to 60 so all ~55 survive, ~42 GB against 442 GB free.
|
||||
Eval/save cadence is 25 rather than Hemingway's 50 because McCarthy has 460 steps/epoch against his
|
||||
887 — matching EPOCH resolution (0.054 vs 0.056), which is the unit the tiebreak is expressed in.
|
||||
|
||||
**Next, in order:** checkpoint selection off the loss curve → the v2 gate (`eval-*.sh` shape) →
|
||||
ship-or-park. ⚠ Before the gate, settle how the VOICE axis is read: D1 pre-registered a
|
||||
punctuation-normalised secondary read, and the `mccarthy` register now states the punctuation tics
|
||||
explicitly, so `--system-from` hands them to the BASE control arm too. That was done deliberately to
|
||||
deny the adapter a cheap char-bigram win, but it changes what the primary number means and must be
|
||||
settled BEFORE a number exists.
|
||||
|
||||
### OPEN LOOP — `ravenpen.com`, and a reply hamr-dev is waiting on
|
||||
|
||||
@@ -234,6 +257,8 @@ fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, i
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-17]` ⭐⭐ **The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top.** Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: **Faulkner** (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the **hard-negative sister the gate has lacked since the Brontë record named it missing**); **Morrison** (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); **Chandler** (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → `persistent-memory.d/2026-09-17-next-voice-seats.md`
|
||||
- `[2026-09-17]` ⭐⭐ **Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst.** Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair **0.537 = 1.2x the worst floor** against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: **Yarros is the cluster OUTLIER** (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and **Maas is the centroid** (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is **Kenyon** — furthest from Yarros at 0.674 and **27 works = 27 val units, the best-powered gate the line could build** (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → `persistent-memory.d/2026-09-17-romantasy-register-measured.md`
|
||||
- `[2026-09-17]` ⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
|
||||
- `[2026-09-17]` **`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.
|
||||
@@ -491,13 +516,9 @@ fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, i
|
||||
|
||||
- `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md`
|
||||
|
||||
- `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md`
|
||||
|
||||
- `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I <wired>` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md`
|
||||
|
||||
- `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md`
|
||||
|
||||
- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md`
|
||||
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
@@ -522,4 +543,4 @@ _5 older entries archived to archival-memory.md._
|
||||
|
||||
- `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md`
|
||||
|
||||
_110 older entries archived to archival-memory.md._
|
||||
_114 older entries archived to archival-memory.md._
|
||||
|
||||
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/bin/bash
|
||||
# lv-mccarthy D5 — pairs LoRA. Run on pfi-gx10 as infra-ops.
|
||||
#
|
||||
# Mirrors the lv-hemingway recipe deliberately: same carrier, seq-len 1536, rank 32 (alpha 64),
|
||||
# lr 1e-4, 3 epochs, batch 1 x accum 8, seed 4919. Those are the script defaults; they are
|
||||
# written out explicitly anyway so this file IS the record of what ran.
|
||||
#
|
||||
# ⚠ TWO DELIBERATE DEVIATIONS FROM THE DEFAULTS, both about the checkpoint tiebreak:
|
||||
#
|
||||
# --save-total-limit 60 The default is 12. At a 25-step cadence over ~1,377 planned steps
|
||||
# that is 55 checkpoints, so a limit of 12 keeps only epochs ~2.35-3.0
|
||||
# and DELETES the epoch-1 and epoch-2 checkpoints. Both lv-bronte and
|
||||
# lv-hemingway were decided on exactly those: Hemingway's ckpt850
|
||||
# (epoch 0.96) tied its eval-loss minimum ckpt1750 (epoch 1.97) to
|
||||
# within the neighbour jitter, and on Brontë the earlier epoch-1
|
||||
# checkpoint won the tiebreak on the axes that resolve. Losing them
|
||||
# would remove the choice before it could be made. 55 x 757 MB is
|
||||
# ~42 GB against 442 GB free.
|
||||
#
|
||||
# --eval-steps 25 Hemingway ran 50 on 887 steps/epoch = 0.056 epoch granularity.
|
||||
# --save-steps 25 McCarthy has 459 steps/epoch, so 25 gives 0.054 -- matching the
|
||||
# EPOCH resolution rather than the step number, which is what the
|
||||
# tiebreak is actually expressed in.
|
||||
set -e
|
||||
cd ~/lv-mccarthy
|
||||
PY=/home/infra-ops/ml/.venv/bin/python
|
||||
OUT=~/r49-runs/mccarthy-4b-pairs-3ep
|
||||
mkdir -p "$OUT"
|
||||
"$PY" scripts/yarros-corpus/train_pairs_lora.py \
|
||||
--pairs pairs/pairs-full.jsonl \
|
||||
--val-pairs pairs/pairs-val.jsonl \
|
||||
--base /home/infra-ops/carriers/Qwen3-4B-Instruct \
|
||||
--out "$OUT" \
|
||||
--seq-len 1536 --rank 32 --lr 1e-4 --epochs 3 --batch 1 --accum 8 --seed 4919 \
|
||||
--eval-steps 25 --save-steps 25 --save-total-limit 60 \
|
||||
> "$OUT/train.log" 2>&1
|
||||
echo DONE > "$OUT/.complete"
|
||||
Reference in New Issue
Block a user