diff --git a/archival-memory.md b/archival-memory.md index ea657aa..d0a0dd9 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -13789,3 +13789,178 @@ Until it ships, the mitigation is entirely sender-side: pipe the body, never `-- **`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step. _Archived 2026-10-04._ + +## Recent decisions (archived) + +# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong. + +⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it. + _Archived 2026-10-05._ + +# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass. + +⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`. + _Archived 2026-10-05._ + +# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug. + +⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`. + _Archived 2026-10-05._ + +# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months + +**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi` +(team 4) and `vastblue` (team 5). Verified by reading membership back AND by +exercising it on claude-bot's own token: create/edit/delete in `pfi` all +succeed, `vh/*` correctly still 404s. + +⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a +**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no +namespace-scoped admin for a user namespace. Measured: per-repo `admin` +collaborator grants read but **not** settings (403 on PATCH) — repo settings are +owner-only. So the only working realization of "admin over vh/*" is the +**instance-wide site-admin flag**, which would have given claude-bot the same +blast radius as the token the credential-migration project exists to retire. +Surfaced rather than executed; the operator chose orgs instead. + +**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved, +old paths 301ing, and — verified — **the old ssh remotes still resolve**, since +Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote +living on a redirect depends on the old path staying unclaimed. + +⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as +**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the +operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The +conclusion survived re-testing with a working credential, but the evidence was +worthless. **A credential that authenticates as nobody returns 403 and 404 for +everything, and that is indistinguishable from a permissions answer.** There +are FOUR token files in that directory; the working one for repo work is +**`gitea-token-repo-create`** (`write:organization`, `write:repository`, +`write:user`). + +**Standing authorization (operator, same day):** routine vh-token use for +`vh/*` repo ops no longer gets a flag. Read it from the **vault** — +`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the +disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make +the token low-blast-radius; it stopped the class of work being exceptional. + +Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in +`reference_infra_ops_vh_gitea_token_and_sdk_publish`. + _Archived 2026-10-05._ + +# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days + +Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache +prune`, npm, pip), then a deep dive found the real shape. + +⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session +scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** — +252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance — +**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`. +`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18 +that **overrides** the stock no-age rule, and the cleaner runs daily. + +**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by +content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256` +recorded at training time. + +⚠ **Two corrections I made to myself during the dive, both worth keeping:** I +alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their +mtimes were *that day*, a live session working. And I suspected my own `du`/ +`find` had reset the atime clock and manufactured the "0 would-remove" result; +it had not (`relatime`, and an untouched comparison file still showed an old +atime) — but it was right to check before trusting a number my own measurement +could have created. + +**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB +`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets` +(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% → +72%.** + +⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session +dir is an unsound liveness test: **a directory's mtime does not change when +files are written into its subdirectories.** `dfacccde/` looked 7+ days idle +while `dfacccde/tasks/` was being written continuously. Cost: one lost tool +output; recreated. All four of my safety assertions (right root, right depth, +own session excluded, shutterchute excluded) passed — I checked the paths were +right and never checked the liveness test was sound. **Do not re-run that +predicate.** A correct version checks the deepest recent file, or +cross-references running `claude` PIDs. + +Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev, +while `ssh infra-ops@localhost` has NOPASSWD. + _Archived 2026-10-05._ + +# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested + +**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`. + _Archived 2026-10-05._ + +# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3) + +**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴. + _Archived 2026-10-05._ + +# `[2026-09-21]` `vh/forgefirm` mirrored + +**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable. + _Archived 2026-10-05._ + +# Beszel fleet wiring — 2026-09-10 + +Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox** +for alerts (Miranda later), approved creation of a dedicated monitoring superuser, +and asked for GPU usage/power telemetry and the card's health detail. + +Completed: all seven requested hosts up, alongside previously registered +corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up, +contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was +healthy but its hub record still pointed at retired 10.100.79.3; fixed to +100.64.0.6. althing-post-office container remained up throughout. + +Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host +overrides under stacks/beszel/hosts provide read-only mounts. Existing project +directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK +and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external +traefik-net network even with agent-only profile. No host Docker upgrade. + +ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close +to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both +GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power +limits or serving workloads changed. GPU watts do not size a whole-host PSU. + +Homepage uses existing discovery labels and version-2 widget; verified one +card and live authenticated data. This overview shows systems/up only; +reachability is not a degraded-health score. Per-system widget can expose +CPU/memory/root disk/network; hub charts contain the additional disks and GPUs. + +Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com, +Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its +credential, labels only placeholders. Existing operator login unchanged. + +Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m, +offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused +email route replaced with verified webhook. nh3-dev system service +beszel-althing forwards JSON via supported postbox CLI, sender/recipient +infra-ops; configurable recipient for later Miranda move. See service README. + +Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread +01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored +85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for +clickable alert deep links. Inbox verification did not mark mail read. + +Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post +office outage detection. Bridge deliberately has no hidden delivery queue; +downstream failure is logged and HTTP 502, not a claimed delivery. + _Archived 2026-10-05._ + +- (orphaned continuation of an earlier-archived domain-transfer entry; its dated head line was archived before this fragment) + ⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24). + _Archived 2026-10-05._ + +## Tried and abandoned (archived) + +# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads + +**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `/tasks/` looks 7+ days idle at `/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs. + _Archived 2026-10-05._ diff --git a/persistent-memory.d/2026-09-10-beszel-fleet-wiring.md b/persistent-memory.d/2026-09-10-beszel-fleet-wiring.md deleted file mode 100644 index 918eae1..0000000 --- a/persistent-memory.d/2026-09-10-beszel-fleet-wiring.md +++ /dev/null @@ -1,46 +0,0 @@ -# Beszel fleet wiring — 2026-09-10 - -Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox** -for alerts (Miranda later), approved creation of a dedicated monitoring superuser, -and asked for GPU usage/power telemetry and the card's health detail. - -Completed: all seven requested hosts up, alongside previously registered -corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up, -contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was -healthy but its hub record still pointed at retired 10.100.79.3; fixed to -100.64.0.6. althing-post-office container remained up throughout. - -Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host -overrides under stacks/beszel/hosts provide read-only mounts. Existing project -directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK -and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external -traefik-net network even with agent-only profile. No host Docker upgrade. - -ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close -to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both -GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power -limits or serving workloads changed. GPU watts do not size a whole-host PSU. - -Homepage uses existing discovery labels and version-2 widget; verified one -card and live authenticated data. This overview shows systems/up only; -reachability is not a degraded-health score. Per-system widget can expose -CPU/memory/root disk/network; hub charts contain the additional disks and GPUs. - -Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com, -Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its -credential, labels only placeholders. Existing operator login unchanged. - -Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m, -offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused -email route replaced with verified webhook. nh3-dev system service -beszel-althing forwards JSON via supported postbox CLI, sender/recipient -infra-ops; configurable recipient for later Miranda move. See service README. - -Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread -01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored -85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for -clickable alert deep links. Inbox verification did not mark mail read. - -Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post -office outage detection. Bridge deliberately has no hidden delivery queue; -downstream failure is logged and HTTP 502, not a claimed delivery. diff --git a/persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md b/persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md deleted file mode 100644 index 5f90d93..0000000 --- a/persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3) - -**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴. diff --git a/persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md b/persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md deleted file mode 100644 index 110b0f0..0000000 --- a/persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md +++ /dev/null @@ -1,41 +0,0 @@ -# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days - -Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache -prune`, npm, pip), then a deep dive found the real shape. - -⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session -scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** — -252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance — -**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`. -`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18 -that **overrides** the stock no-age rule, and the cleaner runs daily. - -**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by -content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256` -recorded at training time. - -⚠ **Two corrections I made to myself during the dive, both worth keeping:** I -alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their -mtimes were *that day*, a live session working. And I suspected my own `du`/ -`find` had reset the atime clock and manufactured the "0 would-remove" result; -it had not (`relatime`, and an untouched comparison file still showed an old -atime) — but it was right to check before trusting a number my own measurement -could have created. - -**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB -`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets` -(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% → -72%.** - -⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session -dir is an unsound liveness test: **a directory's mtime does not change when -files are written into its subdirectories.** `dfacccde/` looked 7+ days idle -while `dfacccde/tasks/` was being written continuously. Cost: one lost tool -output; recreated. All four of my safety assertions (right root, right depth, -own session excluded, shutterchute excluded) passed — I checked the paths were -right and never checked the liveness test was sound. **Do not re-run that -predicate.** A correct version checks the deepest recent file, or -cross-references running `claude` PIDs. - -Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev, -while `ssh infra-ops@localhost` has NOPASSWD. diff --git a/persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md b/persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md deleted file mode 100644 index 4e25a7d..0000000 --- a/persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested - -**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`. diff --git a/persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md b/persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md deleted file mode 100644 index 4cea9cd..0000000 --- a/persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md +++ /dev/null @@ -1,39 +0,0 @@ -# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months - -**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi` -(team 4) and `vastblue` (team 5). Verified by reading membership back AND by -exercising it on claude-bot's own token: create/edit/delete in `pfi` all -succeed, `vh/*` correctly still 404s. - -⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a -**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no -namespace-scoped admin for a user namespace. Measured: per-repo `admin` -collaborator grants read but **not** settings (403 on PATCH) — repo settings are -owner-only. So the only working realization of "admin over vh/*" is the -**instance-wide site-admin flag**, which would have given claude-bot the same -blast radius as the token the credential-migration project exists to retire. -Surfaced rather than executed; the operator chose orgs instead. - -**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved, -old paths 301ing, and — verified — **the old ssh remotes still resolve**, since -Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote -living on a redirect depends on the old path staying unclaimed. - -⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as -**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the -operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The -conclusion survived re-testing with a working credential, but the evidence was -worthless. **A credential that authenticates as nobody returns 403 and 404 for -everything, and that is indistinguishable from a permissions answer.** There -are FOUR token files in that directory; the working one for repo work is -**`gitea-token-repo-create`** (`write:organization`, `write:repository`, -`write:user`). - -**Standing authorization (operator, same day):** routine vh-token use for -`vh/*` repo ops no longer gets a flag. Read it from the **vault** — -`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the -disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make -the token low-blast-radius; it stopped the class of work being exceptional. - -Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in -`reference_infra_ops_vh_gitea_token_and_sdk_publish`. diff --git a/persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md b/persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md deleted file mode 100644 index 53f73fe..0000000 --- a/persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug. - -⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`. diff --git a/persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md b/persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md deleted file mode 100644 index 9c41ddb..0000000 --- a/persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass. - -⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`. diff --git a/persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md b/persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md deleted file mode 100644 index 41f2be3..0000000 --- a/persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong. - -⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it. diff --git a/persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md b/persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md deleted file mode 100644 index e9a8d2b..0000000 --- a/persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads - -**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `/tasks/` looks 7+ days idle at `/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs. diff --git a/persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md b/persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md deleted file mode 100644 index 9d87cc9..0000000 --- a/persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-21]` `vh/forgefirm` mirrored - -**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable. diff --git a/persistent-memory.md b/persistent-memory.md index 38d79dc..0ed5269 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-04 ~2335 PT (albok-service moved to its own VM nh3-albok on nh3-pve-2; every hypervisor on PVE 9.2.21 + both PBS on 4.2.7; pfi-ana-webhost destroyed, phasefinal.com on Cloudflare Pages; sudo-io archiver + bulk-data exemption; fv-ml1 root pool cleaned; fv-ml1 Docker move + guest patch campaign parked.)_ +_Last updated: 2026-10-05 ~2035 PT (new SVOS on nh3-svos deployed + accepted: mesh-only https://svos.phasefinal.com, Basic auth `Vuong`, offline shell, hosting bounds, backup drill; backup check now flags fs-freeze failures (DEGRADED, exit 4) and covers nh3-pve-2; fleet guest patching, reboots and Docker engine bumps done except the held list; fv-ml1 Docker on tank; irv-ml1 cleanup.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -13,8 +13,6 @@ _Last updated: 2026-10-04 ~2335 PT (albok-service moved to its own VM nh3-albok ## Repo purpose -- **2026-09-10 Beszel fleet wiring:** all seven requested hosts plus existing corviduo-dev report up. `/tank` and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to **infra-ops**, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See `persistent-memory.d/2026-09-10-beszel-fleet-wiring.md` and `stacks/beszel/README.md`. - Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under @@ -115,21 +113,18 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections carry their own dates._ +_As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections carry their own dates._ -### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) +### 2026-10-05: live now (as of 2026-10-05 ~2035 PT) -- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev, pfi-pteradactyl done 10-05 ~1415–1418); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed, patched, engine done; pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). -- **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. -- **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` -- **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` -- **Scheduled with infra-hermes (do not redo):** 10-05 0815 PT check that sfsrv-ana's (PVE 8) 04:00 backup reached pbs-ana, then `qm delsnapshot` 100/105 `pre-pbs4` (if stale/errored he sends infra-ops URGENT and keeps them); on/after 10-05 0900 drop `pve/root-pre-pve9` on pfi-pve + esh-pve (job `pve9-drop-pre-snapshot`); 10-11 0900 delete the two one-off PBS VM vzdumps. esh-nas-pve `nvme/ROOT@pre-pve9`: destroy ~10-10 (ours). Never `zpool upgrade` any pool yet. -- **Weekly sudo-io archiver** first scheduled run Sun 2026-10-11 05:40 (nh3-dev user timer); bulk data no longer recorded fleet-wide. → `services/sudo-io-archive/README.md` -- **phasefinal.com on Cloudflare Pages; pfi-ana-webhost destroyed; dragonfireacoustics.com is the client's Pages.** → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md` -- **Worldtree memory gate 20261004T143011Z FAIL** (one run filed the forbidden `u-negative-ephemeral` fact; extractor is `gen` on fv-ml1). With worldtree-dev, with the infra timeline (an esh-pve reboot overlapped run 1). The 2026-10-05 0730 run shows whether it repeats. -- **fv-ml1:** root pool 194 G free; lkraven's `~/.cache/huggingface/hub/models--Cloudflare--clef-flash` (18 G, duplicate of tank) was left because bench-jev-2026-10-04 was using it; delete once that bench is done. +- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`); eshpfi pushed through `1210b4a`. +- **New SVOS on nh3-svos (VM 111, nh3-pve-2, 10.100.50.43): DEPLOYED and ACCEPTED by svos-dev 2026-10-05 ~2024** (thread `01M47ET6XH0M7JZSH98Z19SKR4` closed). Separate from the High Seat (`svos.service` :8770 on nh3-dev) and Miranda, which are NOT migrated. Front door `https://svos.phasefinal.com`, mesh only, Basic auth login `Vuong` (Prime has the password; vault `nh3-svos/basic-auth-vuong`). ⚠ **After every SVOS release** (`/opt/svos/current` moves): `sudo /usr/local/sbin/svos-front-inventory`, and keep the previous release dir while it is "previous" (svos-dev's runbook carries both). Only open item is Prime's: whether his iPhone home-screen app keeps the login. → `servers/nh3-svos/README.md` +- **Backup check: first real run with the new layer is 2026-10-06 0800** (`scripts/backup-freshness-alert.sh`, user timer on nh3-dev, alerts infra-hermes). Expect: nh3-pve-2 coverage read cleanly, svos drill B under ACKNOWLEDGED, verdict not DEGRADED. If it says DEGRADED for anything else, that is a real crash-consistent-only backup. +- **Waiting on Prime (updated 2026-10-05 2035):** push Booth commit `ce095c1` (touch-layout fix, `~/development/booth`, browser test + mutation); a trusted HTTPS name for ESH Homepage (https://10.0.50.45 serves Traefik's default cert; the ESH ACME resolver, Route53, is dead); nh3-dev reboot (runs kernel 6.1.0-49, 6.1.0-53 installed; its engine is ALREADY 29.8.2 since 10-05 1413, so the old "reboot + engine bump" wording was stale, checked 2035); Docker engine bumps held on nh3-docker (carries the post office) and esh-docker-vm (NFS D-state history); fv-ml1's held ZFS 2.3.9, nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build). ⚠ **irv-ml1: DO NOT PATCH (Prime).** ana-nas is out of scope. +- **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. +- **Worldtree memory gate 20261004T143011Z FAILED** (one run filed the forbidden `u-negative-ephemeral` fact; an esh-pve reboot overlapped run 1). Whether the 2026-10-05 0730 run repeated it was not checked here; it is a regression check with worldtree-dev, not a gate. +- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm. - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. -- **Live, nothing open:** albok-service 0.1.8 (all three wings reindexed, albok-dev verified); cira-tunnel-watchdog on pfi-tacticalrmm; RustDesk 1.1.16 on ana-docker (OSS refuses "Change ID" by design; see `servers/ana-docker/README.md`); nh3-pve-2 follow-ups done; VM 101 (DC) + VM 104 (nh3-laser) retired. ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01) @@ -262,23 +257,9 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### augaman: face recognition for Cicada -- **v0.1.3 LIVE on esh-ml1:8040** (v0.1.1 first deployed 2026-09-26 2347 PT), `stacks/augaman`, - healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag. - `pytest -m gpu tests/vision` 3/3 PASS on v0.1.3. -- **Gallery backup WIRED + RESTORE-VERIFIED 2026-09-27** (Prime OK'd off-site): restic - daily 0100 PT → rest-server-ana `esh-ml1/`, fail-closed hook, secrets vaulted under - `esh-ml1/etc/restic/`. Identity-level restore matched the live gallery (canary, - snapshot `fd3061a1`). **The backup gate for real enrollments is MET.** -- **Deploy CLOSED by augaman-dev 2026-09-27 0009 PT:** HTTP checks passed; the canary - survived the recreate and was then deleted, so the gallery is empty and ready for real - enrollments. `/recognize` p50 186 ms (1080p, one face). The detector-latency follow-up - is theirs. Their next release pins the container gid to 10001; nothing is owed by infra-ops. -- **The fv-ml1 instance was REMOVED by Prime on 2026-09-27 (~0130 PT)** after the bench. esh-ml1 is the only - instance: in the house, backed up, 48 ms per face on v0.1.3. -- **v0.1.3 went live on both hosts (2026-09-27 ~0105 PT); the fv-ml1 instance was removed later.** Speed bench (`docs/pfi/augaman-speed-bench/`), - server-side one face, v0.1.2 → v0.1.3: esh GPU 144 → 48 ms, fv GPU 75 → 27 ms. CPU mode - REGRESSED (fv CPU6 152 → 205 ms; suspected ORT thread-pool spinning) and is reported to - augaman-dev; the deployment does not use CPU mode. +- **v0.1.3 LIVE on esh-ml1:8040** (`stacks/augaman`), the only instance (fv-ml1's was benched and removed 2026-09-27). Gallery + backup wired and identity-level restore verified (restic → rest-server-ana `esh-ml1/`), so real enrollments are allowed. + 48 ms per face. CPU-mode regression is augaman-dev's. Nothing owed by infra-ops. ### esh-ml1 @@ -293,7 +274,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### Live threads -- git: **eshpfi-management pushed to `6b66207` and worldtree-instance-configs to `b6fdd81` (2026-10-01 ~0418, Prime's go).** Anything after that is unpushed. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- ` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated. +- git: **eshpfi-management pushed through `1210b4a` (2026-10-05 ~2020); worldtree-instance-configs at `b6fdd81`.** ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- ` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated. - nh3-dev root disk was cleaned 2026-09-30 1704 (uv prune, dangling images, old build cache): 86% → 82%. The Beszel 85% alert flaps near the line. - Booth submit-all fix (Prime's report) is LIVE since 2026-09-27 1705, via booth-dev (booth `50bfc7b`). Pushing it is booth's call, per Prime; it is not ours. @@ -302,11 +283,13 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions - `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`. +- `[2026-10-05]` **ESH Homepage also on https://10.0.50.45 (Prime)** — a Traefik file-provider router (`homepage-https.yml`, `tls: {}`) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (`2c4c7ca`) +- `[2026-10-05]` **bench-jev servers killed and removed, their HF cache cleaned (Prime)**; pfi-pteradactyl fixed + patched + engine upgraded (`3429b5c`; manual dockerd + wings restart). - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md` - `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md` - `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md` - `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md` -- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result +- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result — `[2026-10-05 later]` reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight. - `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md` - `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md` - `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml` @@ -315,8 +298,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca - `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md` - `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md` - `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart. -- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). -- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). +- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). — `[2026-10-05]` DONE (entry above). +- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). — `[2026-10-05]` DONE (entry above). - `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98). - `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md` @@ -415,34 +398,11 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca - `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → `persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md` - `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md` -- `[2026-09-21]` **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** — On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one… → `persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md` -- `[2026-09-21]` **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** → `persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md` -- `[2026-09-21]` **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** → `persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md` - `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md` - `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md` -- `[2026-09-21]` ⭐⭐ **claude-bot is an Owner in corviduo/pfi/vastblue; cicada + draupnir moved to `pfi`; vh-token use is now standing-authorized from the vault.** ⚠ `vh` is a USER not an org, so no namespace-scoped admin exists — the only realization of the original ask was site-admin, surfaced rather than executed. ⚠⚠ `~/.config/claude-bot/gitea-token` is DEAD and had been misreporting permissions; the working one is `gitea-token-repo-create`. → `persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md` - -- `[2026-09-21]` ⭐⭐ **nh3-dev 84%→72%: 27 GB reclaimed, and a 252 MB LoRA adapter rescued from `/tmp`, which this box sweeps at 3 days.** `babyyarros` existed nowhere else; hash-verified to smithy before any deletion. ⚠⚠ My 7-day session prune then deleted an ACTIVE session's dir — directory mtime does not reflect subdirectory writes. Do not re-run that predicate. → `persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md` - -- `[2026-09-21]` **Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** — (34c4179, e574b91, 2e08edc). → `persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md` - -- `[2026-09-21]` **Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** — (7fe4102). → `persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md` - -- `[2026-09-21]` **`vh/forgefirm` mirrored** — from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/`… → `persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md` - - - - - - - - - - - ⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24). - `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md` - `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md` @@ -456,7 +416,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md` -_187 older entries archived to archival-memory.md._ +_197 older entries archived to archival-memory.md._ ## Tried and abandoned @@ -484,8 +444,4 @@ _187 older entries archived to archival-memory.md._ - `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button. - `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage. -- `[2026-09-21]` **Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working… → `persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md` - - - -_120 older entries archived to archival-memory.md._ +_121 older entries archived to archival-memory.md._