memory: snapshot — 2026-10-05 SVOS deployment closed; in-flight refreshed; 11 entries archived

This commit is contained in:
vh
2026-10-05 21:04:42 -07:00
parent 1210b4a0d5
commit 1273163173
12 changed files with 196 additions and 212 deletions
+175
View File
@@ -13789,3 +13789,178 @@ Until it ships, the mitigation is entirely sender-side: pipe the body, never `--
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step. **`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
_Archived 2026-10-04._ _Archived 2026-10-04._
## Recent decisions (archived)
# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.
⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
_Archived 2026-10-05._
# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.
⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
_Archived 2026-10-05._
# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.
⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
_Archived 2026-10-05._
# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months
**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi`
(team 4) and `vastblue` (team 5). Verified by reading membership back AND by
exercising it on claude-bot's own token: create/edit/delete in `pfi` all
succeed, `vh/*` correctly still 404s.
⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a
**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no
namespace-scoped admin for a user namespace. Measured: per-repo `admin`
collaborator grants read but **not** settings (403 on PATCH) — repo settings are
owner-only. So the only working realization of "admin over vh/*" is the
**instance-wide site-admin flag**, which would have given claude-bot the same
blast radius as the token the credential-migration project exists to retire.
Surfaced rather than executed; the operator chose orgs instead.
**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved,
old paths 301ing, and — verified — **the old ssh remotes still resolve**, since
Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote
living on a redirect depends on the old path staying unclaimed.
⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as
**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the
operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The
conclusion survived re-testing with a working credential, but the evidence was
worthless. **A credential that authenticates as nobody returns 403 and 404 for
everything, and that is indistinguishable from a permissions answer.** There
are FOUR token files in that directory; the working one for repo work is
**`gitea-token-repo-create`** (`write:organization`, `write:repository`,
`write:user`).
**Standing authorization (operator, same day):** routine vh-token use for
`vh/*` repo ops no longer gets a flag. Read it from the **vault** —
`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the
disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make
the token low-blast-radius; it stopped the class of work being exceptional.
Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in
`reference_infra_ops_vh_gitea_token_and_sdk_publish`.
_Archived 2026-10-05._
# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days
Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache
prune`, npm, pip), then a deep dive found the real shape.
⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session
scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** —
252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance —
**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`.
`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18
that **overrides** the stock no-age rule, and the cleaner runs daily.
**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by
content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256`
recorded at training time.
⚠ **Two corrections I made to myself during the dive, both worth keeping:** I
alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their
mtimes were *that day*, a live session working. And I suspected my own `du`/
`find` had reset the atime clock and manufactured the "0 would-remove" result;
it had not (`relatime`, and an untouched comparison file still showed an old
atime) — but it was right to check before trusting a number my own measurement
could have created.
**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB
`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets`
(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% →
72%.**
⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session
dir is an unsound liveness test: **a directory's mtime does not change when
files are written into its subdirectories.** `dfacccde/` looked 7+ days idle
while `dfacccde/tasks/` was being written continuously. Cost: one lost tool
output; recreated. All four of my safety assertions (right root, right depth,
own session excluded, shutterchute excluded) passed — I checked the paths were
right and never checked the liveness test was sound. **Do not re-run that
predicate.** A correct version checks the deepest recent file, or
cross-references running `claude` PIDs.
Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev,
while `ssh infra-ops@localhost` has NOPASSWD.
_Archived 2026-10-05._
# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested
**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`.
_Archived 2026-10-05._
# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)
**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.
_Archived 2026-10-05._
# `[2026-09-21]` `vh/forgefirm` mirrored
**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.
_Archived 2026-10-05._
# Beszel fleet wiring — 2026-09-10
Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
and asked for GPU usage/power telemetry and the card's health detail.
Completed: all seven requested hosts up, alongside previously registered
corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
healthy but its hub record still pointed at retired 10.100.79.3; fixed to
100.64.0.6. althing-post-office container remained up throughout.
Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
overrides under stacks/beszel/hosts provide read-only mounts. Existing project
directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
traefik-net network even with agent-only profile. No host Docker upgrade.
ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
limits or serving workloads changed. GPU watts do not size a whole-host PSU.
Homepage uses existing discovery labels and version-2 widget; verified one
card and live authenticated data. This overview shows systems/up only;
reachability is not a degraded-health score. Per-system widget can expose
CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
credential, labels only placeholders. Existing operator login unchanged.
Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
email route replaced with verified webhook. nh3-dev system service
beszel-althing forwards JSON via supported postbox CLI, sender/recipient
infra-ops; configurable recipient for later Miranda move. See service README.
Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
clickable alert deep links. Inbox verification did not mark mail read.
Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
office outage detection. Bridge deliberately has no hidden delivery queue;
downstream failure is logged and HTTP 502, not a claimed delivery.
_Archived 2026-10-05._
- (orphaned continuation of an earlier-archived domain-transfer entry; its dated head line was archived before this fragment)
⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
_Archived 2026-10-05._
## Tried and abandoned (archived)
# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads
**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `<id>/tasks/` looks 7+ days idle at `<id>/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs.
_Archived 2026-10-05._
@@ -1,46 +0,0 @@
# Beszel fleet wiring — 2026-09-10
Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
and asked for GPU usage/power telemetry and the card's health detail.
Completed: all seven requested hosts up, alongside previously registered
corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
healthy but its hub record still pointed at retired 10.100.79.3; fixed to
100.64.0.6. althing-post-office container remained up throughout.
Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
overrides under stacks/beszel/hosts provide read-only mounts. Existing project
directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
traefik-net network even with agent-only profile. No host Docker upgrade.
ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
limits or serving workloads changed. GPU watts do not size a whole-host PSU.
Homepage uses existing discovery labels and version-2 widget; verified one
card and live authenticated data. This overview shows systems/up only;
reachability is not a degraded-health score. Per-system widget can expose
CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
credential, labels only placeholders. Existing operator login unchanged.
Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
email route replaced with verified webhook. nh3-dev system service
beszel-althing forwards JSON via supported postbox CLI, sender/recipient
infra-ops; configurable recipient for later Miranda move. See service README.
Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
clickable alert deep links. Inbox verification did not mark mail read.
Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
office outage detection. Bridge deliberately has no hidden delivery queue;
downstream failure is logged and HTTP 502, not a claimed delivery.
@@ -1,3 +0,0 @@
# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)
**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.
@@ -1,41 +0,0 @@
# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days
Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache
prune`, npm, pip), then a deep dive found the real shape.
⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session
scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** —
252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance —
**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`.
`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18
that **overrides** the stock no-age rule, and the cleaner runs daily.
**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by
content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256`
recorded at training time.
⚠ **Two corrections I made to myself during the dive, both worth keeping:** I
alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their
mtimes were *that day*, a live session working. And I suspected my own `du`/
`find` had reset the atime clock and manufactured the "0 would-remove" result;
it had not (`relatime`, and an untouched comparison file still showed an old
atime) — but it was right to check before trusting a number my own measurement
could have created.
**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB
`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets`
(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% →
72%.**
⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session
dir is an unsound liveness test: **a directory's mtime does not change when
files are written into its subdirectories.** `dfacccde/` looked 7+ days idle
while `dfacccde/tasks/` was being written continuously. Cost: one lost tool
output; recreated. All four of my safety assertions (right root, right depth,
own session excluded, shutterchute excluded) passed — I checked the paths were
right and never checked the liveness test was sound. **Do not re-run that
predicate.** A correct version checks the deepest recent file, or
cross-references running `claude` PIDs.
Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev,
while `ssh infra-ops@localhost` has NOPASSWD.
@@ -1,3 +0,0 @@
# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested
**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`.
@@ -1,39 +0,0 @@
# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months
**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi`
(team 4) and `vastblue` (team 5). Verified by reading membership back AND by
exercising it on claude-bot's own token: create/edit/delete in `pfi` all
succeed, `vh/*` correctly still 404s.
⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a
**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no
namespace-scoped admin for a user namespace. Measured: per-repo `admin`
collaborator grants read but **not** settings (403 on PATCH) — repo settings are
owner-only. So the only working realization of "admin over vh/*" is the
**instance-wide site-admin flag**, which would have given claude-bot the same
blast radius as the token the credential-migration project exists to retire.
Surfaced rather than executed; the operator chose orgs instead.
**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved,
old paths 301ing, and — verified — **the old ssh remotes still resolve**, since
Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote
living on a redirect depends on the old path staying unclaimed.
⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as
**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the
operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The
conclusion survived re-testing with a working credential, but the evidence was
worthless. **A credential that authenticates as nobody returns 403 and 404 for
everything, and that is indistinguishable from a permissions answer.** There
are FOUR token files in that directory; the working one for repo work is
**`gitea-token-repo-create`** (`write:organization`, `write:repository`,
`write:user`).
**Standing authorization (operator, same day):** routine vh-token use for
`vh/*` repo ops no longer gets a flag. Read it from the **vault** —
`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the
disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make
the token low-blast-radius; it stopped the class of work being exceptional.
Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in
`reference_infra_ops_vh_gitea_token_and_sdk_publish`.
@@ -1,3 +0,0 @@
# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.
⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
@@ -1,3 +0,0 @@
# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.
⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
@@ -1,3 +0,0 @@
# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.
⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
@@ -1,3 +0,0 @@
# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads
**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `<id>/tasks/` looks 7+ days idle at `<id>/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs.
@@ -1,3 +0,0 @@
# `[2026-09-21]` `vh/forgefirm` mirrored
**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.
+21 -65
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management # Persistent memory — eshpfi-management
_Last updated: 2026-10-04 ~2335 PT (albok-service moved to its own VM nh3-albok on nh3-pve-2; every hypervisor on PVE 9.2.21 + both PBS on 4.2.7; pfi-ana-webhost destroyed, phasefinal.com on Cloudflare Pages; sudo-io archiver + bulk-data exemption; fv-ml1 root pool cleaned; fv-ml1 Docker move + guest patch campaign parked.)_ _Last updated: 2026-10-05 ~2035 PT (new SVOS on nh3-svos deployed + accepted: mesh-only https://svos.phasefinal.com, Basic auth `Vuong`, offline shell, hosting bounds, backup drill; backup check now flags fs-freeze failures (DEGRADED, exit 4) and covers nh3-pve-2; fleet guest patching, reboots and Docker engine bumps done except the held list; fv-ml1 Docker on tank; irv-ml1 cleanup.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -13,8 +13,6 @@ _Last updated: 2026-10-04 ~2335 PT (albok-service moved to its own VM nh3-albok
## Repo purpose ## Repo purpose
- **2026-09-10 Beszel fleet wiring:** all seven requested hosts plus existing corviduo-dev report up. `/tank` and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to **infra-ops**, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See `persistent-memory.d/2026-09-10-beszel-fleet-wiring.md` and `stacks/beszel/README.md`.
Reference workspace for PFI infrastructure: server inventory, canonical Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under copies of compose files live on the servers under
@@ -115,21 +113,18 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight ## Current state / in-flight
_As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections carry their own dates._ _As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections carry their own dates._
### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) ### 2026-10-05: live now (as of 2026-10-05 ~2035 PT)
- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev, pfi-pteradactyl done 10-05 ~1415–1418); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed, patched, engine done; pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). - **Nothing is mid-action.** No claims held (`scripts/ops-log claims`); eshpfi pushed through `1210b4a`.
- **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. - **New SVOS on nh3-svos (VM 111, nh3-pve-2, 10.100.50.43): DEPLOYED and ACCEPTED by svos-dev 2026-10-05 ~2024** (thread `01M47ET6XH0M7JZSH98Z19SKR4` closed). Separate from the High Seat (`svos.service` :8770 on nh3-dev) and Miranda, which are NOT migrated. Front door `https://svos.phasefinal.com`, mesh only, Basic auth login `Vuong` (Prime has the password; vault `nh3-svos/basic-auth-vuong`). ⚠ **After every SVOS release** (`/opt/svos/current` moves): `sudo /usr/local/sbin/svos-front-inventory`, and keep the previous release dir while it is "previous" (svos-dev's runbook carries both). Only open item is Prime's: whether his iPhone home-screen app keeps the login. → `servers/nh3-svos/README.md`
- **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - **Backup check: first real run with the new layer is 2026-10-06 0800** (`scripts/backup-freshness-alert.sh`, user timer on nh3-dev, alerts infra-hermes). Expect: nh3-pve-2 coverage read cleanly, svos drill B under ACKNOWLEDGED, verdict not DEGRADED. If it says DEGRADED for anything else, that is a real crash-consistent-only backup.
- **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` - **Waiting on Prime (updated 2026-10-05 2035):** push Booth commit `ce095c1` (touch-layout fix, `~/development/booth`, browser test + mutation); a trusted HTTPS name for ESH Homepage (https://10.0.50.45 serves Traefik's default cert; the ESH ACME resolver, Route53, is dead); nh3-dev reboot (runs kernel 6.1.0-49, 6.1.0-53 installed; its engine is ALREADY 29.8.2 since 10-05 1413, so the old "reboot + engine bump" wording was stale, checked 2035); Docker engine bumps held on nh3-docker (carries the post office) and esh-docker-vm (NFS D-state history); fv-ml1's held ZFS 2.3.9, nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build). ⚠ **irv-ml1: DO NOT PATCH (Prime).** ana-nas is out of scope.
- **Scheduled with infra-hermes (do not redo):** 10-05 0815 PT check that sfsrv-ana's (PVE 8) 04:00 backup reached pbs-ana, then `qm delsnapshot` 100/105 `pre-pbs4` (if stale/errored he sends infra-ops URGENT and keeps them); on/after 10-05 0900 drop `pve/root-pre-pve9` on pfi-pve + esh-pve (job `pve9-drop-pre-snapshot`); 10-11 0900 delete the two one-off PBS VM vzdumps. esh-nas-pve `nvme/ROOT@pre-pve9`: destroy ~10-10 (ours). Never `zpool upgrade` any pool yet. - **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
- **Weekly sudo-io archiver** first scheduled run Sun 2026-10-11 05:40 (nh3-dev user timer); bulk data no longer recorded fleet-wide. → `services/sudo-io-archive/README.md` - **Worldtree memory gate 20261004T143011Z FAILED** (one run filed the forbidden `u-negative-ephemeral` fact; an esh-pve reboot overlapped run 1). Whether the 2026-10-05 0730 run repeated it was not checked here; it is a regression check with worldtree-dev, not a gate.
- **phasefinal.com on Cloudflare Pages; pfi-ana-webhost destroyed; dragonfireacoustics.com is the client's Pages.** → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md` - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm.
- **Worldtree memory gate 20261004T143011Z FAIL** (one run filed the forbidden `u-negative-ephemeral` fact; extractor is `gen` on fv-ml1). With worldtree-dev, with the infra timeline (an esh-pve reboot overlapped run 1). The 2026-10-05 0730 run shows whether it repeats.
- **fv-ml1:** root pool 194 G free; lkraven's `~/.cache/huggingface/hub/models--Cloudflare--clef-flash` (18 G, duplicate of tank) was left because bench-jev-2026-10-04 was using it; delete once that bench is done.
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
- **Live, nothing open:** albok-service 0.1.8 (all three wings reindexed, albok-dev verified); cira-tunnel-watchdog on pfi-tacticalrmm; RustDesk 1.1.16 on ana-docker (OSS refuses "Change ID" by design; see `servers/ana-docker/README.md`); nh3-pve-2 follow-ups done; VM 101 (DC) + VM 104 (nh3-laser) retired.
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01) ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
@@ -262,23 +257,9 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
### augaman: face recognition for Cicada ### augaman: face recognition for Cicada
- **v0.1.3 LIVE on esh-ml1:8040** (v0.1.1 first deployed 2026-09-26 2347 PT), `stacks/augaman`, - **v0.1.3 LIVE on esh-ml1:8040** (`stacks/augaman`), the only instance (fv-ml1's was benched and removed 2026-09-27). Gallery
healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag. backup wired and identity-level restore verified (restic → rest-server-ana `esh-ml1/`), so real enrollments are allowed.
`pytest -m gpu tests/vision` 3/3 PASS on v0.1.3. 48 ms per face. CPU-mode regression is augaman-dev's. Nothing owed by infra-ops.
- **Gallery backup WIRED + RESTORE-VERIFIED 2026-09-27** (Prime OK'd off-site): restic
daily 0100 PT → rest-server-ana `esh-ml1/`, fail-closed hook, secrets vaulted under
`esh-ml1/etc/restic/`. Identity-level restore matched the live gallery (canary,
snapshot `fd3061a1`). **The backup gate for real enrollments is MET.**
- **Deploy CLOSED by augaman-dev 2026-09-27 0009 PT:** HTTP checks passed; the canary
survived the recreate and was then deleted, so the gallery is empty and ready for real
enrollments. `/recognize` p50 186 ms (1080p, one face). The detector-latency follow-up
is theirs. Their next release pins the container gid to 10001; nothing is owed by infra-ops.
- **The fv-ml1 instance was REMOVED by Prime on 2026-09-27 (~0130 PT)** after the bench. esh-ml1 is the only
instance: in the house, backed up, 48 ms per face on v0.1.3.
- **v0.1.3 went live on both hosts (2026-09-27 ~0105 PT); the fv-ml1 instance was removed later.** Speed bench (`docs/pfi/augaman-speed-bench/`),
server-side one face, v0.1.2 → v0.1.3: esh GPU 144 → 48 ms, fv GPU 75 → 27 ms. CPU mode
REGRESSED (fv CPU6 152 → 205 ms; suspected ORT thread-pool spinning) and is reported to
augaman-dev; the deployment does not use CPU mode.
### esh-ml1 ### esh-ml1
@@ -293,7 +274,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
### Live threads ### Live threads
- git: **eshpfi-management pushed to `6b66207` and worldtree-instance-configs to `b6fdd81` (2026-10-01 ~0418, Prime's go).** Anything after that is unpushed. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated. - git: **eshpfi-management pushed through `1210b4a` (2026-10-05 ~2020); worldtree-instance-configs at `b6fdd81`.** ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated.
- nh3-dev root disk was cleaned 2026-09-30 1704 (uv prune, dangling images, old build cache): 86% → 82%. The Beszel 85% alert flaps near the line. - nh3-dev root disk was cleaned 2026-09-30 1704 (uv prune, dangling images, old build cache): 86% → 82%. The Beszel 85% alert flaps near the line.
- Booth submit-all fix (Prime's report) is LIVE since 2026-09-27 1705, via booth-dev (booth `50bfc7b`). - Booth submit-all fix (Prime's report) is LIVE since 2026-09-27 1705, via booth-dev (booth `50bfc7b`).
Pushing it is booth's call, per Prime; it is not ours. Pushing it is booth's call, per Prime; it is not ours.
@@ -302,11 +283,13 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
## Recent decisions ## Recent decisions
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`. - `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
- `[2026-10-05]` **ESH Homepage also on https://10.0.50.45 (Prime)** — a Traefik file-provider router (`homepage-https.yml`, `tls: {}`) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (`2c4c7ca`)
- `[2026-10-05]` **bench-jev servers killed and removed, their HF cache cleaned (Prime)**; pfi-pteradactyl fixed + patched + engine upgraded (`3429b5c`; manual dockerd + wings restart).
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md` - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md` - `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md` - `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
- `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md` - `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md`
- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result - `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result — `[2026-10-05 later]` reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md` - `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md` - `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml` - `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
@@ -315,8 +298,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md` - `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md` - `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart. - `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). - `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). — `[2026-10-05]` DONE (entry above).
- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). - `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). — `[2026-10-05]` DONE (entry above).
- `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98). - `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98).
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md` - `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
@@ -415,34 +398,11 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
- `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → `persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md` - `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → `persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md`
- `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md` - `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`
- `[2026-09-21]` **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** — On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one… → `persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md`
- `[2026-09-21]` **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** → `persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md`
- `[2026-09-21]` **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** → `persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md`
- `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md` - `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md`
- `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md` - `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md`
- `[2026-09-21]` ⭐⭐ **claude-bot is an Owner in corviduo/pfi/vastblue; cicada + draupnir moved to `pfi`; vh-token use is now standing-authorized from the vault.** ⚠ `vh` is a USER not an org, so no namespace-scoped admin exists — the only realization of the original ask was site-admin, surfaced rather than executed. ⚠⚠ `~/.config/claude-bot/gitea-token` is DEAD and had been misreporting permissions; the working one is `gitea-token-repo-create`. → `persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md`
- `[2026-09-21]` ⭐⭐ **nh3-dev 84%→72%: 27 GB reclaimed, and a 252 MB LoRA adapter rescued from `/tmp`, which this box sweeps at 3 days.** `babyyarros` existed nowhere else; hash-verified to smithy before any deletion. ⚠⚠ My 7-day session prune then deleted an ACTIVE session's dir — directory mtime does not reflect subdirectory writes. Do not re-run that predicate. → `persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md`
- `[2026-09-21]` **Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** — (34c4179, e574b91, 2e08edc). → `persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md`
- `[2026-09-21]` **Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** — (7fe4102). → `persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md`
- `[2026-09-21]` **`vh/forgefirm` mirrored** — from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/`… → `persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md`
⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md` - `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md` - `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
@@ -456,7 +416,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md` - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
_187 older entries archived to archival-memory.md._ _197 older entries archived to archival-memory.md._
## Tried and abandoned ## Tried and abandoned
@@ -484,8 +444,4 @@ _187 older entries archived to archival-memory.md._
- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button. - `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage. - `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
- `[2026-09-21]` **Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working… → `persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md` _121 older entries archived to archival-memory.md._
_120 older entries archived to archival-memory.md._