memory: snapshot — 2026-10-05 SVOS deployment closed; in-flight refreshed; 11 entries archived

2026-10-05 21:04:42 -07:00
parent 1210b4a0d5
commit 1273163173
12 changed files with 196 additions and 212 deletions
@@ -1,46 +0,0 @@
-# Beszel fleet wiring — 2026-09-10
-
-Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
-for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
-and asked for GPU usage/power telemetry and the card's health detail.
-
-Completed: all seven requested hosts up, alongside previously registered
-corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
-contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
-healthy but its hub record still pointed at retired 10.100.79.3; fixed to
-100.64.0.6. althing-post-office container remained up throughout.
-
-Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
-overrides under stacks/beszel/hosts provide read-only mounts. Existing project
-directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
-and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
-traefik-net network even with agent-only profile. No host Docker upgrade.
-
-ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
-to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
-GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
-limits or serving workloads changed. GPU watts do not size a whole-host PSU.
-
-Homepage uses existing discovery labels and version-2 widget; verified one
-card and live authenticated data. This overview shows systems/up only;
-reachability is not a degraded-health score. Per-system widget can expose
-CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
-
-Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
-Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
-credential, labels only placeholders. Existing operator login unchanged.
-
-Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
-offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
-email route replaced with verified webhook. nh3-dev system service
-beszel-althing forwards JSON via supported postbox CLI, sender/recipient
-infra-ops; configurable recipient for later Miranda move. See service README.
-
-Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
-01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
-85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
-clickable alert deep links. Inbox verification did not mark mail read.
-
-Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
-office outage detection. Bridge deliberately has no hidden delivery queue;
-downstream failure is logged and HTTP 502, not a claimed delivery.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)
-
-**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.
@@ -1,41 +0,0 @@
-# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days
-
-Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache
-prune`, npm, pip), then a deep dive found the real shape.
-
-⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session
-scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** —
-252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance —
-**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`.
-`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18
-that **overrides** the stock no-age rule, and the cleaner runs daily.
-
-**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by
-content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256`
-recorded at training time.
-
-⚠ **Two corrections I made to myself during the dive, both worth keeping:** I
-alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their
-mtimes were *that day*, a live session working. And I suspected my own `du`/
-`find` had reset the atime clock and manufactured the "0 would-remove" result;
-it had not (`relatime`, and an untouched comparison file still showed an old
-atime) — but it was right to check before trusting a number my own measurement
-could have created.
-
-**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB
-`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets`
-(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% →
-72%.**
-
-⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session
-dir is an unsound liveness test: **a directory's mtime does not change when
-files are written into its subdirectories.** `dfacccde/` looked 7+ days idle
-while `dfacccde/tasks/` was being written continuously. Cost: one lost tool
-output; recreated. All four of my safety assertions (right root, right depth,
-own session excluded, shutterchute excluded) passed — I checked the paths were
-right and never checked the liveness test was sound. **Do not re-run that
-predicate.** A correct version checks the deepest recent file, or
-cross-references running `claude` PIDs.
-
-Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev,
-while `ssh infra-ops@localhost` has NOPASSWD.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested
-
-**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`.
@@ -1,39 +0,0 @@
-# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months
-
-**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi`
-(team 4) and `vastblue` (team 5). Verified by reading membership back AND by
-exercising it on claude-bot's own token: create/edit/delete in `pfi` all
-succeed, `vh/*` correctly still 404s.
-
-⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a
-**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no
-namespace-scoped admin for a user namespace. Measured: per-repo `admin`
-collaborator grants read but **not** settings (403 on PATCH) — repo settings are
-owner-only. So the only working realization of "admin over vh/*" is the
-**instance-wide site-admin flag**, which would have given claude-bot the same
-blast radius as the token the credential-migration project exists to retire.
-Surfaced rather than executed; the operator chose orgs instead.
-
-**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved,
-old paths 301ing, and — verified — **the old ssh remotes still resolve**, since
-Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote
-living on a redirect depends on the old path staying unclaimed.
-
-⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as
-**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the
-operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The
-conclusion survived re-testing with a working credential, but the evidence was
-worthless. **A credential that authenticates as nobody returns 403 and 404 for
-everything, and that is indistinguishable from a permissions answer.** There
-are FOUR token files in that directory; the working one for repo work is
-**`gitea-token-repo-create`** (`write:organization`, `write:repository`,
-`write:user`).
-
-**Standing authorization (operator, same day):** routine vh-token use for
-`vh/*` repo ops no longer gets a flag. Read it from the **vault** —
-`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the
-disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make
-the token low-blast-radius; it stopped the class of work being exceptional.
-
-Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in
-`reference_infra_ops_vh_gitea_token_and_sdk_publish`.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.
-
-⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.
-
-⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.
-
-⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads
-
-**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `<id>/tasks/` looks 7+ days idle at `<id>/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs.
@@ -1,3 +0,0 @@
-# `[2026-09-21]` `vh/forgefirm` mirrored
-
-**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.