memory: snapshot — 2026-10-05 SVOS deployment closed; in-flight refreshed; 11 entries archived
This commit is contained in:
@@ -13789,3 +13789,178 @@ Until it ships, the mitigation is entirely sender-side: pipe the body, never `--
|
|||||||
|
|
||||||
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
||||||
_Archived 2026-10-04._
|
_Archived 2026-10-04._
|
||||||
|
|
||||||
|
## Recent decisions (archived)
|
||||||
|
|
||||||
|
# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.
|
||||||
|
|
||||||
|
⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.
|
||||||
|
|
||||||
|
⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.
|
||||||
|
|
||||||
|
⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months
|
||||||
|
|
||||||
|
**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi`
|
||||||
|
(team 4) and `vastblue` (team 5). Verified by reading membership back AND by
|
||||||
|
exercising it on claude-bot's own token: create/edit/delete in `pfi` all
|
||||||
|
succeed, `vh/*` correctly still 404s.
|
||||||
|
|
||||||
|
⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a
|
||||||
|
**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no
|
||||||
|
namespace-scoped admin for a user namespace. Measured: per-repo `admin`
|
||||||
|
collaborator grants read but **not** settings (403 on PATCH) — repo settings are
|
||||||
|
owner-only. So the only working realization of "admin over vh/*" is the
|
||||||
|
**instance-wide site-admin flag**, which would have given claude-bot the same
|
||||||
|
blast radius as the token the credential-migration project exists to retire.
|
||||||
|
Surfaced rather than executed; the operator chose orgs instead.
|
||||||
|
|
||||||
|
**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved,
|
||||||
|
old paths 301ing, and — verified — **the old ssh remotes still resolve**, since
|
||||||
|
Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote
|
||||||
|
living on a redirect depends on the old path staying unclaimed.
|
||||||
|
|
||||||
|
⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as
|
||||||
|
**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the
|
||||||
|
operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The
|
||||||
|
conclusion survived re-testing with a working credential, but the evidence was
|
||||||
|
worthless. **A credential that authenticates as nobody returns 403 and 404 for
|
||||||
|
everything, and that is indistinguishable from a permissions answer.** There
|
||||||
|
are FOUR token files in that directory; the working one for repo work is
|
||||||
|
**`gitea-token-repo-create`** (`write:organization`, `write:repository`,
|
||||||
|
`write:user`).
|
||||||
|
|
||||||
|
**Standing authorization (operator, same day):** routine vh-token use for
|
||||||
|
`vh/*` repo ops no longer gets a flag. Read it from the **vault** —
|
||||||
|
`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the
|
||||||
|
disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make
|
||||||
|
the token low-blast-radius; it stopped the class of work being exceptional.
|
||||||
|
|
||||||
|
Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in
|
||||||
|
`reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days
|
||||||
|
|
||||||
|
Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache
|
||||||
|
prune`, npm, pip), then a deep dive found the real shape.
|
||||||
|
|
||||||
|
⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session
|
||||||
|
scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** —
|
||||||
|
252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance —
|
||||||
|
**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`.
|
||||||
|
`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18
|
||||||
|
that **overrides** the stock no-age rule, and the cleaner runs daily.
|
||||||
|
|
||||||
|
**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by
|
||||||
|
content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256`
|
||||||
|
recorded at training time.
|
||||||
|
|
||||||
|
⚠ **Two corrections I made to myself during the dive, both worth keeping:** I
|
||||||
|
alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their
|
||||||
|
mtimes were *that day*, a live session working. And I suspected my own `du`/
|
||||||
|
`find` had reset the atime clock and manufactured the "0 would-remove" result;
|
||||||
|
it had not (`relatime`, and an untouched comparison file still showed an old
|
||||||
|
atime) — but it was right to check before trusting a number my own measurement
|
||||||
|
could have created.
|
||||||
|
|
||||||
|
**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB
|
||||||
|
`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets`
|
||||||
|
(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% →
|
||||||
|
72%.**
|
||||||
|
|
||||||
|
⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session
|
||||||
|
dir is an unsound liveness test: **a directory's mtime does not change when
|
||||||
|
files are written into its subdirectories.** `dfacccde/` looked 7+ days idle
|
||||||
|
while `dfacccde/tasks/` was being written continuously. Cost: one lost tool
|
||||||
|
output; recreated. All four of my safety assertions (right root, right depth,
|
||||||
|
own session excluded, shutterchute excluded) passed — I checked the paths were
|
||||||
|
right and never checked the liveness test was sound. **Do not re-run that
|
||||||
|
predicate.** A correct version checks the deepest recent file, or
|
||||||
|
cross-references running `claude` PIDs.
|
||||||
|
|
||||||
|
Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev,
|
||||||
|
while `ssh infra-ops@localhost` has NOPASSWD.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested
|
||||||
|
|
||||||
|
**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)
|
||||||
|
|
||||||
|
**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# `[2026-09-21]` `vh/forgefirm` mirrored
|
||||||
|
|
||||||
|
**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
# Beszel fleet wiring — 2026-09-10
|
||||||
|
|
||||||
|
Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
|
||||||
|
for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
|
||||||
|
and asked for GPU usage/power telemetry and the card's health detail.
|
||||||
|
|
||||||
|
Completed: all seven requested hosts up, alongside previously registered
|
||||||
|
corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
|
||||||
|
contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
|
||||||
|
healthy but its hub record still pointed at retired 10.100.79.3; fixed to
|
||||||
|
100.64.0.6. althing-post-office container remained up throughout.
|
||||||
|
|
||||||
|
Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
|
||||||
|
overrides under stacks/beszel/hosts provide read-only mounts. Existing project
|
||||||
|
directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
|
||||||
|
and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
|
||||||
|
traefik-net network even with agent-only profile. No host Docker upgrade.
|
||||||
|
|
||||||
|
ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
|
||||||
|
to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
|
||||||
|
GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
|
||||||
|
limits or serving workloads changed. GPU watts do not size a whole-host PSU.
|
||||||
|
|
||||||
|
Homepage uses existing discovery labels and version-2 widget; verified one
|
||||||
|
card and live authenticated data. This overview shows systems/up only;
|
||||||
|
reachability is not a degraded-health score. Per-system widget can expose
|
||||||
|
CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
|
||||||
|
|
||||||
|
Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
|
||||||
|
Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
|
||||||
|
credential, labels only placeholders. Existing operator login unchanged.
|
||||||
|
|
||||||
|
Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
|
||||||
|
offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
|
||||||
|
email route replaced with verified webhook. nh3-dev system service
|
||||||
|
beszel-althing forwards JSON via supported postbox CLI, sender/recipient
|
||||||
|
infra-ops; configurable recipient for later Miranda move. See service README.
|
||||||
|
|
||||||
|
Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
|
||||||
|
01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
|
||||||
|
85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
|
||||||
|
clickable alert deep links. Inbox verification did not mark mail read.
|
||||||
|
|
||||||
|
Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
|
||||||
|
office outage detection. Bridge deliberately has no hidden delivery queue;
|
||||||
|
downstream failure is logged and HTTP 502, not a claimed delivery.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
- (orphaned continuation of an earlier-archived domain-transfer entry; its dated head line was archived before this fragment)
|
||||||
|
⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|
||||||
|
## Tried and abandoned (archived)
|
||||||
|
|
||||||
|
# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads
|
||||||
|
|
||||||
|
**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `<id>/tasks/` looks 7+ days idle at `<id>/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs.
|
||||||
|
_Archived 2026-10-05._
|
||||||
|
|||||||
@@ -1,46 +0,0 @@
|
|||||||
# Beszel fleet wiring — 2026-09-10
|
|
||||||
|
|
||||||
Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
|
|
||||||
for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
|
|
||||||
and asked for GPU usage/power telemetry and the card's health detail.
|
|
||||||
|
|
||||||
Completed: all seven requested hosts up, alongside previously registered
|
|
||||||
corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
|
|
||||||
contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
|
|
||||||
healthy but its hub record still pointed at retired 10.100.79.3; fixed to
|
|
||||||
100.64.0.6. althing-post-office container remained up throughout.
|
|
||||||
|
|
||||||
Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
|
|
||||||
overrides under stacks/beszel/hosts provide read-only mounts. Existing project
|
|
||||||
directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
|
|
||||||
and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
|
|
||||||
traefik-net network even with agent-only profile. No host Docker upgrade.
|
|
||||||
|
|
||||||
ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
|
|
||||||
to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
|
|
||||||
GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
|
|
||||||
limits or serving workloads changed. GPU watts do not size a whole-host PSU.
|
|
||||||
|
|
||||||
Homepage uses existing discovery labels and version-2 widget; verified one
|
|
||||||
card and live authenticated data. This overview shows systems/up only;
|
|
||||||
reachability is not a degraded-health score. Per-system widget can expose
|
|
||||||
CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
|
|
||||||
|
|
||||||
Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
|
|
||||||
Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
|
|
||||||
credential, labels only placeholders. Existing operator login unchanged.
|
|
||||||
|
|
||||||
Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
|
|
||||||
offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
|
|
||||||
email route replaced with verified webhook. nh3-dev system service
|
|
||||||
beszel-althing forwards JSON via supported postbox CLI, sender/recipient
|
|
||||||
infra-ops; configurable recipient for later Miranda move. See service README.
|
|
||||||
|
|
||||||
Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
|
|
||||||
01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
|
|
||||||
85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
|
|
||||||
clickable alert deep links. Inbox verification did not mark mail read.
|
|
||||||
|
|
||||||
Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
|
|
||||||
office outage detection. Bridge deliberately has no hidden delivery queue;
|
|
||||||
downstream failure is logged and HTTP 502, not a claimed delivery.
|
|
||||||
-3
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)
|
|
||||||
|
|
||||||
**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
# `[2026-09-21]` nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days
|
|
||||||
|
|
||||||
Beszel alerted root >85%. Reclaimed **7 GB** from regenerable caches (`uv cache
|
|
||||||
prune`, npm, pip), then a deep dive found the real shape.
|
|
||||||
|
|
||||||
⚠⚠ **The headline was not disk.** `/tmp` held **19 GB** of Claude Code session
|
|
||||||
scratchpads, and inside one of them sat the **`babyyarros` LoRA adapter** —
|
|
||||||
252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance —
|
|
||||||
**existing nowhere else**: not on `/mnt/smithy`, not under `~/development`.
|
|
||||||
`/etc/tmpfiles.d/tmp.conf` sets `D /tmp … 3d`, an admin file from 2026-07-18
|
|
||||||
that **overrides** the stock no-age rule, and the cleaner runs daily.
|
|
||||||
|
|
||||||
**Rescued** to `/mnt/smithy/adapter-rescue/babyyarros-20260921`, verified by
|
|
||||||
content: sha256 `63fda6cc…` matching both the source and the `adapter.sha256`
|
|
||||||
recorded at training time.
|
|
||||||
|
|
||||||
⚠ **Two corrections I made to myself during the dive, both worth keeping:** I
|
|
||||||
alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their
|
|
||||||
mtimes were *that day*, a live session working. And I suspected my own `du`/
|
|
||||||
`find` had reset the atime clock and manufactured the "0 would-remove" result;
|
|
||||||
it had not (`relatime`, and an untouched comparison file still showed an old
|
|
||||||
atime) — but it was right to check before trusting a number my own measurement
|
|
||||||
could have created.
|
|
||||||
|
|
||||||
**Operator-authorized deletions:** 7.6 GB duplicate Qwen base shards, 12 GB
|
|
||||||
`models-staging/retro-diffusion` (cold since 09 Aug), 7.5 GB `splat-assets`
|
|
||||||
(cold since 15 Aug), 178 session dirs idle 7d+. **39 GB → 66 GB free, 84% →
|
|
||||||
72%.**
|
|
||||||
|
|
||||||
⚠⚠ **MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY.** `-mtime +7` on a session
|
|
||||||
dir is an unsound liveness test: **a directory's mtime does not change when
|
|
||||||
files are written into its subdirectories.** `dfacccde/` looked 7+ days idle
|
|
||||||
while `dfacccde/tasks/` was being written continuously. Cost: one lost tool
|
|
||||||
output; recreated. All four of my safety assertions (right root, right depth,
|
|
||||||
own session excluded, shutterchute excluded) passed — I checked the paths were
|
|
||||||
right and never checked the liveness test was sound. **Do not re-run that
|
|
||||||
predicate.** A correct version checks the deepest recent file, or
|
|
||||||
cross-references running `claude` PIDs.
|
|
||||||
|
|
||||||
Also found: `sudo -n` requires a password as **lkraven locally** on nh3-dev,
|
|
||||||
while `ssh infra-ops@localhost` has NOPASSWD.
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested
|
|
||||||
|
|
||||||
**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# `[2026-09-21]` claude-bot became an org Owner, repos moved to `pfi`, and a dead token had been misreporting permissions for months
|
|
||||||
|
|
||||||
**Operator ruling:** claude-bot is an **Owner** in `corviduo` (team 1), `pfi`
|
|
||||||
(team 4) and `vastblue` (team 5). Verified by reading membership back AND by
|
|
||||||
exercising it on claude-bot's own token: create/edit/delete in `pfi` all
|
|
||||||
succeed, `vh/*` correctly still 404s.
|
|
||||||
|
|
||||||
⚠ **Why not "admin on vh/*", which is what was originally asked:** `vh` is a
|
|
||||||
**USER account (id 1), not an organization** — `/orgs/vh` 404s. Gitea has no
|
|
||||||
namespace-scoped admin for a user namespace. Measured: per-repo `admin`
|
|
||||||
collaborator grants read but **not** settings (403 on PATCH) — repo settings are
|
|
||||||
owner-only. So the only working realization of "admin over vh/*" is the
|
|
||||||
**instance-wide site-admin flag**, which would have given claude-bot the same
|
|
||||||
blast radius as the token the credential-migration project exists to retire.
|
|
||||||
Surfaced rather than executed; the operator chose orgs instead.
|
|
||||||
|
|
||||||
**`vh/cicada` and `vh/draupnir` transferred into `pfi`** with SHAs preserved,
|
|
||||||
old paths 301ing, and — verified — **the old ssh remotes still resolve**, since
|
|
||||||
Gitea redirects git-over-ssh and not just the web URL. Repoint anyway: a remote
|
|
||||||
living on a redirect depends on the old path staying unclaimed.
|
|
||||||
|
|
||||||
⚠⚠ **`~/.config/claude-bot/gitea-token` IS DEAD** — it authenticates as
|
|
||||||
**nobody** (`/user` → `None`). I had cited its 403s and 404s twice, to the
|
|
||||||
operator and to a peer, as evidence that claude-bot lacked rights in `vh/*`. The
|
|
||||||
conclusion survived re-testing with a working credential, but the evidence was
|
|
||||||
worthless. **A credential that authenticates as nobody returns 403 and 404 for
|
|
||||||
everything, and that is indistinguishable from a permissions answer.** There
|
|
||||||
are FOUR token files in that directory; the working one for repo work is
|
|
||||||
**`gitea-token-repo-create`** (`write:organization`, `write:repository`,
|
|
||||||
`write:user`).
|
|
||||||
|
|
||||||
**Standing authorization (operator, same day):** routine vh-token use for
|
|
||||||
`vh/*` repo ops no longer gets a flag. Read it from the **vault** —
|
|
||||||
`secret get 'nh3-dev/.config/gitea/vh-token'` — verified byte-identical to the
|
|
||||||
disk copy and authenticating as `vh` (id 1, is_admin=True). ⚠ This did not make
|
|
||||||
the token low-blast-radius; it stopped the class of work being exceptional.
|
|
||||||
|
|
||||||
Recorded in auto-memory as `feedback_wrong_resource_before_wrong_peer` and in
|
|
||||||
`reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
|
||||||
-3
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.
|
|
||||||
|
|
||||||
⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.
|
|
||||||
|
|
||||||
⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
|
|
||||||
-3
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.
|
|
||||||
|
|
||||||
⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
|
|
||||||
-3
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads
|
|
||||||
|
|
||||||
**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `<id>/tasks/` looks 7+ days idle at `<id>/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs.
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
# `[2026-09-21]` `vh/forgefirm` mirrored
|
|
||||||
|
|
||||||
**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.
|
|
||||||
+21
-65
@@ -1,6 +1,6 @@
|
|||||||
# Persistent memory — eshpfi-management
|
# Persistent memory — eshpfi-management
|
||||||
|
|
||||||
_Last updated: 2026-10-04 ~2335 PT (albok-service moved to its own VM nh3-albok on nh3-pve-2; every hypervisor on PVE 9.2.21 + both PBS on 4.2.7; pfi-ana-webhost destroyed, phasefinal.com on Cloudflare Pages; sudo-io archiver + bulk-data exemption; fv-ml1 root pool cleaned; fv-ml1 Docker move + guest patch campaign parked.)_
|
_Last updated: 2026-10-05 ~2035 PT (new SVOS on nh3-svos deployed + accepted: mesh-only https://svos.phasefinal.com, Basic auth `Vuong`, offline shell, hosting bounds, backup drill; backup check now flags fs-freeze failures (DEGRADED, exit 4) and covers nh3-pve-2; fleet guest patching, reboots and Docker engine bumps done except the held list; fv-ml1 Docker on tank; irv-ml1 cleanup.)_
|
||||||
|
|
||||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||||
@@ -13,8 +13,6 @@ _Last updated: 2026-10-04 ~2335 PT (albok-service moved to its own VM nh3-albok
|
|||||||
|
|
||||||
## Repo purpose
|
## Repo purpose
|
||||||
|
|
||||||
- **2026-09-10 Beszel fleet wiring:** all seven requested hosts plus existing corviduo-dev report up. `/tank` and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to **infra-ops**, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See `persistent-memory.d/2026-09-10-beszel-fleet-wiring.md` and `stacks/beszel/README.md`.
|
|
||||||
|
|
||||||
Reference workspace for PFI infrastructure: server inventory, canonical
|
Reference workspace for PFI infrastructure: server inventory, canonical
|
||||||
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
||||||
copies of compose files live on the servers under
|
copies of compose files live on the servers under
|
||||||
@@ -115,21 +113,18 @@ no longer deployed sidecars here. See Recent decisions.)
|
|||||||
|
|
||||||
## Current state / in-flight
|
## Current state / in-flight
|
||||||
|
|
||||||
_As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections carry their own dates._
|
_As of 2026-10-05 ~2035 PT. The newest subsection is first; older subsections carry their own dates._
|
||||||
|
|
||||||
### 2026-10-04: live now (as of 2026-10-04 ~2300 PT)
|
### 2026-10-05: live now (as of 2026-10-05 ~2035 PT)
|
||||||
|
|
||||||
- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev, pfi-pteradactyl done 10-05 ~1415–1418); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed, patched, engine done; pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05).
|
- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`); eshpfi pushed through `1210b4a`.
|
||||||
- **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%.
|
- **New SVOS on nh3-svos (VM 111, nh3-pve-2, 10.100.50.43): DEPLOYED and ACCEPTED by svos-dev 2026-10-05 ~2024** (thread `01M47ET6XH0M7JZSH98Z19SKR4` closed). Separate from the High Seat (`svos.service` :8770 on nh3-dev) and Miranda, which are NOT migrated. Front door `https://svos.phasefinal.com`, mesh only, Basic auth login `Vuong` (Prime has the password; vault `nh3-svos/basic-auth-vuong`). ⚠ **After every SVOS release** (`/opt/svos/current` moves): `sudo /usr/local/sbin/svos-front-inventory`, and keep the previous release dir while it is "previous" (svos-dev's runbook carries both). Only open item is Prime's: whether his iPhone home-screen app keeps the login. → `servers/nh3-svos/README.md`
|
||||||
- **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
|
- **Backup check: first real run with the new layer is 2026-10-06 0800** (`scripts/backup-freshness-alert.sh`, user timer on nh3-dev, alerts infra-hermes). Expect: nh3-pve-2 coverage read cleanly, svos drill B under ACKNOWLEDGED, verdict not DEGRADED. If it says DEGRADED for anything else, that is a real crash-consistent-only backup.
|
||||||
- **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
- **Waiting on Prime (updated 2026-10-05 2035):** push Booth commit `ce095c1` (touch-layout fix, `~/development/booth`, browser test + mutation); a trusted HTTPS name for ESH Homepage (https://10.0.50.45 serves Traefik's default cert; the ESH ACME resolver, Route53, is dead); nh3-dev reboot (runs kernel 6.1.0-49, 6.1.0-53 installed; its engine is ALREADY 29.8.2 since 10-05 1413, so the old "reboot + engine bump" wording was stale, checked 2035); Docker engine bumps held on nh3-docker (carries the post office) and esh-docker-vm (NFS D-state history); fv-ml1's held ZFS 2.3.9, nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build). ⚠ **irv-ml1: DO NOT PATCH (Prime).** ana-nas is out of scope.
|
||||||
- **Scheduled with infra-hermes (do not redo):** 10-05 0815 PT check that sfsrv-ana's (PVE 8) 04:00 backup reached pbs-ana, then `qm delsnapshot` 100/105 `pre-pbs4` (if stale/errored he sends infra-ops URGENT and keeps them); on/after 10-05 0900 drop `pve/root-pre-pve9` on pfi-pve + esh-pve (job `pve9-drop-pre-snapshot`); 10-11 0900 delete the two one-off PBS VM vzdumps. esh-nas-pve `nvme/ROOT@pre-pve9`: destroy ~10-10 (ours). Never `zpool upgrade` any pool yet.
|
- **Scheduled / to verify:** 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete after its 04:00 backup; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) — outcome not checked in this session, read `scripts/ops-log tail`; esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours); 10-11 0540 first weekly sudo-io archiver run (check it), 10-11 0900 infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet.
|
||||||
- **Weekly sudo-io archiver** first scheduled run Sun 2026-10-11 05:40 (nh3-dev user timer); bulk data no longer recorded fleet-wide. → `services/sudo-io-archive/README.md`
|
- **Worldtree memory gate 20261004T143011Z FAILED** (one run filed the forbidden `u-negative-ephemeral` fact; an esh-pve reboot overlapped run 1). Whether the 2026-10-05 0730 run repeated it was not checked here; it is a regression check with worldtree-dev, not a gate.
|
||||||
- **phasefinal.com on Cloudflare Pages; pfi-ana-webhost destroyed; dragonfireacoustics.com is the client's Pages.** → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm.
|
||||||
- **Worldtree memory gate 20261004T143011Z FAIL** (one run filed the forbidden `u-negative-ephemeral` fact; extractor is `gen` on fv-ml1). With worldtree-dev, with the infra timeline (an esh-pve reboot overlapped run 1). The 2026-10-05 0730 run shows whether it repeats.
|
|
||||||
- **fv-ml1:** root pool 194 G free; lkraven's `~/.cache/huggingface/hub/models--Cloudflare--clef-flash` (18 G, duplicate of tank) was left because bench-jev-2026-10-04 was using it; delete once that bench is done.
|
|
||||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||||
- **Live, nothing open:** albok-service 0.1.8 (all three wings reindexed, albok-dev verified); cira-tunnel-watchdog on pfi-tacticalrmm; RustDesk 1.1.16 on ana-docker (OSS refuses "Change ID" by design; see `servers/ana-docker/README.md`); nh3-pve-2 follow-ups done; VM 101 (DC) + VM 104 (nh3-laser) retired.
|
|
||||||
|
|
||||||
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
|
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
|
||||||
|
|
||||||
@@ -262,23 +257,9 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
|||||||
|
|
||||||
### augaman: face recognition for Cicada
|
### augaman: face recognition for Cicada
|
||||||
|
|
||||||
- **v0.1.3 LIVE on esh-ml1:8040** (v0.1.1 first deployed 2026-09-26 2347 PT), `stacks/augaman`,
|
- **v0.1.3 LIVE on esh-ml1:8040** (`stacks/augaman`), the only instance (fv-ml1's was benched and removed 2026-09-27). Gallery
|
||||||
healthy on CUDA, in `nvidia-smi`. Built on-box from a `git archive` of the tag.
|
backup wired and identity-level restore verified (restic → rest-server-ana `esh-ml1/`), so real enrollments are allowed.
|
||||||
`pytest -m gpu tests/vision` 3/3 PASS on v0.1.3.
|
48 ms per face. CPU-mode regression is augaman-dev's. Nothing owed by infra-ops.
|
||||||
- **Gallery backup WIRED + RESTORE-VERIFIED 2026-09-27** (Prime OK'd off-site): restic
|
|
||||||
daily 0100 PT → rest-server-ana `esh-ml1/`, fail-closed hook, secrets vaulted under
|
|
||||||
`esh-ml1/etc/restic/`. Identity-level restore matched the live gallery (canary,
|
|
||||||
snapshot `fd3061a1`). **The backup gate for real enrollments is MET.**
|
|
||||||
- **Deploy CLOSED by augaman-dev 2026-09-27 0009 PT:** HTTP checks passed; the canary
|
|
||||||
survived the recreate and was then deleted, so the gallery is empty and ready for real
|
|
||||||
enrollments. `/recognize` p50 186 ms (1080p, one face). The detector-latency follow-up
|
|
||||||
is theirs. Their next release pins the container gid to 10001; nothing is owed by infra-ops.
|
|
||||||
- **The fv-ml1 instance was REMOVED by Prime on 2026-09-27 (~0130 PT)** after the bench. esh-ml1 is the only
|
|
||||||
instance: in the house, backed up, 48 ms per face on v0.1.3.
|
|
||||||
- **v0.1.3 went live on both hosts (2026-09-27 ~0105 PT); the fv-ml1 instance was removed later.** Speed bench (`docs/pfi/augaman-speed-bench/`),
|
|
||||||
server-side one face, v0.1.2 → v0.1.3: esh GPU 144 → 48 ms, fv GPU 75 → 27 ms. CPU mode
|
|
||||||
REGRESSED (fv CPU6 152 → 205 ms; suspected ORT thread-pool spinning) and is reported to
|
|
||||||
augaman-dev; the deployment does not use CPU mode.
|
|
||||||
|
|
||||||
### esh-ml1
|
### esh-ml1
|
||||||
|
|
||||||
@@ -293,7 +274,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
|||||||
|
|
||||||
### Live threads
|
### Live threads
|
||||||
|
|
||||||
- git: **eshpfi-management pushed to `6b66207` and worldtree-instance-configs to `b6fdd81` (2026-10-01 ~0418, Prime's go).** Anything after that is unpushed. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated.
|
- git: **eshpfi-management pushed through `1210b4a` (2026-10-05 ~2020); worldtree-instance-configs at `b6fdd81`.** ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose: it is auto-regenerated.
|
||||||
- nh3-dev root disk was cleaned 2026-09-30 1704 (uv prune, dangling images, old build cache): 86% → 82%. The Beszel 85% alert flaps near the line.
|
- nh3-dev root disk was cleaned 2026-09-30 1704 (uv prune, dangling images, old build cache): 86% → 82%. The Beszel 85% alert flaps near the line.
|
||||||
- Booth submit-all fix (Prime's report) is LIVE since 2026-09-27 1705, via booth-dev (booth `50bfc7b`).
|
- Booth submit-all fix (Prime's report) is LIVE since 2026-09-27 1705, via booth-dev (booth `50bfc7b`).
|
||||||
Pushing it is booth's call, per Prime; it is not ours.
|
Pushing it is booth's call, per Prime; it is not ours.
|
||||||
@@ -302,11 +283,13 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
|||||||
## Recent decisions
|
## Recent decisions
|
||||||
|
|
||||||
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
|
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
|
||||||
|
- `[2026-10-05]` **ESH Homepage also on https://10.0.50.45 (Prime)** — a Traefik file-provider router (`homepage-https.yml`, `tls: {}`) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (`2c4c7ca`)
|
||||||
|
- `[2026-10-05]` **bench-jev servers killed and removed, their HF cache cleaned (Prime)**; pfi-pteradactyl fixed + patched + engine upgraded (`3429b5c`; manual dockerd + wings restart).
|
||||||
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
|
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
|
||||||
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
|
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
|
||||||
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
|
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
|
||||||
- `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md`
|
- `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md`
|
||||||
- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result
|
- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result — `[2026-10-05 later]` reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.
|
||||||
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
|
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
|
||||||
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
|
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
|
||||||
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
|
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
|
||||||
@@ -315,8 +298,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
|||||||
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||||||
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
|
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
|
||||||
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
|
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
|
||||||
- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99).
|
- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). — `[2026-10-05]` DONE (entry above).
|
||||||
- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100).
|
- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). — `[2026-10-05]` DONE (entry above).
|
||||||
- `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98).
|
- `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98).
|
||||||
|
|
||||||
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
|
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
|
||||||
@@ -415,34 +398,11 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
|||||||
- `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → `persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md`
|
- `[2026-09-22]` **irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → `persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md`
|
||||||
|
|
||||||
- `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`
|
- `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`
|
||||||
- `[2026-09-21]` **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** — On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one… → `persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md`
|
|
||||||
- `[2026-09-21]` **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** → `persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md`
|
|
||||||
- `[2026-09-21]` **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** → `persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md`
|
|
||||||
|
|
||||||
- `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md`
|
- `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md`
|
||||||
|
|
||||||
- `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md`
|
- `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md`
|
||||||
|
|
||||||
- `[2026-09-21]` ⭐⭐ **claude-bot is an Owner in corviduo/pfi/vastblue; cicada + draupnir moved to `pfi`; vh-token use is now standing-authorized from the vault.** ⚠ `vh` is a USER not an org, so no namespace-scoped admin exists — the only realization of the original ask was site-admin, surfaced rather than executed. ⚠⚠ `~/.config/claude-bot/gitea-token` is DEAD and had been misreporting permissions; the working one is `gitea-token-repo-create`. → `persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md`
|
|
||||||
|
|
||||||
- `[2026-09-21]` ⭐⭐ **nh3-dev 84%→72%: 27 GB reclaimed, and a 252 MB LoRA adapter rescued from `/tmp`, which this box sweeps at 3 days.** `babyyarros` existed nowhere else; hash-verified to smithy before any deletion. ⚠⚠ My 7-day session prune then deleted an ACTIVE session's dir — directory mtime does not reflect subdirectory writes. Do not re-run that predicate. → `persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md`
|
|
||||||
|
|
||||||
- `[2026-09-21]` **Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** — (34c4179, e574b91, 2e08edc). → `persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md`
|
|
||||||
|
|
||||||
- `[2026-09-21]` **Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** — (7fe4102). → `persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md`
|
|
||||||
|
|
||||||
- `[2026-09-21]` **`vh/forgefirm` mirrored** — from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/`… → `persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md`
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
|
|
||||||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
|
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
|
||||||
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
|
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
|
||||||
|
|
||||||
@@ -456,7 +416,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
|||||||
|
|
||||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||||
|
|
||||||
_187 older entries archived to archival-memory.md._
|
_197 older entries archived to archival-memory.md._
|
||||||
|
|
||||||
## Tried and abandoned
|
## Tried and abandoned
|
||||||
|
|
||||||
@@ -484,8 +444,4 @@ _187 older entries archived to archival-memory.md._
|
|||||||
- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
|
- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
|
||||||
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
|
- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
|
||||||
|
|
||||||
- `[2026-09-21]` **Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working… → `persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md`
|
_121 older entries archived to archival-memory.md._
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
_120 older entries archived to archival-memory.md._
|
|
||||||
|
|||||||
Reference in New Issue
Block a user