memory: snapshot — nh3-ml1 load-shared + foundry seats, nh3-pve AMT live, esh-matter live, augaman created; 74 log entries split two-tier, 2 archived
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
# `[2026-08-19]` AI-tab Dormant regrouping BELAYED by the operator
|
||||
|
||||
**AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-08-25]` Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).
|
||||
|
||||
**Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
@@ -1,19 +0,0 @@
|
||||
# esh-vm-db Restic repaired
|
||||
|
||||
User asked diagnose/fix two service issues. Only check was systemd-failed:
|
||||
Sep6 repository network timeout after boot. Backup exited0 nightly but pg_dumpall
|
||||
used TCP localhost, required a password, and silently kept April23 dump.
|
||||
Mongo dumps worked. Root cause was the WARN-only hook masking PG failure.
|
||||
|
||||
Fixed PG to /var/run/postgresql peer auth with -w, both DB failures now fail
|
||||
backup preserving prior per-DB dump. Three red-green regression tests pass.
|
||||
Added drop-ins to both jobs: network-online ordering, Restart=on-failure,
|
||||
RestartSec=5min, StartLimitIntervalSec=1h, StartLimitBurst=3.
|
||||
|
||||
Verified real backup bc5eeaff 07:01PDT; fresh PG gzip 3460215 bytes. Retrieved
|
||||
from repo, decompression and completion marker pass (not full restore).
|
||||
Check at07:02 passed 99 snapshots/10% data (19 packs). No failed units remain;
|
||||
both timers active, PG and Mongo active. Inactive/dead between jobs is normal.
|
||||
Old hook/dump preserved root-only /var/lib/restic/repair-20260912.
|
||||
Canonical configs/restic/esh-vm-db and playbooks/esh-vm-db-restic-repair.yaml.
|
||||
No DB restarts or auth-policy changes; changes saved locally, no commit.
|
||||
@@ -1,92 +0,0 @@
|
||||
# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving
|
||||
|
||||
Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the
|
||||
ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading.
|
||||
|
||||
## Final state
|
||||
|
||||
fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare)
|
||||
vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8
|
||||
fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation
|
||||
fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge
|
||||
seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr
|
||||
|
||||
`tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned.
|
||||
Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim
|
||||
LiteLLM gateway, across the mesh, to FV seats on different ports.
|
||||
|
||||
## ⭐⭐ FOUR GPUs, not two — every record was wrong
|
||||
|
||||
`nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4),
|
||||
independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device
|
||||
is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and
|
||||
the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and
|
||||
seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate.
|
||||
⚠ Also means the FV circuit was likely specced against half the real draw; operator says
|
||||
it is on its own circuit and fine for now.
|
||||
|
||||
## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side
|
||||
|
||||
The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was
|
||||
**correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a).
|
||||
`ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was
|
||||
tagging every frame into an untagged access port, so the switch silently discarded them.
|
||||
|
||||
**No cable position could ever have fixed this**, and no network-side diagnostic can see
|
||||
it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an
|
||||
in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`,
|
||||
which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static.
|
||||
Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200.
|
||||
|
||||
## Routing: what actually had to change (not what we assumed)
|
||||
|
||||
The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned
|
||||
`10.251.0.0/16` automatically the moment headscale approved the route. What was missing:
|
||||
|
||||
1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first
|
||||
UDM static route appeared to do nothing — asymmetric drop, not a missing forward route.
|
||||
2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and
|
||||
the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first).
|
||||
|
||||
All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact
|
||||
afterwards.
|
||||
|
||||
## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs
|
||||
|
||||
fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine.
|
||||
But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly
|
||||
(`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at
|
||||
the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS
|
||||
(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite
|
||||
anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED.
|
||||
|
||||
## ⚠ Removed a stale if-up hook
|
||||
|
||||
`/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes
|
||||
via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is
|
||||
down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`.
|
||||
|
||||
## Credentials, all vaulted and read-back verified
|
||||
|
||||
`fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`,
|
||||
`fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus
|
||||
`headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19).
|
||||
⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io`
|
||||
(fv-ml1) — redundant now, should be deleted.
|
||||
|
||||
## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours
|
||||
|
||||
Every write API (reboot, service restart, tunables) returns **403** with only a session
|
||||
cookie. The CSRF token lives in a `<script>` block as
|
||||
`xhr.setRequestHeader("X-CSRFToken", "…")` — **not** a hidden form input, so scraping for
|
||||
`<input type=hidden>` finds nothing. With the header supplied, writes return 200.
|
||||
Config *restores* via the legacy `diag_backup.php` form always worked (it carries a token).
|
||||
|
||||
⚠⚠ **Because the reboot API 403'd silently, I repeatedly reported reboots as mine when
|
||||
the operator was power-cycling the box by hand.** A 403 that coincides with the box going
|
||||
down looks exactly like success. Verify a reboot by observing the box go down AND the
|
||||
change take effect — never by the API's response.
|
||||
|
||||
Runbook: `docs/runbooks/fv-ml1-cutover.md`, `docs/runbooks/fv-onsite-playbook.md`.
|
||||
Commits `91bda3c` (rename/renumber/DNS/LiteLLM), `17586ab` (proxy retired),
|
||||
`87d7555` + `48e34e1` (playbook bodies + remaining live targets).
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-13]` Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP.
|
||||
|
||||
**Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP.** The sweep allowlist was built from files that mention the HOST and a `homepage.href` mentions only an IP, so every label-only stack fell outside it by construction; 24 files fixed plus host copies, allowlist extended with how to derive it next time. Two bugs fell out: `deploy-stack.sh` **rejected any stack name containing a dot** (so `qwen3.5-122b`/`qwopus3.5-122b`/`mistral-medium-3.5` could not be deployed at all), and scriberr's CORS allowlist held only the dead IP and the dead `scriberr.ana.internal`. ⚠ Incomplete: the 10 running containers were never recreated and a plain power-on will NOT apply labels — the staged `compose up -d` recovery does. Eight stacks deliberately not pushed (real host drift); three of those are untracked host-only stacks. Commits `3132a16`, `969a1b6`, `d79f104`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` ana-docker resolves NO `.internal` names
|
||||
|
||||
⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`)
|
||||
|
||||
Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`). ⚠ When the `svos_miranda` plugin arrives it will reference the **dispatch key, not the bearer** (expected), and its `tools` array is legitimately **seven or eight** entries; any other number is a real fault. Commit `e641931`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` irv-ml1 parakeet RETIRED; voice-studio STOPPED.
|
||||
|
||||
**irv-ml1 parakeet RETIRED; voice-studio STOPPED.** Both operator rulings. Parakeet lost tts-dev's bench to the FV seat at both clip lengths and to whisper-large-v3 at 6.24 s; no gateway alias depended on it and every other host reference was a port-register comment. voice-studio existed for the dots mint loop, which Breeze obsoleted 2026-09-06 — retired rather than repaired.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` Parakeet bench settled by tts-dev — FV wins at both clip lengths and beats the incumbent Whisper; IRV seat is now retirable.
|
||||
|
||||
**Parakeet bench settled by tts-dev — FV wins at both clip lengths and beats the incumbent Whisper; IRV seat is now retirable.** FV 155 ms / 391 ms on 1.84 s / 6.24 s clips vs IRV 354 / 1010 vs whisper-large-v3 457 / 690 — IRV is *slower than Whisper* at 6.24 s. Length sweep (n=9/cell, first 3 discarded) fits ~58 ms fixed + 56 ms per audio-second, asymptote ~17.8x realtime, which independently reproduces our 17x on a different harness. Gateway hop measured **below harness resolution** (±30 ms), so `ext-stt` is the right consumer path. ⚠ tts-dev retracted their own plan's 60-120 ms projection: **published RTFx is BATCHED THROUGHPUT, not single-stream latency — the two differ by ~200x.** ⚠ Their between-run variance is ±20% because GPU 0 is the live chat path; our 0.50 s median was taken on an idle GPU 3 and is a best case.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` `secret get` returned EMPTY with exit 0 under concurrency
|
||||
|
||||
**`secret get` returned EMPTY with exit 0 under concurrency** (svos-dev found it; 0/4 succeeded here). Root cause is `bw unlock` racing at **session establishment**, not item reads — so a lock inside the read wrapper cannot work. Fixed: command-level lock, `cmd_get` refuses an empty value, and `find()` no longer coerces empty stdout to `[]`. ⚠ `~/.local/bin/secret` was a plain COPY — now a symlink. `0193b31`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` talk v10 LIVE on nh3-dev :8092 — the fleet speaks and listens on one page.
|
||||
|
||||
**talk v10 LIVE on nh3-dev :8092 — the fleet speaks and listens on one page.** First consumer of the `ext-stt` Parakeet seat: `POST /api/listen`, push-to-talk, barge-in. Gated build→throwaway→teardown→cutover, then **re-gated against production** (a gate that only ran against the throwaway proves the image, not the deployment). ⚠ Deploys route through infra-ops only because tts-dev's identity is not in nh3-dev's `docker` group — a permissions accident, not a judgement call; group-vs-relay is in front of the operator.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-15]` Two restart patterns from svos-dev worth stealing: (a) DRY-RUN BOOT against the still-held port
|
||||
|
||||
⭐⭐ **Two restart patterns from svos-dev worth stealing: (a) DRY-RUN BOOT against the still-held port** — start the new process while the old one holds the socket; it proves every check above the bind and dies on `[Errno 98]`, so a one-way restart becomes a rehearsed one at zero cost. **(b) ⚠ SIGTERM freed the port but left the process alive for 35 s** — a script waiting on the port would have run two copies. **Kill by PID, wait on the PID, never on the port.** A freed port is not evidence of a dead process.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` A stoplist entry is an assertion the leak gate can no longer check
|
||||
|
||||
⭐ **A stoplist entry is an assertion the leak gate can no longer check** — stoplisting removes a surface from the entity map, so a wrongly stoplisted CHARACTER is an undetectable leak. Three were wrong on Brontë (Leaven, Pierrot, Samuel); `scripts/r49-corpus/audit_stoplist.py` finds them by honorific and now gates the pipeline. Commit `8bb7686`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters".
|
||||
|
||||
⭐⭐ **A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters".** `scripts/r49-corpus/split_units.py`: a marker mode qualifies only if its median unit is inside [600, 12000] AND no unit holds half the work; among qualifying modes PRIORITY breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title), and paragraph-block sections are the fallback for works with no divisions. ⭐ **Both rules exist because a control caught them**: scoring by "median closest to target" chose `caps-title` (6 units, one holding **97%** of the book) over True at First Light's real 20 chapters, because a median cannot see that distribution and a max bound can. Positive control: 8/10 Hemingway works reproduce the shipped mode and count exactly. Negative control: 40,000 words with no blank lines → 1 unit, refuses to fabricate divisions. Commit `705fa3a`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` `audit_entity_map.py` — the rename can DAMAGE the prose and no gate will ever say so.
|
||||
|
||||
⭐ **`audit_entity_map.py` — the rename can DAMAGE the prose and no gate will ever say so.** Mirror of `audit_stoplist.py`: surfaces wrongly held IN the map rather than out of it. 130 of 946 Hemingway surfaces flagged (1,616 instances, 0.162% of words) — `African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado` renamed into invented names — plus 16 bare initials incl. `C` at 274 occurrences. Signal is a preceding article; controls derived from the corpus, not hand-picked. Every hit reported for READING: `the Widow` and `the Informer` are genuine epithet-names that should be renamed. Commit `051b99e`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.
|
||||
|
||||
⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.
|
||||
|
||||
**`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` gitea was reaching the PUBLIC route from every repo on nh3-dev
|
||||
|
||||
**gitea was reaching the PUBLIC route from every repo on nh3-dev** — brokkr-smithy, sleipnir, Galdrabok, kvasir — and brokkr-smithy is pushed several times a week, so the fail2ban trigger was live, not dormant. Measured before acting (no split-horizon rewrite, no ssh alias, `ssh -G` confirmed port 22 to 38.120.12.44). Fixed by overriding the NAME once in `~/.ssh/config` rather than rewriting N remotes, so fresh clones and unaudited repos are covered too. Verified with a real `git ls-remote`, not by inspection. Commit `dcc1abc`. Flagged by brokkr-smithy-dev; `vh/imogen` created for them the same session.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` headscale now split-DNSes `nh3.phasefinal.com` to the three AdGuards, so mesh clients can resolve the internal-only wildcard
|
||||
|
||||
**headscale now split-DNSes `nh3.phasefinal.com` to the three AdGuards, so mesh clients can resolve the internal-only wildcard** (`talk`, `booth` — public DNS has no record for them; the fleet AdGuard answers 10.100.10.50). Operator-approved, scoped to nh3 rather than all of `phasefinal.com`. Config `/etc/headscale/config.yaml` in CT 106 on nh3-pve, backup `config.yaml.bak-2026-09-17-splitdns`, restarted, and the new route **read back from a node's netmap** rather than assumed. ⚠ Two things worth knowing: split DNS works fine here with `global: []` — headscale issue #1161's "split ignored without global" does NOT apply to v0.29.3, verified on the live mesh — and `override_local_dns: true` would REQUIRE global, which is the config that makes a roaming laptop lose ALL DNS when the mesh is down. That is why split, not global. Routing was never the problem: nh3-scale already serves 10.100.0.0/16.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` Hemingway ships as-is: operator ruled "ship stands" on both measured corpus defects
|
||||
|
||||
**Hemingway ships as-is: operator ruled "ship stands" on both measured corpus defects** — the 0.96% beat contamination and the 130 non-name entity-map surfaces. `audit_pairs_sourcenames.py --filter-out` and `audit_entity_map.py` exist and are the instruments if that is ever revisited; neither was run against the shipped adapter.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` lv-krakauer D1 built — 126 units, 422,880 words — and its name guard caught three defects nothing else would have reported.
|
||||
|
||||
⭐ **lv-krakauer D1 built — 126 units, 422,880 words — and its name guard caught three defects nothing else would have reported.** Back matter searched only the LAST unit while the apparatus sat in unit 37 of 41; relying on the splitter to drop front matter failed because the ebook TOC sits above the author's note and gave it a `Chapter Thirty-Two` to start on; and **zero was the wrong bar** — 2 survivors are Krakauer writing about his own father in Into the Wild's autobiographical chapters, so the allowance is pinned at 2 with every survivor printed. ⚠ Both strips are windowed in the OPPOSITE direction from McCarthy's, because Krakauer's `ALSO BY`/`Copyright`/`About the Author` sit at 0.0–0.6% of the file. Commit `4be0630`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` lv-mccarthy D1 built — 167 units, 584,756 words — and the whole job was protecting a style that reads as damage.
|
||||
|
||||
⭐ **lv-mccarthy D1 built — 167 units, 584,756 words — and the whole job was protecting a style that reads as damage.** 0.0 quote marks per 10k (Hemingway 838), `dont`/`aint`/`wont`. The builder runs NO typography normalisation and asserts the quote density afterwards. Two truncated catalogue rows dropped for complete mobi siblings; all 15 containment pairs measured (worst 0.10%); back matter in 4 of 6 works carried the author's name 26 times → 0. ⚠ The back-matter strip runs BEFORE the split here — Blood Meridian and The Crossing end with a dumped TOC of bare roman numerals, the exact shape of a chapter marker. Commit `f3bf3ca`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` lv-mccarthy's D1→D3 chain was RECOVERED, not remembered — there was no runbook and the commands went over non-interactive ssh, so no history survived.
|
||||
|
||||
**lv-mccarthy's D1→D3 chain was RECOVERED, not remembered — there was no runbook and the commands went over non-interactive ssh, so no history survived.** Rebuilt candidates and matched sha256 against the artifacts on disk: 6 works, the entity map, the final map and all 36 copy files byte-identical. Now pinned in `scripts/mccarthy-corpus/RUNBOOK.md` with every deviation. ⚠ **D1 must run on nh3-dev** (the builder reads the kvasir catalogue by absolute path); the prior "on gx10" note is true of D2 onward only. ⚠ No phrase map exists for this corpus, so the gate's phrase audit never ran — Yarros and Brontë both had one.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` Measured and DELIBERATELY not changed, three of them.
|
||||
|
||||
**Measured and DELIBERATELY not changed, three of them.** The oversize-passage drop is 13.9% of McCarthy's train words, between Hemingway's 10.0% and the shipped Brontë's 18.3% — in range, no change. `BEAT_PROMPT` asserts the passage is first-person and McCarthy is third; measured inert (**0** narrator-retries against Hemingway's 615 of 7,094), so the prompt was left alone. Blood Meridian's 131 dash-separated chapter-argument paragraphs DID warrant a change and `--drop-leading-heading` now eats them (0 in every other work of all three corpora).
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.
|
||||
|
||||
⭐⭐⭐ **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.** Operator: *"he's a great writer because of his research, not because he has a strong identifiable voice."* A voice adapter is worth its ~6 hours only if the target has a prose signature a reader could pick out blind; Krakauer's excellence is reporting, which an adapter cannot carry. I surveyed, built, measured containment and fixed three stripping defects before anyone asked the question that decided it. **For each candidate, say what the voice IS in one sentence and how it shows up in char-bigram space, before the first catalogue query.** henge id 82. → auto-memory `feedback_voice_worth_adapting_before_corpus`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.
|
||||
|
||||
⭐⭐ **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.** Operator: *"he's a great writer because of his research, not because he has a strong identifiable voice."* I surveyed, built the corpus, measured containment and fixed three stripping defects before anyone asked the question that decided it. henge id 82. → auto-memory `feedback_voice_worth_adapting_before_corpus`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` `servers/fv-ml1/ssh-target` was bare `10.251.50.54`, so `deploy-stack.sh` connected as `lkraven` and could not write the infra-ops-owned `/opt/docker/compose/`
|
||||
|
||||
**`servers/fv-ml1/ssh-target` was bare `10.251.50.54`, so `deploy-stack.sh` connected as `lkraven` and could not write the infra-ops-owned `/opt/docker/compose/`** — and lkraven's sudo on fv-ml1 needs a password, so `DEPLOY_SUDO=1` failed too. Now `infra-ops@10.251.50.54`; `--validate-only` still clean, deploy works. ⚠ Other hosts' `ssh-target` files may carry the same gap — a read-only refresh works as either user, so the fault only surfaces on a deploy.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev
|
||||
|
||||
**The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared at a different minute. A relaunched pane therefore drops to pull mode and stops receiving mail. Raised with the operator, not acted on, because installing it edits `~/.claude/settings.json` and a peer's question is the wrong authorisation for a config change. Tracked at althing thread `01M2R0KPPE85SVJ96KSQ3YKKQP`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` The beat-contamination leak IS present in Hemingway — 70 of 7,094 train beats (0.96%), 0 of 200 val.
|
||||
|
||||
⭐ **The beat-contamination leak IS present in Hemingway — 70 of 7,094 train beats (0.96%), 0 of 200 val.** `scripts/r49-corpus/audit_pairs_sourcenames.py` closes the blind spot `leak_gate.py` has by construction (it reads the corpus and the renamed copies, never the generated beats). Cross-validated on real data: the fixed Brontë pairs return 0 of 3,858 and `pairs-full.CONTAMINATED` returns 15 of 792 = 1.89% with the recorded names. `--filter-out` yields a verified-clean 7,024-pair set in one command; the retrain is the operator's call. **The val split being clean is why the gate could run at all.**
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` The `mccarthy` register names the punctuation ON PURPOSE, and that is a gate-design call made before any McCarthy number existed.
|
||||
|
||||
⭐ **The `mccarthy` register names the punctuation ON PURPOSE, and that is a gate-design call made before any McCarthy number existed.** `eval-*.sh` drives the base control arm with the SAME system prompt via `--system-from`, and `voice_distance.py` is Burrows's Delta over character bigrams — so a tic left OUT of the register is a cheap win only the adapter can take, on a corpus measuring 0.0 quote marks per 10k against Hemingway's 838. Stating them hands them to the control too. Cost stated up front: the voice axis gets harder, and McCarthy's 276-passage val split (against Brontë's 44) is why that trade is affordable here and was not there.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` The SHIPPED lv-bronte adapter emits mid-sentence line breaks at 12.46 per 1k chars, and nothing downstream looks for it.
|
||||
|
||||
⭐⭐ **The SHIPPED lv-bronte adapter emits mid-sentence line breaks at 12.46 per 1k chars, and nothing downstream looks for it.** Its corpus is 100% hard-wrapped at ~68 chars (Gutenberg plain text) and the wrap transfers: base control 0.00, ckpt475 (shipped) 12.46, ckpt925 11.79, every Hemingway arm 0.00 on a 0%-wrapped corpus. Both controls fire. `score_beats.py` passed Brontë's damage axis anyway. McCarthy is the MIXED case — The Road wrapped, the other five works not — which is worse to learn than either pure one, so `build_sft_pairs.py --reflow-hard-wraps` (DEFECT 4) fixes it at pair time, off by default. ⚠ The obvious fix, joining every interior newline, CORRUPTS 46 two-speaker exchanges whose blank line was lost — and unmarked dialogue is the one thing this adapter exists to learn. The rule splits on sentence-final punctuation and takes the cheaper error deliberately.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` The two-epoch recipe is now 0 for 2 and should stop being carried forward.
|
||||
|
||||
**The two-epoch recipe is now 0 for 2 and should stop being carried forward.** Hemingway's eval minimum is step 1750, but step 850 is +0.0040 against a 0.0044 median neighbour jitter — three checkpoints inside one jitter — and 850 won every resolving axis (2.3x tighter seed spread, lower memorisation, less ran-on). Same outcome as Brontë. What IS robust on this schedule is the epoch-3 collapse: +0.0762 = **17.4x jitter**.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` The v2 voice floor is now PAIRWISE, and it retroactively passes lv-bronte.
|
||||
|
||||
⭐⭐ **The v2 voice floor is now PAIRWISE, and it retroactively passes lv-bronte.** lv-bronte's ckpt475 shipped as a voice-axis FAILURE at +0.193 against a 0.251 floor contributed entirely by ckpt925 — a third arm nobody was shipping, on one outlier seed. Scored against the arm it was actually compared to the floor is 0.091 and it clears at **2.1x**. The rule was changed **prospectively**, pre-registered for lv-hemingway before any Hemingway number existed, on an argument independent of the answer: the sampling variability of a difference A−B depends on A and B, never on a third arm C. The previous session found the defect and deliberately declined to exploit it; this follows from fixing it. lv-hemingway passes under **both** rules, so its verdict does not lean on the change. Caveats amended append-only in the compose, the NFS README and the gate record. Commits `0bb4938` `2e9b118`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-17]` `triage_disposition = 'accepted'` in the Kvasir catalogue does NOT mean the extraction succeeded.
|
||||
|
||||
⚠ **`triage_disposition = 'accepted'` in the Kvasir catalogue does NOT mean the extraction succeeded.** Blood Meridian's epub row holds 1,167 words of a 117,000-word book, The Crossing's 222 of 150,000 — real prose, real titles, accepted. Faulkner's *The Mansion* is 39 words. `near_dup_pairs` holds ONE row in the entire 1,284-work library and is blind to a fragment beside its full sibling. **Word-count every master before trusting a row**, and note that word count alone cannot tell a truncated novel from a legitimately short work.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-18]` `api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings
|
||||
|
||||
**`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-18]` Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy
|
||||
|
||||
**Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy** (operator-approved). Verified empirically — `is_dir()`/`exists()` are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is `uv run python scripts/sync_plugin.py --check`; `hermes plugins list` is confirmation-after, never permission-before. Copy kept at `~/backups/svos_miranda-copy-20260918-1444`. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-18]` Worldtree's `env.sh` secrets are vaulted
|
||||
|
||||
**Worldtree's `env.sh` secrets are vaulted** — 10 entries under `worldtree/` (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ `Z_AI_API_KEY` and `ZAI_API_KEY` are DIFFERENT keys despite the near-identical names (fp `26bde3fa` vs `a4884152`). `ANTHROPIC_API_KEY` was empty and skipped.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.
|
||||
|
||||
⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.
|
||||
|
||||
⭐⭐ **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** A root-requiring `when:` (`pvesh get …` exits 255 unprivileged) returns non-zero → elway reports **SKIPPED** → a playbook that never performed its change reports overall **OK**, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; **only the unconditional verify phase caught it.** `creates:` failed the other way — a `/root/…` path is unreadable to the login user, so `test -e` said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never `grep -A<n>` a PVE job in `jobs.cfg` (`vmid` sorts LAST, so adding a `comment` pushed it out of the window and every condition inverted); and `pvesh` JSON carries both a top-level `"enabled":1` and a nested `"fleecing":{"enabled":"0"}`, so a greedy `.*"enabled":` sed matches the wrong one.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.
|
||||
|
||||
**ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed nightly since ~09-06 with `timeout waiting on systemd`, manufacturing a real `job errors` every night. infra-hermes argued correctly that a stopped guest still holds state and the job defect is the real fault; the operator overrode on what the guest actually IS — the one input neither agent had. ⚠ **Policy, not a fix:** the systemd-scope failure on stopped guests is UNDIAGNOSED, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. Job uses an explicit `vmid` INCLUDE list, so exclusion = removal; reason recorded in the job's own `comment`. e50e57b, `playbooks/esh-exclude-vm102-from-backup.yaml`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).
|
||||
|
||||
⭐⭐ **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** The ceiling is **1920 W**, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC `dcmi power reading` 390 W instantaneous / 461 W max over 2423 s **at idle GPUs**, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped **275 W** each against a **300 W** stock TGP (`power.default_limit`) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case **~1625 W capped (85%)** vs **~1725 W at stock (90%)**. ⚠ **Keep the caps** — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ **The coupling is worse than the trip:** OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ **NOT measured:** fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in `servers/fv-ml1/README.md` § Power. ⚠ Also fixed there: the README claimed **2x** GPUs; `nvidia-smi` reports **four**.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` `infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.
|
||||
|
||||
⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.
|
||||
|
||||
⭐⭐⭐ **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** (1) `backup-freshness-alert.sh` called `althing-cli`, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to `postbox`, added the missing `ALTHING_POST_OFFICE` to the unit **and its installer** (the binary swap alone would have failed differently), recipient `infra-ops`→`infra-hermes` (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and `--test-alert` is a positive control because nothing had ever exercised the healthy path. (2) The checker is now **coverage-aware** — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one `exclude` list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads **vzdump TASK STATUS**, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ **`ops-log` earned its keep on day one**: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.
|
||||
|
||||
⭐⭐⭐ **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the `infra-ops`/`lkraven` axis). Claim is `mkdir`-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. `deploy-stack.sh` refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-19]` Three agents commit as one git author, and closing that gap took three instruments to get right.
|
||||
|
||||
⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-20]` `ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.
|
||||
|
||||
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)
|
||||
|
||||
**Backup alarm verdict split: `STALE` (exit 1) vs `ERRORED-JOBS` (exit 3)** (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested
|
||||
|
||||
**Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested** (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 **containerised** — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by `ldd`), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes `result.json` into CWD on EVERY invocation, `--help` included. Playbook `playbooks/irv-ml1-draupnir-engine.yaml`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.
|
||||
|
||||
⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.
|
||||
|
||||
⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.
|
||||
|
||||
⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` Using directory mtime as a liveness test when pruning session scratchpads
|
||||
|
||||
**Using directory mtime as a liveness test when pruning session scratchpads** — `find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7` deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to `<id>/tasks/` looks 7+ days idle at `<id>/`. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running `claude` PIDs.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-21]` `vh/forgefirm` mirrored
|
||||
|
||||
**`vh/forgefirm` mirrored** from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/` namespace, upstream casing, `8h0m0s` interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (`08b29fee`) against upstream, not by the 201. ⚠ `vh/NetAlertX` interval `0s` is **deliberate** — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` Backup coverage is a property of the SYSTEM, never one job's scope
|
||||
|
||||
**Backup coverage is a property of the SYSTEM, never one job's scope** — establish it by querying the repo for the path in a real snapshot, never by reading a job's `SRC=`. A peer read `dev-backup` alone and told the operator, unhedged, that Miranda's conversation had "never been backed up"; all five paths were in that night's restic snapshot. `docs/runbooks/backups.md`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` Beszel and Uptime Kuma are DISJOINT, not redundant
|
||||
|
||||
**Beszel and Uptime Kuma are DISJOINT, not redundant** — Beszel's alerts bind to a *system* with a threshold; there is no URL column, so it is structurally incapable of "this endpoint should return 200". Homepage is DISPLAY ONLY (38 polls, alerts nobody). The empty seam between them is where homepage sat dead for three days. Settled; do not re-litigate.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` D-0010/D-0011 were misrouted to this seat by `pane_find` matching a ROLLING PANE TITLE.
|
||||
|
||||
**D-0010/D-0011 were misrouted to this seat by `pane_find` matching a ROLLING PANE TITLE.** Genuine and operator-approved, wrong seat; `fleet_telemetry` held the right mapping and carries the warning, while `pane_find` is the title search and carries none. Belayed both; svos-dev owns the fix in `server/bifrost.py`. ⚠ Do not edit that file.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` Failed-START alarms on 23 nh3-dev units
|
||||
|
||||
**Failed-START alarms on 23 nh3-dev units** (`services/althing-notify-failure/`). ⚠ **Coverage is far narrower than it looks** — only the 7 timer oneshots and `svos.service` can actually reach `failed`; ten daemons flap forever instead. `svos.service`'s divergent 5-min window is LOAD-BEARING (operator ruling 2026-09-11) — **do not harmonise it**. Commits `1be39d1`, `163bb97`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` Fleet alert bridge generalized
|
||||
|
||||
**Fleet alert bridge generalized** — `beszel-althing` → `althing-alert-bridge`, route registry (`/beszel` + `/kuma`), each with its own prefix/footer/parser. `/beszel` FROZEN and proven byte-identical (its three original tests kept unchanged). An alert that lies about its own source is worse than no alert. Commit `6f0a9b9`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control
|
||||
|
||||
**irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control** — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). **No fix on our side; do not go looking.** ⚠ Diagnose reachability with `tailscale ping`, NOT status output — headscale said `online` and status said `active, 19.7GB` while nothing answered. Commit `38bb20c`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` restic checks now assert CONTENT and are DISCOVERED not enumerated
|
||||
|
||||
**restic checks now assert CONTENT and are DISCOVERED not enumerated** — conjunctive (recent AND present AND restores non-zero bytes); repos discovered from each NAS, hand list demoted to an EXPECTED set that distinguishes "vanished" from "stale". nh3-dev had never been on the watch list at all. Commits `ba60fda`, `4979869`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.
|
||||
|
||||
⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh).
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` The acceptance probe for a guard must not be able to destroy what it tests
|
||||
|
||||
**The acceptance probe for a guard must not be able to destroy what it tests** (infra-hermes). `rm -rf /home` to prove safe-rm refuses is a test whose premise IS the thing under test. Guarded form is now standard: `[ "$(command -v rm)" = /usr/share/safe-rm/bin/rm ] && rm -rf /home || echo NOT-GUARDED`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-22]` Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker.
|
||||
|
||||
**Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker.** ⚠ `:latest` is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. No REST CRUD API in either major; automation is Socket.IO via first-party `scripts/kuma` (the community wrapper is abandoned at 1.23.1). Monitors + channel + status page all declared in `stacks/uptimekuma/monitors.yaml`. Commits `3a85a6b`, `94899d6`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move
|
||||
|
||||
**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`).
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-23]` headscale-ddns hardened
|
||||
|
||||
**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression).
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash
|
||||
|
||||
**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter
|
||||
|
||||
**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`).
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-25]` AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).
|
||||
|
||||
**AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).** Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale (ANA/ESH→NH3 rides CT 107 on nh3-pve today; recommended: 10.100.250.0/24 over the FortiGate↔UDM IPsec `pfi-ana-nh3`); AMT in TRMM's MeshCentral (public, wants 2FA). Tracked: `c923727`, `servers/nh3-pve/README.md`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-25]` MS-01 foot-gun: a GPU in the PCIe slot renames every NIC
|
||||
|
||||
**MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-25]` nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None
|
||||
|
||||
**nh3-pve AMT LIVE: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None** (Prime). Root cause of "no IP": Linux held the I226-LM down, so igc powered off the PHY; the fix is `auto enp88s0` + IPv6 off + `arp_ignore=8`. Password vaulted `nh3-pve/amt-admin`; tool `scripts/amt-wsman.py`. `0f032fa`…`50f113c`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-26]` augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.
|
||||
|
||||
**augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** esh-ml1 is outside vzdump, so a scheduled `sqlite3 .backup` into restic, with a verified restore, before any real enrollment; then deploy on esh-ml1 (suggested `:8040`). Tracked: `pfi/augaman` `HANDOFF.md`, `d6626ce`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-26]` Coder seat STAYS on fv-ml1 (Prime).
|
||||
|
||||
**Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-26]` Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).
|
||||
|
||||
**Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB, `/model/new`). Without failover, a dead backend failed 7/40 embeds; `router_settings.enable_weighted_failover: true` (affects only multi-deployment groups) brought it to rerank 80/80 and embed 38/40 at onset, 60/60 sustained. In LiteLLM 1.97 the embedding path does not stamp the failed deployment, so it relies on cooldown. LiteLLM restarted 3× tonight, ~36 s each.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-26]` esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only
|
||||
|
||||
**esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only**, for ha-dev (operator-approved, relayed). It is on the IoT link because Matter/Thread IPv6 (Echo ULA + RIO routes) is link-only. Kernel RA with rt_info_max_plen=64, forwarding off; Docker ip-forward/iptables off. The in-CT nftables admits :5580 from HA 10.0.50.46 only and SSH from mgmt ranges only, and accepts everything else so Matter UDP never meets conntrack. The CT is in esh-pve vzdump (fabric creds). Acceptance all green: fdad:: SLAAC; ping6 thermostat; 2 Thread routes learned (Aqara + Echo); ws OK from inside the HA container; refused from 10.0.50.45, nh3-dev and a temp VLAN 90 netns. → `servers/esh-matter/README.md`
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-26]` Greedy exact-match as parity for a small generative seat (coder)
|
||||
|
||||
**Greedy exact-match as parity for a small generative seat (coder)** — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie tokens cascade). Use teacher-forced logprobs of real text with `cache_salt`: each host bit-exact with itself. `stacks/coder-seat/README.md`.
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
# `[2026-09-26]` Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):
|
||||
|
||||
**Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and VibeVoice-ASR-Streaming-1.5B on **audio.cpp** `:8031`, not vibevoice.cpp as specced; the GGUF card names audio.cpp. Controls: VL read a synthetic image exactly, but hallucinates when no image is sent; ASR WER 3/69 on the bundled LibriSpeech clips. libsoxr added (linear resampling misheard a word). First ASR request ~31 s cold. Answered brokkr: the old 27B image seat (10.250.50.54:8015) was retired 09-14; baseline = gateway `image-judge` (Flash-Next).
|
||||
Reference in New Issue
Block a user