From bc6476ea97c756997d5296e55de397d4c3b58b99 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 26 Sep 2026 16:19:59 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20nh3-ml1=20load?= =?UTF-8?q?-shared=20+=20foundry=20seats,=20nh3-pve=20AMT=20live,=20esh-ma?= =?UTF-8?q?tter=20live,=20augaman=20created;=2074=20log=20entries=20split?= =?UTF-8?q?=20two-tier,=202=20archived?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- archival-memory.md | 118 ++++++++ ...mant-regrouping-belayed-by-the-operator.md | 3 + ...path-deferred-tracked-at-park-fused-moe.md | 3 + .../2026-09-12-esh-vm-db-restic-repair.md | 19 -- .../2026-09-12-fv-cutover-executed.md | 92 ------ ...l2fv-ml1-renumber-the-cutover-missed-16.md | 3 + ...5-ana-docker-resolves-no-internal-names.md | 3 + ...tion-hold-released-svos-dev-split-their.md | 3 + ...1-parakeet-retired-voice-studio-stopped.md | 3 + ...settled-by-tts-dev-fv-wins-at-both-clip.md | 3 + ...ned-empty-with-exit-0-under-concurrency.md | 3 + ...3-dev-8092-the-fleet-speaks-and-listens.md | 3 + ...-from-svos-dev-worth-stealing-a-dry-run.md | 3 + ...ry-is-an-assertion-the-leak-gate-can-no.md | 3 + ...er-must-choose-by-size-not-by-count-the.md | 3 + ...-the-rename-can-damage-the-prose-and-no.md | 3 + ...ics-com-expires-2026-10-30-six-weeks-at.md | 3 + ...cs-com-is-configured-on-pfi-ana-webhost.md | 3 + ...the-public-route-from-every-repo-on-nh3.md | 3 + ...t-dnses-nh3-phasefinal-com-to-the-three.md | 3 + ...s-is-operator-ruled-ship-stands-on-both.md | 3 + ...lt-126-units-422-880-words-and-its-name.md | 3 + ...t-167-units-584-756-words-and-the-whole.md | 3 + ...hain-was-recovered-not-remembered-there.md | 3 + ...-deliberately-not-changed-three-of-them.md | 3 + ...d-the-reason-is-a-selection-criterion-2.md | 3 + ...and-the-reason-is-a-selection-criterion.md | 3 + ...-target-was-bare-10-251-50-54-so-deploy.md | 3 + ...claring-sessionstart-hook-is-documented.md | 3 + ...on-leak-is-present-in-hemingway-70-of-7.md | 3 + ...er-names-the-punctuation-on-purpose-and.md | 3 + ...-bronte-adapter-emits-mid-sentence-line.md | 3 + ...pe-is-now-0-for-2-and-should-stop-being.md | 3 + ...or-is-now-pairwise-and-it-retroactively.md | 3 + ...cepted-in-the-kvasir-catalogue-does-not.md | 3 + ...-searxngbraveapikey-in-searxng-settings.md | 3 + ...-plugin-install-is-now-a-symlink-to-the.md | 3 + ...-worldtree-s-env-sh-secrets-are-vaulted.md | 3 + ...-out-to-both-infra-ops-surfaces-and-the.md | 3 + ...hen-creates-removes-changedwhen-without.md | 3 + ...m-workstation-excluded-from-the-nightly.md | 3 + ...-a-circuit-carrying-fv-ml1-and-the-r420.md | 3 + ...is-session-s-assistant-and-the-division.md | 3 + ...d-no-wire-for-three-weeks-and-fixing-it.md | 3 + ...t-scripts-ops-log-automatic-writers-and.md | 3 + ...-as-one-git-author-and-closing-that-gap.md | 3 + ...tered-by-the-operator-the-09-18-hold-is.md | 3 + ...dict-split-stale-exit-1-vs-errored-jobs.md | 3 + ...metry-engine-provisioned-on-irv-ml1-and.md | 3 + ...ector-was-counting-apostrophes-as-quote.md | 3 + ...tion-control-that-shipped-broken-is-now.md | 3 + ...e-is-now-0-for-3-and-this-time-the-loss.md | 3 + ...y-mtime-as-a-liveness-test-when-pruning.md | 3 + .../2026-09-21-vh-forgefirm-mirrored.md | 3 + ...-property-of-the-system-never-one-job-s.md | 3 + ...-uptime-kuma-are-disjoint-not-redundant.md | 3 + ...were-misrouted-to-this-seat-by-panefind.md | 3 + ...failed-start-alarms-on-23-nh3-dev-units.md | 3 + ...26-09-22-fleet-alert-bridge-generalized.md | 3 + ...-tenancy-behind-a-fortinet-pfi-does-not.md | 3 + ...w-assert-content-and-are-discovered-not.md | 3 + ...-on-nh3-dev-and-delegated-fleet-wide-to.md | 3 + ...e-probe-for-a-guard-must-not-be-able-to.md | 3 + ...-from-scratch-on-2-5-5-moved-esh-docker.md | 3 + ...tic-was-skipped-09-22-23-by-my-own-kuma.md | 3 + .../2026-09-23-headscale-ddns-hardened.md | 3 + ...art-exit-1-is-a-hermes-race-not-a-crash.md | 3 + ...ing-order-is-a-repo-claude-md-operating.md | 3 + ...ked-until-the-two-ms-03s-arrive-and-esh.md | 3 + ...-gpu-in-the-pcie-slot-renames-every-nic.md | 3 + ...atic-10-100-250-61-on-nh3-mgmt-udm-port.md | 3 + ...ps-owes-the-gallery-backup-first-deploy.md | 3 + ...-09-26-coder-seat-stays-on-fv-ml1-prime.md | 3 + ...oad-shared-across-esh-ml1-nh3-ml1-prime.md | 3 + ...matter-server-matter-js-1-4-0-on-ct-111.md | 3 + ...h-as-parity-for-a-small-generative-seat.md | 3 + ...ry-utility-seats-live-on-nh3-ml1-brokkr.md | 3 + persistent-memory.md | 284 ++++++++---------- 78 files changed, 459 insertions(+), 276 deletions(-) create mode 100644 persistent-memory.d/2026-08-19-ai-tab-dormant-regrouping-belayed-by-the-operator.md create mode 100644 persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md delete mode 100644 persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md delete mode 100644 persistent-memory.d/2026-09-12-fv-cutover-executed.md create mode 100644 persistent-memory.d/2026-09-13-finished-the-ana-ml2fv-ml1-renumber-the-cutover-missed-16.md create mode 100644 persistent-memory.d/2026-09-15-ana-docker-resolves-no-internal-names.md create mode 100644 persistent-memory.d/2026-09-15-hermes-bearer-rotation-hold-released-svos-dev-split-their.md create mode 100644 persistent-memory.d/2026-09-15-irv-ml1-parakeet-retired-voice-studio-stopped.md create mode 100644 persistent-memory.d/2026-09-15-parakeet-bench-settled-by-tts-dev-fv-wins-at-both-clip.md create mode 100644 persistent-memory.d/2026-09-15-secret-get-returned-empty-with-exit-0-under-concurrency.md create mode 100644 persistent-memory.d/2026-09-15-talk-v10-live-on-nh3-dev-8092-the-fleet-speaks-and-listens.md create mode 100644 persistent-memory.d/2026-09-15-two-restart-patterns-from-svos-dev-worth-stealing-a-dry-run.md create mode 100644 persistent-memory.d/2026-09-17-a-stoplist-entry-is-an-assertion-the-leak-gate-can-no.md create mode 100644 persistent-memory.d/2026-09-17-a-unit-splitter-must-choose-by-size-not-by-count-the.md create mode 100644 persistent-memory.d/2026-09-17-auditentitymap-py-the-rename-can-damage-the-prose-and-no.md create mode 100644 persistent-memory.d/2026-09-17-dragonfireacoustics-com-expires-2026-10-30-six-weeks-at.md create mode 100644 persistent-memory.d/2026-09-17-dragonfireacoustics-com-is-configured-on-pfi-ana-webhost.md create mode 100644 persistent-memory.d/2026-09-17-gitea-was-reaching-the-public-route-from-every-repo-on-nh3.md create mode 100644 persistent-memory.d/2026-09-17-headscale-now-split-dnses-nh3-phasefinal-com-to-the-three.md create mode 100644 persistent-memory.d/2026-09-17-hemingway-ships-as-is-operator-ruled-ship-stands-on-both.md create mode 100644 persistent-memory.d/2026-09-17-lv-krakauer-d1-built-126-units-422-880-words-and-its-name.md create mode 100644 persistent-memory.d/2026-09-17-lv-mccarthy-d1-built-167-units-584-756-words-and-the-whole.md create mode 100644 persistent-memory.d/2026-09-17-lv-mccarthy-s-d1d3-chain-was-recovered-not-remembered-there.md create mode 100644 persistent-memory.d/2026-09-17-measured-and-deliberately-not-changed-three-of-them.md create mode 100644 persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md create mode 100644 persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md create mode 100644 persistent-memory.d/2026-09-17-servers-fv-ml1-ssh-target-was-bare-10-251-50-54-so-deploy.md create mode 100644 persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md create mode 100644 persistent-memory.d/2026-09-17-the-beat-contamination-leak-is-present-in-hemingway-70-of-7.md create mode 100644 persistent-memory.d/2026-09-17-the-mccarthy-register-names-the-punctuation-on-purpose-and.md create mode 100644 persistent-memory.d/2026-09-17-the-shipped-lv-bronte-adapter-emits-mid-sentence-line.md create mode 100644 persistent-memory.d/2026-09-17-the-two-epoch-recipe-is-now-0-for-2-and-should-stop-being.md create mode 100644 persistent-memory.d/2026-09-17-the-v2-voice-floor-is-now-pairwise-and-it-retroactively.md create mode 100644 persistent-memory.d/2026-09-17-triagedisposition-accepted-in-the-kvasir-catalogue-does-not.md create mode 100644 persistent-memory.d/2026-09-18-apikey-env-searxngbraveapikey-in-searxng-settings.md create mode 100644 persistent-memory.d/2026-09-18-miranda-s-hermes-plugin-install-is-now-a-symlink-to-the.md create mode 100644 persistent-memory.d/2026-09-18-worldtree-s-env-sh-secrets-are-vaulted.md create mode 100644 persistent-memory.d/2026-09-19-althing-3-7-0-rolled-out-to-both-infra-ops-surfaces-and-the.md create mode 100644 persistent-memory.d/2026-09-19-elway-evaluated-when-creates-removes-changedwhen-without.md create mode 100644 persistent-memory.d/2026-09-19-esh-vm-102-esh-vm-workstation-excluded-from-the-nightly.md create mode 100644 persistent-memory.d/2026-09-19-fv-is-a-dedicated-20-a-circuit-carrying-fv-ml1-and-the-r420.md create mode 100644 persistent-memory.d/2026-09-19-infra-hermes-is-this-session-s-assistant-and-the-division.md create mode 100644 persistent-memory.d/2026-09-19-the-backup-alarm-had-no-wire-for-three-weeks-and-fixing-it.md create mode 100644 persistent-memory.d/2026-09-19-the-ops-log-is-built-scripts-ops-log-automatic-writers-and.md create mode 100644 persistent-memory.d/2026-09-19-three-agents-commit-as-one-git-author-and-closing-that-gap.md create mode 100644 persistent-memory.d/2026-09-20-ravenpen-com-registered-by-the-operator-the-09-18-hold-is.md create mode 100644 persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md create mode 100644 persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md create mode 100644 persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md create mode 100644 persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md create mode 100644 persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md create mode 100644 persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md create mode 100644 persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md create mode 100644 persistent-memory.d/2026-09-22-backup-coverage-is-a-property-of-the-system-never-one-job-s.md create mode 100644 persistent-memory.d/2026-09-22-beszel-and-uptime-kuma-are-disjoint-not-redundant.md create mode 100644 persistent-memory.d/2026-09-22-d-0010-d-0011-were-misrouted-to-this-seat-by-panefind.md create mode 100644 persistent-memory.d/2026-09-22-failed-start-alarms-on-23-nh3-dev-units.md create mode 100644 persistent-memory.d/2026-09-22-fleet-alert-bridge-generalized.md create mode 100644 persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md create mode 100644 persistent-memory.d/2026-09-22-restic-checks-now-assert-content-and-are-discovered-not.md create mode 100644 persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md create mode 100644 persistent-memory.d/2026-09-22-the-acceptance-probe-for-a-guard-must-not-be-able-to.md create mode 100644 persistent-memory.d/2026-09-22-uptime-kuma-rebuilt-from-scratch-on-2-5-5-moved-esh-docker.md create mode 100644 persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md create mode 100644 persistent-memory.d/2026-09-23-headscale-ddns-hardened.md create mode 100644 persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md create mode 100644 persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md create mode 100644 persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md create mode 100644 persistent-memory.d/2026-09-25-ms-01-foot-gun-a-gpu-in-the-pcie-slot-renames-every-nic.md create mode 100644 persistent-memory.d/2026-09-25-nh3-pve-amt-live-static-10-100-250-61-on-nh3-mgmt-udm-port.md create mode 100644 persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md create mode 100644 persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md create mode 100644 persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md create mode 100644 persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md create mode 100644 persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md create mode 100644 persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md diff --git a/archival-memory.md b/archival-memory.md index bf3d252..5c2d7e0 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -10511,3 +10511,121 @@ Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]] - `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established. _Archived 2026-09-25._ + +## Recent decisions (archived 2026-09-26 batch) + +# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving + +Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the +ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading. + +## Final state + + fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare) + vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8 + fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation + fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge + seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr + +`tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned. +Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim +LiteLLM gateway, across the mesh, to FV seats on different ports. + +## ⭐⭐ FOUR GPUs, not two — every record was wrong + +`nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4), +independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device +is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and +the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and +seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate. +⚠ Also means the FV circuit was likely specced against half the real draw; operator says +it is on its own circuit and fine for now. + +## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side + +The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was +**correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a). +`ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was +tagging every frame into an untagged access port, so the switch silently discarded them. + +**No cable position could ever have fixed this**, and no network-side diagnostic can see +it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an +in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`, +which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static. +Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200. + +## Routing: what actually had to change (not what we assumed) + +The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned +`10.251.0.0/16` automatically the moment headscale approved the route. What was missing: + +1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first + UDM static route appeared to do nothing — asymmetric drop, not a missing forward route. +2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and + the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first). + +All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact +afterwards. + +## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs + +fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine. +But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly +(`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at +the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS +(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite +anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED. + +## ⚠ Removed a stale if-up hook + +`/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes +via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is +down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`. + +## Credentials, all vaulted and read-back verified + +`fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`, +`fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus +`headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19). +⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io` +(fv-ml1) — redundant now, should be deleted. + +## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours + +Every write API (reboot, service restart, tunables) returns **403** with only a session +cookie. The CSRF token lives in a `