diff --git a/archival-memory.md b/archival-memory.md index bf3d252..5c2d7e0 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -10511,3 +10511,121 @@ Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]] - `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established. _Archived 2026-09-25._ + +## Recent decisions (archived 2026-09-26 batch) + +# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving + +Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the +ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading. + +## Final state + + fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare) + vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8 + fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation + fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge + seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr + +`tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned. +Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim +LiteLLM gateway, across the mesh, to FV seats on different ports. + +## ⭐⭐ FOUR GPUs, not two — every record was wrong + +`nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4), +independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device +is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and +the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and +seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate. +⚠ Also means the FV circuit was likely specced against half the real draw; operator says +it is on its own circuit and fine for now. + +## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side + +The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was +**correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a). +`ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was +tagging every frame into an untagged access port, so the switch silently discarded them. + +**No cable position could ever have fixed this**, and no network-side diagnostic can see +it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an +in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`, +which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static. +Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200. + +## Routing: what actually had to change (not what we assumed) + +The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned +`10.251.0.0/16` automatically the moment headscale approved the route. What was missing: + +1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first + UDM static route appeared to do nothing — asymmetric drop, not a missing forward route. +2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and + the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first). + +All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact +afterwards. + +## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs + +fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine. +But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly +(`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at +the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS +(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite +anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED. + +## ⚠ Removed a stale if-up hook + +`/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes +via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is +down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`. + +## Credentials, all vaulted and read-back verified + +`fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`, +`fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus +`headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19). +⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io` +(fv-ml1) — redundant now, should be deleted. + +## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours + +Every write API (reboot, service restart, tunables) returns **403** with only a session +cookie. The CSRF token lives in a `