memory: snapshot — gx10 unracked and next up as an inference+training box

Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.

Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-28_
+_Last updated: 2026-09-01_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,19 +108,51 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-08-28 16:20 PDT — **althing v3 is live fleet-wide at 3.1.1 with the post office on nh3-docker. `sec` is serving on ana-ml2 GPU0. Nothing is training.**_
+_As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._
 
-- **⏸ RUN 3c STILL HELD — power capacity, unchanged.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, config `/tank/erp-tune/run-03c.json` validated, relaunch is one command. **Do not relaunch until the power triage lands.** ⚠ `sec` now occupies GPU0 (~51 GB), so a 3c relaunch needs GPU0 freed OR accepts three-way contention. Exactly TWO 3c launches, only one died. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
-- **🟢 `sec` (M.O.G.-SEC-27B) IS UP on ana-ml2 GPU0 :8019**, operator-directed. 51,532 MiB, idle 16 W. ⚠ Re-arms the two-GPU load condition that tripped the rack breaker; the risk is concurrent load, not idle. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md`
-- **🟢 althing v3.1.1 on both heralds; post office on nh3-docker `http://10.100.50.40:8390`.** Container untouched since the move. All 73 handles seeded; 5 agents push-reachable (infra-ops, forseti, and the four smithy peers via pane routes). → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
-- **🔴 `/tank` DEGRADED on ana-ml2** — 7 physical NVMe where the pool expects 8, raidz2, one parity spent, no data errors. **Operator replacing it.** → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
-- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). Sheddable on ana-ml2: `vllm-gen` 46 GB, `sec` 51 GB, embed 9.8, coder 8.4, rerank-a3 3.5, reward 2.1, scriberr.
-- **⏸ Owed to brokkr-smithy-dev when there is a card again:** nothing blocking. They hold the dose curve and the entanglement finding.
-- **⏸ Awaiting pewpew-dev:** which other boxes/CI runners resolve `pypotrace` and need the potrace headers. nh3-dev is done; `playbooks/install-potrace-headers.yaml` makes each additional box one command.
-- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
+- **▶ NEXT: pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
+  still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered.
+  **The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and*
+  **run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight.
+  **First move is a throughput probe, not a port**: ARM64 torch, load the base, ten steps,
+  report s/it — that decides whether 604 steps is an overnight run or unusable. Racking is
+  one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`).
+  ⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns.
+  → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
+- **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json`
+  validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10
+  instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
+  without deciding that first. Exactly TWO 3c launches, only one died.
+  → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
+- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
+  `zfs send`, ~3.9 h for 1.38 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
+  Branch (b) was never available. **comfy-dev is holding a ~112 GB pull awaiting the
+  operator's go**, not infra-ops'. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
+- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
+  by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
+  "trust the HTTP status, not `error_code`" still applies on running instances.
+- **🟢 Matrix upgraded and hardened.** Synapse v1.159.0, Element v1.12.27, `/_synapse/admin`
+  LAN-only, appservice namespace opened, alias convention ratified. Miranda provisioned; her
+  summons reached the operator's watch. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
+- **🟢 phasefinal.com LIVE** behind Cloudflare with edge caching + Always Online; apex 301s to
+  www. `inquiry@` alias added by the operator. Stack `stacks/phasefinal-web/`.
+- **🟢 char-rp restored** on ana-ml2 **GPU1** (was dead 7 days on a quant-flag/checkpoint
+  mismatch crash-loop). ⚠ **GPU1 is now at 96,384 / 97,887 MiB — ~1.5 GB free, six tenants.**
+  GPU0's ~44 GB free is a DELIBERATE scratch reserve, not headroom to reclaim.
+- **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD`
+  from a `.env` — until then `stacks/synapse/` and the live file have DIVERGED and
+  `deploy-stack.sh` must not be used on it (README says so).
+- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design.
+  → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
 
 ## Recent decisions
 
+- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
+- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
+- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
+- `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md`
+- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
+- `[2026-09-01]` **Idle VRAM on this fleet is a RESERVED scratch pool, not waste.** Operator declined raising `vllm-mog-sec` from `gpu-memory-utilization 0.52`: single-user dev fleet, KV headroom nobody will consume is worth less than room for ephemeral models and small training runs. vLLM's "fully utilize gpu memory" startup hint does NOT apply here. Tracked in auto-memory `feedback_idle_vram_is_reserved_not_waste`.
 - `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
 - `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md`
 - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`