memory: snapshot — Ada sizing corrected; prune and drives are orthogonal

comfy-dev's disk-vs-catalog diff and a recount of my own figures both
landed on this thread. Two numbers were wrong and both were headed for
the operator's sizing conversation.

- My ~90% was a double-count. I read ALLOC 1.45T while their pull was
  running and then added the full ~112 GB on top; most of it was already
  in that reading. "Onboarded" is not "landed". Settled payload is
  ~1.47 TiB and the as-bought mirror lands at 84%, not 90%.
- comfy-dev's "pruning gets us nearer 45%" is the striped figure. On the
  as-bought pair mirrored, deleting all ~215 GiB of unreferenced weights
  still lands at 72%, with ~140 GiB of runway on a store that took on
  ~100 GiB in one day. The constraint is vdev layout, not payload — a
  1.75 TiB pool stays 1.75 TiB whatever goes in it.
- So the prune audit and the drive purchase are independent decisions
  and neither gates the cutover. Presenting them to the operator that
  way rather than as a trade.

Also recorded: pool arithmetic (1.75 / 3.49 / 3.57 TiB), the mirror-vdev
smallest-member gotcha if the new drives get paired one-each with the
1.92s, comfy-dev's 34 GiB of uncatalogued LTX 2.5, and an open question
back to them on whether the H3 encoder's nvfp4 pin was set under a
Blackwell assumption that sm_89 does not satisfy.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 17:11:04 -07:00
parent 72065b45aa
commit 3180dc85fa
3 changed files with 120 additions and 31 deletions
+9 -6
View File
@@ -122,8 +122,9 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
(the as-bought 1.92 TB pair mirrored is ~1.74 TiB — SMALLER than the 1.81 TiB pool it
receives from, landing the migration at ~90% full). Everything else waits
(the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it
receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout
can). Everything else waits
on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
@@ -139,10 +140,12 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
`/storetank`, so (b) was unavailable during the transition, not structurally).
**TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order
or the pool lands at ~90%; (2) retain vs reclaim irv-ml1's `/storetank` after the move —
or the pool lands at ~84%; (2) retain vs reclaim irv-ml1's `/storetank` after the move —
RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the
stack; and under a no-spend striped destination it is the only redundancy). comfy-dev's
~112 GB pull is landing now. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
stack; and under a no-spend striped destination it is the only redundancy). comfy-dev also
found ~215 GiB of unreferenced weights and offers a per-file prune audit — **explicitly NOT
gating**, because pruning all of it still lands the as-bought mirror at 72%.
→ `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
"trust the HTTP status, not `error_code`" still applies on running instances.
@@ -164,7 +167,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). The R750xa's as-bought 2× 1.92 TB mirrored is ~1.74 TiB → arrival at **~90%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.49 TiB at ~45% with redundancy. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
- `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md`
- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.