memory: snapshot — Ada destination pool is smaller than the source

Measured irv-ml1's storetank against the R750xa's as-bought drives while
answering comfy-dev's "does the Ada box have ZFS?" question.

- storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression
  off at compressratio 1.00x (safetensors incompressible — no win at
  recv). comfy-dev's ~112 GB batch is landing into it now.
- The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB,
  smaller than the pool it receives from. Migration would arrive at
  ~90% full with no growth room.
- Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) ->
  two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact.
- Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN
  recommended, surfaced to the operator.

Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all
twelve running containers: only comfyui mounts /storetank, so (b) was
unavailable during the transition, not structurally. Right conclusion,
wrong reason — infra-ops reasoned about the box when the question was
about the mount.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 17:06:32 -07:00
parent ace839c768
commit 72065b45aa
3 changed files with 101 additions and 16 deletions
+12 -5
View File
@@ -121,7 +121,9 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
factory 256 GB) and 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters.** Everything else waits
factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
(the as-bought 1.92 TB pair mirrored is ~1.74 TiB — SMALLER than the 1.81 TiB pool it
receives from, landing the migration at ~90% full). Everything else waits
on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
@@ -133,9 +135,14 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
without deciding that first. Exactly TWO 3c launches, only one died.
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
`zfs send`, ~3.9 h for 1.38 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
Branch (b) was never available. **comfy-dev is holding a ~112 GB pull awaiting the
operator's go**, not infra-ops'. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
`/storetank`, so (b) was unavailable during the transition, not structurally).
**TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order
or the pool lands at ~90%; (2) retain vs reclaim irv-ml1's `/storetank` after the move —
RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the
stack; and under a no-spend striped destination it is the only redundancy). comfy-dev's
~112 GB pull is landing now. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
"trust the HTTP status, not `error_code`" still applies on running instances.
@@ -157,7 +164,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). The R750xa's as-bought 2× 1.92 TB mirrored is ~1.74 TiB → arrival at **~90%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.49 TiB at ~45% with redundancy. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
- `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md`
- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.