diff --git a/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md b/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md index 9d02f35..486116b 100644 --- a/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md +++ b/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md @@ -73,6 +73,21 @@ The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint. 256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is both cheaper and likely faster than four more SATA drives. +⚠ **CAPACITY, not throughput, is the binding constraint — the as-bought drives are SMALLER +than the pool they receive from** (measured 2026-09-01). ComfyUI's `/storetank` on irv-ml1 is +a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**, and growing as +comfy-dev's ~112 GB batch lands (-> ~1.56 TiB). The R750xa's 2x **1.92 TB** mirrored is only +**~1.74 TiB** — the migration would arrive at **~90% full** with zero growth room, past the +~80% line where ZFS allocation degrades. Compression buys nothing: safetensors measured +`compressratio 1.00x`, `logicalused == used`. + +**-> Add 2x 2 TB SATA SSD to the buy list.** Six of eight bays free, HBA355i has the ports; +two mirror vdevs striped = **~3.49 TiB at ~45%**, redundancy intact. Cheapest line on this +buy list and it does not gate the cutover window. The no-spend alternative is striping the +as-bought pair (same ~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its +copy, which makes retain a requirement rather than a recommendation. +→ [[2026-09-01-ada-migration-branch-a]] + **UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is `330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane @@ -98,12 +113,16 @@ circuit this lands on before racking, not after.** |---|---|---| | RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket | | NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** | +| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.49 TiB at ~45%; without it the migration lands at ~90% | **Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`, `XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`, `FD00R` ×6. -**Spend order:** RAM → check PCIe slots → only then consider more SATA drives. +**Spend order:** RAM → **2 SATA SSDs (now certain, capacity-driven)** → check PCIe slots → +only then consider filling the remaining bays. (The original order deprioritised drives on a +*throughput* argument, which still holds; the drives are now on the certain list for +**capacity**, which is a different axis.) ⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit hardware — names and quantities only. The factory CSV keyed to the service tag is the only diff --git a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md index fa8bf33..51861b8 100644 --- a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md +++ b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md @@ -12,19 +12,30 @@ comfy-dev asked whether `/storetank` rides along or the stack is rebuilt from so `zfs send` is **incremental**: snapshot now, ship the base over ~4 hours while irv-ml1 keeps serving, then a small delta at cutover. Near-zero service interruption. -## Why (b) — physically moving the disks — was never on the table +## Why (b) — physically moving the disks — was rejected `/storetank` is a two-disk **mirror** of Crucial MX500 2TB SATA SSDs, so it is genuinely -portable hardware. That fact is true and **irrelevant**. +portable hardware. Rejected anyway: the send is non-disruptive at a measured 99 MB/s, and +physically handling disks across sites to save ~4 hours of background transfer is a bad +trade. Not reopened. -**irv-ml1 is NOT being decommissioned.** It runs arbo, comfyui, tts-gateway, dots-tts, -voice-studio, waterland-studio, yt-voice-clipper and dockge — and **comfyui bind-mounts -`/storetank/arbo/models`**. Pulling those disks does not inconvenience a source box that no -longer needs them; it guts a live one. +⚠ **CORRECTED 2026-09-01 by comfy-dev — the ORIGINAL reason given was wrong.** infra-ops +wrote that (b) "was never available" because *"irv-ml1 keeps running its eight services and +needs the pool."* comfy-dev enumerated the mounts of all twelve running containers: -⚠ **The lesson:** infra-ops asked whether the DATA could move and never asked whether the -SOURCE still needed it. One `docker ps` away, at any point. Recommended (b) twice on the -strength of a true-but-irrelevant fact. See [[2026-09-01-wrong-object-measurement]]. + comfyui -> /storetank/arbo/models:/basedir/models <- the complete list + +arbo, dockge, tts-gateway, dots-tts, voice-studio, waterland-studio, yt-voice-clipper (x2), +omnivoice-ref, studio-gate and parakeet **do not touch `/storetank` at all.** Exactly one +container mounts it, and it is the one that moves. So (b) was unavailable **during the +transition** — while comfyui still serves from those disks — not structurally. After cutover +the constraint evaporates. Right conclusion, wrong reason, and the wrong reason mattered +because it mis-describes what irv-ml1 is holding afterwards (see the retain question below). + +⚠ **The lesson, restated:** infra-ops reasoned about the BOX when the question was about the +MOUNT — the same wrong-object shape as asking whether the DATA could move without asking +whether the SOURCE still needed it. Both were one `docker ps` / `docker inspect` away. +See [[2026-09-01-wrong-object-measurement]]. ## (c) rebuild-from-source: rejected on reproducibility, not time @@ -65,11 +76,59 @@ safetensors are architecture-neutral and travel anywhere. Container image, torch and attention kernels do not. "Migrate the model store" and "migrate ComfyUI" are different jobs, and the 1.38 TB transfer is the easy half. +## ⚠ DESTINATION SIZING — the as-bought Ada disks are SMALLER than the source pool + +Measured 2026-09-01 (`zpool list` / `zfs list` on irv-ml1), not derived: + + source pool storetank SIZE 1.81T ALLOC 1.45T FREE 367G CAP 80% + mirror-0: 2x ata-CT2000MX500SSD1 (2 TB class) + compression off, compressratio 1.00x, logicalused == used, 0 snapshots + +Safetensors are **incompressible** (1.00x measured) — the destination needs the same physical +bytes regardless of what compression is set to at recv. There is no free win there. + +The R750xa shipped **2x 1.92 TB** SATA SSD. Those are smaller than the 2 TB MX500s: + + dest pool (2x 1.92 TB mirror) ~1.74 TiB <- SMALLER than the 1.81 TiB source + payload today 1.45 TiB + payload after comfy-dev's batch ~1.56 TiB + occupancy on arrival ~90% + +It fits, and it arrives already past the ~80% line where ZFS allocation degrades, with no +growth room, for a workload whose whole history is "the model store grows." + +**Recommendation to the operator: add 2x 2 TB SATA SSD to the R750xa order** — six of eight +bays are free, HBA355i has the ports, two mirror vdevs striped gives ~3.49 TiB at ~45% +occupancy with redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6` +and 2x `930-00030-1546-000`, and ordering now does not gate the cutover. +→ [[2026-09-01-ada-inference-server-r750xa]] + +The alternative — **striping the as-bought pair** for the same ~3.49 TiB with no redundancy — +is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy. + +## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN + +After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of +tts/llm odds and ends, so 1.4 TB is either reclaimable capacity or a free full backup. +comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent reasons: + +1. **Reproducibility (comfy-dev's).** Two of their pins have gone permanently paid, so a + from-source rebuild provably cannot reproduce today's stack. The irv-ml1 copy is the only + surviving reproduction path for some of those weights. Reclaiming early converts a + recoverable situation into an unrecoverable one for zero gain — nothing waits on the space. +2. **Redundancy (infra-ops').** If the operator declines the two extra drives and we stripe + the as-bought pair, retain stops being a recommendation and becomes a **requirement**. + +⚠ Do not reclaim `/storetank` as routine post-migration tidying. + ## Open - **Cutover window** — operator's, not yet set. +- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered. +- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended. - **comfy-dev's ~112 GB batch** — unheld by infra-ops, but they are deliberately NOT pulling until the operator approves putting that much onto his infrastructure. Their call, correct - instinct. Manifest pinned and staged (`29324e9`). + instinct. Manifest pinned and staged (`29324e9`). **Landing as of 2026-09-01** — SHAs + verifying green; source pool `used` moved 1.39T -> 1.45T during the exchange. Thread: `01M1EYBSYA4QRYK54PX0K1S8CS`. diff --git a/persistent-memory.md b/persistent-memory.md index ddbf154..077693e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -121,7 +121,9 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores - factory 256 GB) and 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters.** Everything else waits + factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD** + (the as-bought 1.92 TB pair mirrored is ~1.74 TiB — SMALLER than the 1.81 TiB pool it + receives from, landing the migration at ~90% full). Everything else waits on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe add-in card would bypass the SAS/SATA backplane and moot the drive-bay question. @@ -133,9 +135,14 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing without deciding that first. Exactly TWO 3c launches, only one died. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a) - `zfs send`, ~3.9 h for 1.38 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. - Branch (b) was never available. **comfy-dev is holding a ~112 GB pull awaiting the - operator's go**, not infra-ops'. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` + `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. + Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts + `/storetank`, so (b) was unavailable during the transition, not structurally). + **TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order + or the pool lands at ~90%; (2) retain vs reclaim irv-ml1's `/storetank` after the move — + RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the + stack; and under a no-spend striped destination it is the only redundancy). comfy-dev's + ~112 GB pull is landing now. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships, "trust the HTTP status, not `error_code`" still applies on running instances. @@ -157,7 +164,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing - `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` -- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` +- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). The R750xa's as-bought 2× 1.92 TB mirrored is ~1.74 TiB → arrival at **~90%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.49 TiB at ~45% with redundancy. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` - `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md` - `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.