diff --git a/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md b/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md index 486116b..3334422 100644 --- a/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md +++ b/persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md @@ -75,17 +75,26 @@ both cheaper and likely faster than four more SATA drives. ⚠ **CAPACITY, not throughput, is the binding constraint — the as-bought drives are SMALLER than the pool they receive from** (measured 2026-09-01). ComfyUI's `/storetank` on irv-ml1 is -a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**, and growing as -comfy-dev's ~112 GB batch lands (-> ~1.56 TiB). The R750xa's 2x **1.92 TB** mirrored is only -**~1.74 TiB** — the migration would arrive at **~90% full** with zero growth room, past the -~80% line where ZFS allocation degrades. Compression buys nothing: safetensors measured +a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**. Settled payload +**~1.47 TiB**. The R750xa's 2x **1.92 TB** mirrored is only **~1.75 TiB** — smaller than the +source pool — so the migration arrives at **~84% full** with no growth room, past the ~80% +line where ZFS allocation degrades. Compression buys nothing: safetensors measured `compressratio 1.00x`, `logicalused == used`. **-> Add 2x 2 TB SATA SSD to the buy list.** Six of eight bays free, HBA355i has the ports; -two mirror vdevs striped = **~3.49 TiB at ~45%**, redundancy intact. Cheapest line on this -buy list and it does not gate the cutover window. The no-spend alternative is striping the -as-bought pair (same ~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its -copy, which makes retain a requirement rather than a recommendation. +two mirror vdevs = **~3.57 TiB at ~41%**, redundancy intact. Cheapest line on this buy list +and it does not gate the cutover window. The no-spend alternative is striping the as-bought +pair (~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its copy, which makes +retain a requirement rather than a recommendation. + +⚠ **Pruning does NOT substitute for the drives.** comfy-dev found ~215 GiB of unreferenced +weights; deleting every byte still lands the as-bought *mirror* at **72%**. The constraint is +**vdev layout**, not payload — a 1.75 TiB pool stays 1.75 TiB whatever goes in it. The prune +audit and the drive purchase are independent decisions. + +⚠ **Build-time: pair like with like.** A mirror vdev caps at its smallest member; pairing each +new 2 TB with an existing 1.92 TB caps both vdevs at 1.92 TB and wastes ~150 GiB. Correct: +the two 1.92s as one vdev, the two new drives as the other. → [[2026-09-01-ada-migration-branch-a]] **UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is @@ -113,7 +122,7 @@ circuit this lands on before racking, not after.** |---|---|---| | RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket | | NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** | -| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.49 TiB at ~45%; without it the migration lands at ~90% | +| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.57 TiB at ~41%; without it the migration lands at ~84%. Pair the two NEW drives together, not one-each with a 1.92 | **Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`, `XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`, diff --git a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md index 51861b8..83ba139 100644 --- a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md +++ b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md @@ -87,25 +87,98 @@ Measured 2026-09-01 (`zpool list` / `zfs list` on irv-ml1), not derived: Safetensors are **incompressible** (1.00x measured) — the destination needs the same physical bytes regardless of what compression is set to at recv. There is no free win there. -The R750xa shipped **2x 1.92 TB** SATA SSD. Those are smaller than the 2 TB MX500s: +**Pool sizes** (decimal TB -> TiB; a mirror vdev shows ~raw member size, confirmed against +the measured 1.81T for the 2x 2 TB source): - dest pool (2x 1.92 TB mirror) ~1.74 TiB <- SMALLER than the 1.81 TiB source - payload today 1.45 TiB - payload after comfy-dev's batch ~1.56 TiB - occupancy on arrival ~90% + 2x 1.92 TB mirrored 1.75 TiB <- SMALLER than the 1.82 TiB source pool + 2x 1.92 TB striped 3.49 TiB (no redundancy) + 4 drives, two mirror vdevs 3.57 TiB (1.92 pair + 2.0 pair) -It fits, and it arrives already past the ~80% line where ZFS allocation degrades, with no -growth room, for a workload whose whole history is "the model store grows." +**Payload** — ⚠ corrected 2026-09-01, infra-ops' first figure was inflated: + + 1493.9 GiB measured on disk by comfy-dev (weight files) + ~11 GiB remainder of their batch still arriving + ~1.47 TiB settled payload + ~1.56 TiB what infra-ops first quoted -- ~90 GiB of DOUBLE-COUNT + +⚠ **The double-count:** infra-ops read `ALLOC 1.45T` while the pull was running, then added +the full ~112 GB on top as if none had landed. Most of it was already inside that reading — +the 93.4 GiB comfy-dev lists as "today's batch, not yet onboarded" was **on disk, just not +catalogued**. **Onboarded is not landed.** Occupancy quoted as 90% is really **84%**. + + config full (1.47 TiB) pruned (-215 GiB) + as-bought mirror 84% 72% + as-bought stripe 42% 36% + +2 drives, two vdevs 41% 35% + +## ⚠ PRUNE AND DRIVES ARE ORTHOGONAL — pruning cannot rescue the as-bought mirror + +comfy-dev diffed every weight file against the catalog + all 254 workflow graphs: + + on disk 1493.9 GiB + referenced 1121.2 GiB + unreferenced 374.6 GiB (198 files) + - today's batch 93.4 GiB expected, catalogued at onboarding + - *ADA-PENDING* builds 24.2 GiB deliberate, correct on Ada + = UNEXPLAINED 257.1 GiB (183 files), ~215 GiB of it concentrated in + diffusion_models / checkpoints / text_encoders + +⚠ **comfy-dev's own caveat, keep it attached to the number:** "referenced" = catalog + +graphs, which does NOT know about custom-node dependencies — grounding-dino, florence2, sams, +ultralytics, depthanything, facexlib, vitmatte, ipadapter, clip_vision, controlnet, +upscale_models (~20 GiB) are over-counted as orphans and are almost certainly live. + +**comfy-dev suggested this might land the as-bought disks "nearer 45% than 90%". It does +not** — 45% is the *striped* figure. Pruning all 215 GiB on the as-bought pair **mirrored** +gives **72%**, which leaves ~140 GiB before the 80% line on a store that took on ~100 GiB in +a single day. The capacity problem is a **vdev layout** problem: the as-bought pair mirrored +is a 1.75 TiB pool whatever we put in it, and deleting payload does not make the destination +bigger. Only adding drives or dropping redundancy does. + +**Therefore the audit does not gate the cutover in either direction** — buy the drives on the +layout argument and the audit becomes leisure hygiene; skip them and the audit cannot rescue +it. Two independent questions for the operator, not a trade. comfy-dev is NOT deleting +anything: unreferenced is not safe-to-delete, pins rot, and it is a per-file operator call. **Recommendation to the operator: add 2x 2 TB SATA SSD to the R750xa order** — six of eight -bays are free, HBA355i has the ports, two mirror vdevs striped gives ~3.49 TiB at ~45% -occupancy with redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6` -and 2x `930-00030-1546-000`, and ordering now does not gate the cutover. +bays are free, HBA355i has the ports, two mirror vdevs gives ~3.57 TiB at ~41% with +redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6` and 2x +`930-00030-1546-000`, and ordering now does not gate the cutover. → [[2026-09-01-ada-inference-server-r750xa]] -The alternative — **striping the as-bought pair** for the same ~3.49 TiB with no redundancy — +⚠ **BUILD-TIME GOTCHA — pair like with like.** A ZFS mirror vdev is capped at its SMALLEST +member. Pairing each new 2 TB drive with an existing 1.92 TB caps both vdevs at 1.92 TB and +throws away ~150 GiB. Correct layout: the two 1.92s as one vdev, the two new drives as the +other (1.75 + 1.82 = 3.57 TiB). + +The no-spend alternative — **striping the as-bought pair** for ~3.49 TiB with no redundancy — is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy. +## Incidental — LTX 2.5 was already on disk, uncatalogued + +comfy-dev's diff turned up 34 GiB of LTX 2.5 (`ltx-2.5-22b-distilled-transformer` 20.0 GiB + +`gemma4-12b-with-proj-ltx-2.5` 14.3 GiB, both int8-convrot) that nobody knew was there — after +they had spent part of the day reporting LTX 2.5 as the strategic acquisition of the civitai +scan and queuing a 15.9 GB REDGraft 2.5 checkpoint on top of a base already present. Their +domain, no infra-ops action. + +⚠ **OPEN QUESTION back to comfy-dev, unanswered:** `qwen3vl_32b_minimax_h3_int8_convrot` +(25.3 GiB) is listed as an orphan because the pin favours a 15.7 GB nvfp4 build over "the +27.1 GB int8 one". **sm_89 Ada has no native nvfp4** — that is Blackwell, and it is the same +fact that made their OTHER pin (away from a 7.74 GB nvfp4, toward a 12.84 GB int8) correct. +If the H3 pin was set under a Blackwell assumption, the int8 file may be the one that is right +on the new box. Asked, not asserted; infra-ops may be conflating two pins. + +## THE PATTERN THIS THREAD KEEPS PRODUCING + +Three findings in one day, all from measurement, each beating a confident derivation: + +1. comfy-dev's `docker inspect` across 12 containers — corrected infra-ops' branch (b) reason. +2. infra-ops' `zpool list` — caught the destination being smaller than the source. +3. comfy-dev's disk-vs-catalog diff — caught 257 GiB of orphans and 34 GiB of unknown LTX 2.5. + +Same lesson as `docs/pfi/training-throughput-playbook.md` §4, on storage instead of training. + ## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of @@ -126,9 +199,13 @@ comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent rea - **Cutover window** — operator's, not yet set. - **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered. - **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended. -- **comfy-dev's ~112 GB batch** — unheld by infra-ops, but they are deliberately NOT pulling - until the operator approves putting that much onto his infrastructure. Their call, correct - instinct. Manifest pinned and staged (`29324e9`). **Landing as of 2026-09-01** — SHAs - verifying green; source pool `used` moved 1.39T -> 1.45T during the exchange. +- **comfy-dev's per-file prune audit** — offered, explicitly NOT gating the cutover, waiting + on the operator wanting it. Their judgment, not infra-ops'. +- **The H3 encoder nvfp4-vs-int8 pin** — infra-ops asked whether it was set under a Blackwell + assumption; unanswered. +- **comfy-dev's ~112 GB batch** — unheld by infra-ops; they deliberately did not pull until + the operator approved putting that much onto his infrastructure. Manifest pinned and staged + (`29324e9`). **Landed 2026-09-01**, SHAs green; 93.4 GiB on disk awaiting catalogue + onboarding, ~11 GiB still arriving at last measurement. -Thread: `01M1EYBSYA4QRYK54PX0K1S8CS`. +Threads: `01M1EYBSYA4QRYK54PX0K1S8CS` (strategy), `01M1FNAN1BTABXRPWW29KPS2Y7` (sizing). diff --git a/persistent-memory.md b/persistent-memory.md index 077693e..d260d3e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -122,8 +122,9 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing - **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD** - (the as-bought 1.92 TB pair mirrored is ~1.74 TiB — SMALLER than the 1.81 TiB pool it - receives from, landing the migration at ~90% full). Everything else waits + (the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it + receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout + can). Everything else waits on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe add-in card would bypass the SAS/SATA backplane and moot the drive-bay question. @@ -139,10 +140,12 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts `/storetank`, so (b) was unavailable during the transition, not structurally). **TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order - or the pool lands at ~90%; (2) retain vs reclaim irv-ml1's `/storetank` after the move — + or the pool lands at ~84%; (2) retain vs reclaim irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the - stack; and under a no-spend striped destination it is the only redundancy). comfy-dev's - ~112 GB pull is landing now. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` + stack; and under a no-spend striped destination it is the only redundancy). comfy-dev also + found ~215 GiB of unreferenced weights and offers a per-file prune audit — **explicitly NOT + gating**, because pruning all of it still lands the as-bought mirror at 72%. + → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships, "trust the HTTP status, not `error_code`" still applies on running instances. @@ -164,7 +167,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing - `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` -- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). The R750xa's as-bought 2× 1.92 TB mirrored is ~1.74 TiB → arrival at **~90%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.49 TiB at ~45% with redundancy. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` +- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` - `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` - `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md` - `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.