memory: snapshot — Ada sizing corrected; prune and drives are orthogonal
comfy-dev's disk-vs-catalog diff and a recount of my own figures both landed on this thread. Two numbers were wrong and both were headed for the operator's sizing conversation. - My ~90% was a double-count. I read ALLOC 1.45T while their pull was running and then added the full ~112 GB on top; most of it was already in that reading. "Onboarded" is not "landed". Settled payload is ~1.47 TiB and the as-bought mirror lands at 84%, not 90%. - comfy-dev's "pruning gets us nearer 45%" is the striped figure. On the as-bought pair mirrored, deleting all ~215 GiB of unreferenced weights still lands at 72%, with ~140 GiB of runway on a store that took on ~100 GiB in one day. The constraint is vdev layout, not payload — a 1.75 TiB pool stays 1.75 TiB whatever goes in it. - So the prune audit and the drive purchase are independent decisions and neither gates the cutover. Presenting them to the operator that way rather than as a trade. Also recorded: pool arithmetic (1.75 / 3.49 / 3.57 TiB), the mirror-vdev smallest-member gotcha if the new drives get paired one-each with the 1.92s, comfy-dev's 34 GiB of uncatalogued LTX 2.5, and an open question back to them on whether the H3 encoder's nvfp4 pin was set under a Blackwell assumption that sm_89 does not satisfy. Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
@@ -75,17 +75,26 @@ both cheaper and likely faster than four more SATA drives.
|
||||
|
||||
⚠ **CAPACITY, not throughput, is the binding constraint — the as-bought drives are SMALLER
|
||||
than the pool they receive from** (measured 2026-09-01). ComfyUI's `/storetank` on irv-ml1 is
|
||||
a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**, and growing as
|
||||
comfy-dev's ~112 GB batch lands (-> ~1.56 TiB). The R750xa's 2x **1.92 TB** mirrored is only
|
||||
**~1.74 TiB** — the migration would arrive at **~90% full** with zero growth room, past the
|
||||
~80% line where ZFS allocation degrades. Compression buys nothing: safetensors measured
|
||||
a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**. Settled payload
|
||||
**~1.47 TiB**. The R750xa's 2x **1.92 TB** mirrored is only **~1.75 TiB** — smaller than the
|
||||
source pool — so the migration arrives at **~84% full** with no growth room, past the ~80%
|
||||
line where ZFS allocation degrades. Compression buys nothing: safetensors measured
|
||||
`compressratio 1.00x`, `logicalused == used`.
|
||||
|
||||
**-> Add 2x 2 TB SATA SSD to the buy list.** Six of eight bays free, HBA355i has the ports;
|
||||
two mirror vdevs striped = **~3.49 TiB at ~45%**, redundancy intact. Cheapest line on this
|
||||
buy list and it does not gate the cutover window. The no-spend alternative is striping the
|
||||
as-bought pair (same ~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its
|
||||
copy, which makes retain a requirement rather than a recommendation.
|
||||
two mirror vdevs = **~3.57 TiB at ~41%**, redundancy intact. Cheapest line on this buy list
|
||||
and it does not gate the cutover window. The no-spend alternative is striping the as-bought
|
||||
pair (~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its copy, which makes
|
||||
retain a requirement rather than a recommendation.
|
||||
|
||||
⚠ **Pruning does NOT substitute for the drives.** comfy-dev found ~215 GiB of unreferenced
|
||||
weights; deleting every byte still lands the as-bought *mirror* at **72%**. The constraint is
|
||||
**vdev layout**, not payload — a 1.75 TiB pool stays 1.75 TiB whatever goes in it. The prune
|
||||
audit and the drive purchase are independent decisions.
|
||||
|
||||
⚠ **Build-time: pair like with like.** A mirror vdev caps at its smallest member; pairing each
|
||||
new 2 TB with an existing 1.92 TB caps both vdevs at 1.92 TB and wastes ~150 GiB. Correct:
|
||||
the two 1.92s as one vdev, the two new drives as the other.
|
||||
→ [[2026-09-01-ada-migration-branch-a]]
|
||||
|
||||
**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is
|
||||
@@ -113,7 +122,7 @@ circuit this lands on before racking, not after.**
|
||||
|---|---|---|
|
||||
| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket |
|
||||
| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** |
|
||||
| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.49 TiB at ~45%; without it the migration lands at ~90% |
|
||||
| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.57 TiB at ~41%; without it the migration lands at ~84%. Pair the two NEW drives together, not one-each with a 1.92 |
|
||||
|
||||
**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`,
|
||||
`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`,
|
||||
|
||||
@@ -87,25 +87,98 @@ Measured 2026-09-01 (`zpool list` / `zfs list` on irv-ml1), not derived:
|
||||
Safetensors are **incompressible** (1.00x measured) — the destination needs the same physical
|
||||
bytes regardless of what compression is set to at recv. There is no free win there.
|
||||
|
||||
The R750xa shipped **2x 1.92 TB** SATA SSD. Those are smaller than the 2 TB MX500s:
|
||||
**Pool sizes** (decimal TB -> TiB; a mirror vdev shows ~raw member size, confirmed against
|
||||
the measured 1.81T for the 2x 2 TB source):
|
||||
|
||||
dest pool (2x 1.92 TB mirror) ~1.74 TiB <- SMALLER than the 1.81 TiB source
|
||||
payload today 1.45 TiB
|
||||
payload after comfy-dev's batch ~1.56 TiB
|
||||
occupancy on arrival ~90%
|
||||
2x 1.92 TB mirrored 1.75 TiB <- SMALLER than the 1.82 TiB source pool
|
||||
2x 1.92 TB striped 3.49 TiB (no redundancy)
|
||||
4 drives, two mirror vdevs 3.57 TiB (1.92 pair + 2.0 pair)
|
||||
|
||||
It fits, and it arrives already past the ~80% line where ZFS allocation degrades, with no
|
||||
growth room, for a workload whose whole history is "the model store grows."
|
||||
**Payload** — ⚠ corrected 2026-09-01, infra-ops' first figure was inflated:
|
||||
|
||||
1493.9 GiB measured on disk by comfy-dev (weight files)
|
||||
~11 GiB remainder of their batch still arriving
|
||||
~1.47 TiB settled payload
|
||||
~1.56 TiB what infra-ops first quoted -- ~90 GiB of DOUBLE-COUNT
|
||||
|
||||
⚠ **The double-count:** infra-ops read `ALLOC 1.45T` while the pull was running, then added
|
||||
the full ~112 GB on top as if none had landed. Most of it was already inside that reading —
|
||||
the 93.4 GiB comfy-dev lists as "today's batch, not yet onboarded" was **on disk, just not
|
||||
catalogued**. **Onboarded is not landed.** Occupancy quoted as 90% is really **84%**.
|
||||
|
||||
config full (1.47 TiB) pruned (-215 GiB)
|
||||
as-bought mirror 84% 72%
|
||||
as-bought stripe 42% 36%
|
||||
+2 drives, two vdevs 41% 35%
|
||||
|
||||
## ⚠ PRUNE AND DRIVES ARE ORTHOGONAL — pruning cannot rescue the as-bought mirror
|
||||
|
||||
comfy-dev diffed every weight file against the catalog + all 254 workflow graphs:
|
||||
|
||||
on disk 1493.9 GiB
|
||||
referenced 1121.2 GiB
|
||||
unreferenced 374.6 GiB (198 files)
|
||||
- today's batch 93.4 GiB expected, catalogued at onboarding
|
||||
- *ADA-PENDING* builds 24.2 GiB deliberate, correct on Ada
|
||||
= UNEXPLAINED 257.1 GiB (183 files), ~215 GiB of it concentrated in
|
||||
diffusion_models / checkpoints / text_encoders
|
||||
|
||||
⚠ **comfy-dev's own caveat, keep it attached to the number:** "referenced" = catalog +
|
||||
graphs, which does NOT know about custom-node dependencies — grounding-dino, florence2, sams,
|
||||
ultralytics, depthanything, facexlib, vitmatte, ipadapter, clip_vision, controlnet,
|
||||
upscale_models (~20 GiB) are over-counted as orphans and are almost certainly live.
|
||||
|
||||
**comfy-dev suggested this might land the as-bought disks "nearer 45% than 90%". It does
|
||||
not** — 45% is the *striped* figure. Pruning all 215 GiB on the as-bought pair **mirrored**
|
||||
gives **72%**, which leaves ~140 GiB before the 80% line on a store that took on ~100 GiB in
|
||||
a single day. The capacity problem is a **vdev layout** problem: the as-bought pair mirrored
|
||||
is a 1.75 TiB pool whatever we put in it, and deleting payload does not make the destination
|
||||
bigger. Only adding drives or dropping redundancy does.
|
||||
|
||||
**Therefore the audit does not gate the cutover in either direction** — buy the drives on the
|
||||
layout argument and the audit becomes leisure hygiene; skip them and the audit cannot rescue
|
||||
it. Two independent questions for the operator, not a trade. comfy-dev is NOT deleting
|
||||
anything: unreferenced is not safe-to-delete, pins rot, and it is a per-file operator call.
|
||||
|
||||
**Recommendation to the operator: add 2x 2 TB SATA SSD to the R750xa order** — six of eight
|
||||
bays are free, HBA355i has the ports, two mirror vdevs striped gives ~3.49 TiB at ~45%
|
||||
occupancy with redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6`
|
||||
and 2x `930-00030-1546-000`, and ordering now does not gate the cutover.
|
||||
bays are free, HBA355i has the ports, two mirror vdevs gives ~3.57 TiB at ~41% with
|
||||
redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6` and 2x
|
||||
`930-00030-1546-000`, and ordering now does not gate the cutover.
|
||||
→ [[2026-09-01-ada-inference-server-r750xa]]
|
||||
|
||||
The alternative — **striping the as-bought pair** for the same ~3.49 TiB with no redundancy —
|
||||
⚠ **BUILD-TIME GOTCHA — pair like with like.** A ZFS mirror vdev is capped at its SMALLEST
|
||||
member. Pairing each new 2 TB drive with an existing 1.92 TB caps both vdevs at 1.92 TB and
|
||||
throws away ~150 GiB. Correct layout: the two 1.92s as one vdev, the two new drives as the
|
||||
other (1.75 + 1.82 = 3.57 TiB).
|
||||
|
||||
The no-spend alternative — **striping the as-bought pair** for ~3.49 TiB with no redundancy —
|
||||
is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy.
|
||||
|
||||
## Incidental — LTX 2.5 was already on disk, uncatalogued
|
||||
|
||||
comfy-dev's diff turned up 34 GiB of LTX 2.5 (`ltx-2.5-22b-distilled-transformer` 20.0 GiB +
|
||||
`gemma4-12b-with-proj-ltx-2.5` 14.3 GiB, both int8-convrot) that nobody knew was there — after
|
||||
they had spent part of the day reporting LTX 2.5 as the strategic acquisition of the civitai
|
||||
scan and queuing a 15.9 GB REDGraft 2.5 checkpoint on top of a base already present. Their
|
||||
domain, no infra-ops action.
|
||||
|
||||
⚠ **OPEN QUESTION back to comfy-dev, unanswered:** `qwen3vl_32b_minimax_h3_int8_convrot`
|
||||
(25.3 GiB) is listed as an orphan because the pin favours a 15.7 GB nvfp4 build over "the
|
||||
27.1 GB int8 one". **sm_89 Ada has no native nvfp4** — that is Blackwell, and it is the same
|
||||
fact that made their OTHER pin (away from a 7.74 GB nvfp4, toward a 12.84 GB int8) correct.
|
||||
If the H3 pin was set under a Blackwell assumption, the int8 file may be the one that is right
|
||||
on the new box. Asked, not asserted; infra-ops may be conflating two pins.
|
||||
|
||||
## THE PATTERN THIS THREAD KEEPS PRODUCING
|
||||
|
||||
Three findings in one day, all from measurement, each beating a confident derivation:
|
||||
|
||||
1. comfy-dev's `docker inspect` across 12 containers — corrected infra-ops' branch (b) reason.
|
||||
2. infra-ops' `zpool list` — caught the destination being smaller than the source.
|
||||
3. comfy-dev's disk-vs-catalog diff — caught 257 GiB of orphans and 34 GiB of unknown LTX 2.5.
|
||||
|
||||
Same lesson as `docs/pfi/training-throughput-playbook.md` §4, on storage instead of training.
|
||||
|
||||
## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN
|
||||
|
||||
After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of
|
||||
@@ -126,9 +199,13 @@ comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent rea
|
||||
- **Cutover window** — operator's, not yet set.
|
||||
- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered.
|
||||
- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended.
|
||||
- **comfy-dev's ~112 GB batch** — unheld by infra-ops, but they are deliberately NOT pulling
|
||||
until the operator approves putting that much onto his infrastructure. Their call, correct
|
||||
instinct. Manifest pinned and staged (`29324e9`). **Landing as of 2026-09-01** — SHAs
|
||||
verifying green; source pool `used` moved 1.39T -> 1.45T during the exchange.
|
||||
- **comfy-dev's per-file prune audit** — offered, explicitly NOT gating the cutover, waiting
|
||||
on the operator wanting it. Their judgment, not infra-ops'.
|
||||
- **The H3 encoder nvfp4-vs-int8 pin** — infra-ops asked whether it was set under a Blackwell
|
||||
assumption; unanswered.
|
||||
- **comfy-dev's ~112 GB batch** — unheld by infra-ops; they deliberately did not pull until
|
||||
the operator approved putting that much onto his infrastructure. Manifest pinned and staged
|
||||
(`29324e9`). **Landed 2026-09-01**, SHAs green; 93.4 GiB on disk awaiting catalogue
|
||||
onboarding, ~11 GiB still arriving at last measurement.
|
||||
|
||||
Thread: `01M1EYBSYA4QRYK54PX0K1S8CS`.
|
||||
Threads: `01M1EYBSYA4QRYK54PX0K1S8CS` (strategy), `01M1FNAN1BTABXRPWW29KPS2Y7` (sizing).
|
||||
|
||||
Reference in New Issue
Block a user