vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — Ada destination pool is smaller than the source

Browse Source 
Measured irv-ml1's storetank against the R750xa's as-bought drives while
answering comfy-dev's "does the Ada box have ZFS?" question.

- storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression
  off at compressratio 1.00x (safetensors incompressible — no win at
  recv). comfy-dev's ~112 GB batch is landing into it now.
- The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB,
  smaller than the pool it receives from. Migration would arrive at
  ~90% full with no growth room.
- Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) ->
  two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact.
- Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN
  recommended, surfaced to the operator.

Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all
twelve running containers: only comfyui mounts /storetank, so (b) was
unavailable during the transition, not structurally. Right conclusion,
wrong reason — infra-ops reasoned about the box when the question was
about the mount.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 17:06:32 -07:00
parent ace839c768
commit 72065b45aa
3 changed files with 101 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md
+20 -1
View File
@@ -73,6 +73,21 @@ The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint.
256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is
both cheaper and likely faster than four more SATA drives.
⚠ **CAPACITY, not throughput, is the binding constraint — the as-bought drives are SMALLER
than the pool they receive from** (measured 2026-09-01). ComfyUI's `/storetank` on irv-ml1 is
a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**, and growing as
comfy-dev's ~112 GB batch lands (-> ~1.56 TiB). The R750xa's 2x **1.92 TB** mirrored is only
**~1.74 TiB** — the migration would arrive at **~90% full** with zero growth room, past the
~80% line where ZFS allocation degrades. Compression buys nothing: safetensors measured
`compressratio 1.00x`, `logicalused == used`.
**-> Add 2x 2 TB SATA SSD to the buy list.** Six of eight bays free, HBA355i has the ports;
two mirror vdevs striped = **~3.49 TiB at ~45%**, redundancy intact. Cheapest line on this
buy list and it does not gate the cutover window. The no-spend alternative is striping the
as-bought pair (same ~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its
copy, which makes retain a requirement rather than a recommendation.
→ [[2026-09-01-ada-migration-branch-a]]
**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is
`330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of
it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane
@@ -98,12 +113,16 @@ circuit this lands on before racking, not after.**
|---|---|---|
| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket |
| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** |
| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.49 TiB at ~45%; without it the migration lands at ~90% |
**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`,
`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`,
`FD00R` ×6.
**Spend order:** RAM → check PCIe slots → only then consider more SATA drives.
**Spend order:** RAM → **2 SATA SSDs (now certain, capacity-driven)** → check PCIe slots →
only then consider filling the remaining bays. (The original order deprioritised drives on a
*throughput* argument, which still holds; the drives are now on the certain list for
**capacity**, which is a different axis.)
⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit
hardware — names and quantities only. The factory CSV keyed to the service tag is the only
persistent-memory.d/2026-09-01-ada-migration-branch-a.md
+69 -10
View File
@@ -12,19 +12,30 @@ comfy-dev asked whether `/storetank` rides along or the stack is rebuilt from so
`zfs send` is **incremental**: snapshot now, ship the base over ~4 hours while irv-ml1 keeps
serving, then a small delta at cutover. Near-zero service interruption.
## Why (b) — physically moving the disks — was never on the table
## Why (b) — physically moving the disks — was rejected
`/storetank` is a two-disk **mirror** of Crucial MX500 2TB SATA SSDs, so it is genuinely
portable hardware. That fact is true and **irrelevant**.
portable hardware. Rejected anyway: the send is non-disruptive at a measured 99 MB/s, and
physically handling disks across sites to save ~4 hours of background transfer is a bad
trade. Not reopened.
**irv-ml1 is NOT being decommissioned.** It runs arbo, comfyui, tts-gateway, dots-tts,
voice-studio, waterland-studio, yt-voice-clipper and dockge — and **comfyui bind-mounts
`/storetank/arbo/models`**. Pulling those disks does not inconvenience a source box that no
longer needs them; it guts a live one.
⚠ **CORRECTED 2026-09-01 by comfy-dev — the ORIGINAL reason given was wrong.** infra-ops
wrote that (b) "was never available" because *"irv-ml1 keeps running its eight services and
needs the pool."* comfy-dev enumerated the mounts of all twelve running containers:
⚠ **The lesson:** infra-ops asked whether the DATA could move and never asked whether the
SOURCE still needed it. One `docker ps` away, at any point. Recommended (b) twice on the
strength of a true-but-irrelevant fact. See [[2026-09-01-wrong-object-measurement]].
comfyui -> /storetank/arbo/models:/basedir/models <- the complete list
arbo, dockge, tts-gateway, dots-tts, voice-studio, waterland-studio, yt-voice-clipper (x2),
omnivoice-ref, studio-gate and parakeet **do not touch `/storetank` at all.** Exactly one
container mounts it, and it is the one that moves. So (b) was unavailable **during the
transition** — while comfyui still serves from those disks — not structurally. After cutover
the constraint evaporates. Right conclusion, wrong reason, and the wrong reason mattered
because it mis-describes what irv-ml1 is holding afterwards (see the retain question below).
⚠ **The lesson, restated:** infra-ops reasoned about the BOX when the question was about the
MOUNT — the same wrong-object shape as asking whether the DATA could move without asking
whether the SOURCE still needed it. Both were one `docker ps` / `docker inspect` away.
See [[2026-09-01-wrong-object-measurement]].
## (c) rebuild-from-source: rejected on reproducibility, not time
@@ -65,11 +76,59 @@ safetensors are architecture-neutral and travel anywhere. Container image, torch
and attention kernels do not. "Migrate the model store" and "migrate ComfyUI" are different
jobs, and the 1.38 TB transfer is the easy half.
## ⚠ DESTINATION SIZING — the as-bought Ada disks are SMALLER than the source pool
Measured 2026-09-01 (`zpool list` / `zfs list` on irv-ml1), not derived:
source pool storetank SIZE 1.81T ALLOC 1.45T FREE 367G CAP 80%
mirror-0: 2x ata-CT2000MX500SSD1 (2 TB class)
compression off, compressratio 1.00x, logicalused == used, 0 snapshots
Safetensors are **incompressible** (1.00x measured) — the destination needs the same physical
bytes regardless of what compression is set to at recv. There is no free win there.
The R750xa shipped **2x 1.92 TB** SATA SSD. Those are smaller than the 2 TB MX500s:
dest pool (2x 1.92 TB mirror) ~1.74 TiB <- SMALLER than the 1.81 TiB source
payload today 1.45 TiB
payload after comfy-dev's batch ~1.56 TiB
occupancy on arrival ~90%
It fits, and it arrives already past the ~80% line where ZFS allocation degrades, with no
growth room, for a workload whose whole history is "the model store grows."
**Recommendation to the operator: add 2x 2 TB SATA SSD to the R750xa order** — six of eight
bays are free, HBA355i has the ports, two mirror vdevs striped gives ~3.49 TiB at ~45%
occupancy with redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6`
and 2x `930-00030-1546-000`, and ordering now does not gate the cutover.
→ [[2026-09-01-ada-inference-server-r750xa]]
The alternative — **striping the as-bought pair** for the same ~3.49 TiB with no redundancy —
is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy.
## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN
After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of
tts/llm odds and ends, so 1.4 TB is either reclaimable capacity or a free full backup.
comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent reasons:
1. **Reproducibility (comfy-dev's).** Two of their pins have gone permanently paid, so a
from-source rebuild provably cannot reproduce today's stack. The irv-ml1 copy is the only
surviving reproduction path for some of those weights. Reclaiming early converts a
recoverable situation into an unrecoverable one for zero gain — nothing waits on the space.
2. **Redundancy (infra-ops').** If the operator declines the two extra drives and we stripe
the as-bought pair, retain stops being a recommendation and becomes a **requirement**.
⚠ Do not reclaim `/storetank` as routine post-migration tidying.
## Open
- **Cutover window** — operator's, not yet set.
- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered.
- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended.
- **comfy-dev's ~112 GB batch** — unheld by infra-ops, but they are deliberately NOT pulling
until the operator approves putting that much onto his infrastructure. Their call, correct
instinct. Manifest pinned and staged (`29324e9`).
instinct. Manifest pinned and staged (`29324e9`). **Landing as of 2026-09-01** — SHAs
verifying green; source pool `used` moved 1.39T -> 1.45T during the exchange.
Thread: `01M1EYBSYA4QRYK54PX0K1S8CS`.