memory: snapshot — Ada destination pool is smaller than the source
Measured irv-ml1's storetank against the R750xa's as-bought drives while answering comfy-dev's "does the Ada box have ZFS?" question. - storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression off at compressratio 1.00x (safetensors incompressible — no win at recv). comfy-dev's ~112 GB batch is landing into it now. - The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB, smaller than the pool it receives from. Migration would arrive at ~90% full with no growth room. - Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) -> two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact. - Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN recommended, surfaced to the operator. Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all twelve running containers: only comfyui mounts /storetank, so (b) was unavailable during the transition, not structurally. Right conclusion, wrong reason — infra-ops reasoned about the box when the question was about the mount. Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
@@ -73,6 +73,21 @@ The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint.
|
||||
256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is
|
||||
both cheaper and likely faster than four more SATA drives.
|
||||
|
||||
⚠ **CAPACITY, not throughput, is the binding constraint — the as-bought drives are SMALLER
|
||||
than the pool they receive from** (measured 2026-09-01). ComfyUI's `/storetank` on irv-ml1 is
|
||||
a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**, and growing as
|
||||
comfy-dev's ~112 GB batch lands (-> ~1.56 TiB). The R750xa's 2x **1.92 TB** mirrored is only
|
||||
**~1.74 TiB** — the migration would arrive at **~90% full** with zero growth room, past the
|
||||
~80% line where ZFS allocation degrades. Compression buys nothing: safetensors measured
|
||||
`compressratio 1.00x`, `logicalused == used`.
|
||||
|
||||
**-> Add 2x 2 TB SATA SSD to the buy list.** Six of eight bays free, HBA355i has the ports;
|
||||
two mirror vdevs striped = **~3.49 TiB at ~45%**, redundancy intact. Cheapest line on this
|
||||
buy list and it does not gate the cutover window. The no-spend alternative is striping the
|
||||
as-bought pair (same ~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its
|
||||
copy, which makes retain a requirement rather than a recommendation.
|
||||
→ [[2026-09-01-ada-migration-branch-a]]
|
||||
|
||||
**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is
|
||||
`330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of
|
||||
it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane
|
||||
@@ -98,12 +113,16 @@ circuit this lands on before racking, not after.**
|
||||
|---|---|---|
|
||||
| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB, all 16 slots, all 8 channels/socket |
|
||||
| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** |
|
||||
| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.49 TiB at ~45%; without it the migration lands at ~90% |
|
||||
|
||||
**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`,
|
||||
`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`,
|
||||
`FD00R` ×6.
|
||||
|
||||
**Spend order:** RAM → check PCIe slots → only then consider more SATA drives.
|
||||
**Spend order:** RAM → **2 SATA SSDs (now certain, capacity-driven)** → check PCIe slots →
|
||||
only then consider filling the remaining bays. (The original order deprioritised drives on a
|
||||
*throughput* argument, which still holds; the drives are now on the certain list for
|
||||
**capacity**, which is a different axis.)
|
||||
|
||||
⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit
|
||||
hardware — names and quantities only. The factory CSV keyed to the service tag is the only
|
||||
|
||||
@@ -12,19 +12,30 @@ comfy-dev asked whether `/storetank` rides along or the stack is rebuilt from so
|
||||
`zfs send` is **incremental**: snapshot now, ship the base over ~4 hours while irv-ml1 keeps
|
||||
serving, then a small delta at cutover. Near-zero service interruption.
|
||||
|
||||
## Why (b) — physically moving the disks — was never on the table
|
||||
## Why (b) — physically moving the disks — was rejected
|
||||
|
||||
`/storetank` is a two-disk **mirror** of Crucial MX500 2TB SATA SSDs, so it is genuinely
|
||||
portable hardware. That fact is true and **irrelevant**.
|
||||
portable hardware. Rejected anyway: the send is non-disruptive at a measured 99 MB/s, and
|
||||
physically handling disks across sites to save ~4 hours of background transfer is a bad
|
||||
trade. Not reopened.
|
||||
|
||||
**irv-ml1 is NOT being decommissioned.** It runs arbo, comfyui, tts-gateway, dots-tts,
|
||||
voice-studio, waterland-studio, yt-voice-clipper and dockge — and **comfyui bind-mounts
|
||||
`/storetank/arbo/models`**. Pulling those disks does not inconvenience a source box that no
|
||||
longer needs them; it guts a live one.
|
||||
⚠ **CORRECTED 2026-09-01 by comfy-dev — the ORIGINAL reason given was wrong.** infra-ops
|
||||
wrote that (b) "was never available" because *"irv-ml1 keeps running its eight services and
|
||||
needs the pool."* comfy-dev enumerated the mounts of all twelve running containers:
|
||||
|
||||
⚠ **The lesson:** infra-ops asked whether the DATA could move and never asked whether the
|
||||
SOURCE still needed it. One `docker ps` away, at any point. Recommended (b) twice on the
|
||||
strength of a true-but-irrelevant fact. See [[2026-09-01-wrong-object-measurement]].
|
||||
comfyui -> /storetank/arbo/models:/basedir/models <- the complete list
|
||||
|
||||
arbo, dockge, tts-gateway, dots-tts, voice-studio, waterland-studio, yt-voice-clipper (x2),
|
||||
omnivoice-ref, studio-gate and parakeet **do not touch `/storetank` at all.** Exactly one
|
||||
container mounts it, and it is the one that moves. So (b) was unavailable **during the
|
||||
transition** — while comfyui still serves from those disks — not structurally. After cutover
|
||||
the constraint evaporates. Right conclusion, wrong reason, and the wrong reason mattered
|
||||
because it mis-describes what irv-ml1 is holding afterwards (see the retain question below).
|
||||
|
||||
⚠ **The lesson, restated:** infra-ops reasoned about the BOX when the question was about the
|
||||
MOUNT — the same wrong-object shape as asking whether the DATA could move without asking
|
||||
whether the SOURCE still needed it. Both were one `docker ps` / `docker inspect` away.
|
||||
See [[2026-09-01-wrong-object-measurement]].
|
||||
|
||||
## (c) rebuild-from-source: rejected on reproducibility, not time
|
||||
|
||||
@@ -65,11 +76,59 @@ safetensors are architecture-neutral and travel anywhere. Container image, torch
|
||||
and attention kernels do not. "Migrate the model store" and "migrate ComfyUI" are different
|
||||
jobs, and the 1.38 TB transfer is the easy half.
|
||||
|
||||
## ⚠ DESTINATION SIZING — the as-bought Ada disks are SMALLER than the source pool
|
||||
|
||||
Measured 2026-09-01 (`zpool list` / `zfs list` on irv-ml1), not derived:
|
||||
|
||||
source pool storetank SIZE 1.81T ALLOC 1.45T FREE 367G CAP 80%
|
||||
mirror-0: 2x ata-CT2000MX500SSD1 (2 TB class)
|
||||
compression off, compressratio 1.00x, logicalused == used, 0 snapshots
|
||||
|
||||
Safetensors are **incompressible** (1.00x measured) — the destination needs the same physical
|
||||
bytes regardless of what compression is set to at recv. There is no free win there.
|
||||
|
||||
The R750xa shipped **2x 1.92 TB** SATA SSD. Those are smaller than the 2 TB MX500s:
|
||||
|
||||
dest pool (2x 1.92 TB mirror) ~1.74 TiB <- SMALLER than the 1.81 TiB source
|
||||
payload today 1.45 TiB
|
||||
payload after comfy-dev's batch ~1.56 TiB
|
||||
occupancy on arrival ~90%
|
||||
|
||||
It fits, and it arrives already past the ~80% line where ZFS allocation degrades, with no
|
||||
growth room, for a workload whose whole history is "the model store grows."
|
||||
|
||||
**Recommendation to the operator: add 2x 2 TB SATA SSD to the R750xa order** — six of eight
|
||||
bays are free, HBA355i has the ports, two mirror vdevs striped gives ~3.49 TiB at ~45%
|
||||
occupancy with redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6`
|
||||
and 2x `930-00030-1546-000`, and ordering now does not gate the cutover.
|
||||
→ [[2026-09-01-ada-inference-server-r750xa]]
|
||||
|
||||
The alternative — **striping the as-bought pair** for the same ~3.49 TiB with no redundancy —
|
||||
is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy.
|
||||
|
||||
## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN
|
||||
|
||||
After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of
|
||||
tts/llm odds and ends, so 1.4 TB is either reclaimable capacity or a free full backup.
|
||||
comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent reasons:
|
||||
|
||||
1. **Reproducibility (comfy-dev's).** Two of their pins have gone permanently paid, so a
|
||||
from-source rebuild provably cannot reproduce today's stack. The irv-ml1 copy is the only
|
||||
surviving reproduction path for some of those weights. Reclaiming early converts a
|
||||
recoverable situation into an unrecoverable one for zero gain — nothing waits on the space.
|
||||
2. **Redundancy (infra-ops').** If the operator declines the two extra drives and we stripe
|
||||
the as-bought pair, retain stops being a recommendation and becomes a **requirement**.
|
||||
|
||||
⚠ Do not reclaim `/storetank` as routine post-migration tidying.
|
||||
|
||||
## Open
|
||||
|
||||
- **Cutover window** — operator's, not yet set.
|
||||
- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered.
|
||||
- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended.
|
||||
- **comfy-dev's ~112 GB batch** — unheld by infra-ops, but they are deliberately NOT pulling
|
||||
until the operator approves putting that much onto his infrastructure. Their call, correct
|
||||
instinct. Manifest pinned and staged (`29324e9`).
|
||||
instinct. Manifest pinned and staged (`29324e9`). **Landing as of 2026-09-01** — SHAs
|
||||
verifying green; source pool `used` moved 1.39T -> 1.45T during the exchange.
|
||||
|
||||
Thread: `01M1EYBSYA4QRYK54PX0K1S8CS`.
|
||||
|
||||
+12
-5
@@ -121,7 +121,9 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
||||
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
|
||||
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
|
||||
factory 256 GB) and 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters.** Everything else waits
|
||||
factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
|
||||
(the as-bought 1.92 TB pair mirrored is ~1.74 TiB — SMALLER than the 1.81 TiB pool it
|
||||
receives from, landing the migration at ~90% full). Everything else waits
|
||||
on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
|
||||
omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
|
||||
add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
|
||||
@@ -133,9 +135,14 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
||||
without deciding that first. Exactly TWO 3c launches, only one died.
|
||||
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
|
||||
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
|
||||
`zfs send`, ~3.9 h for 1.38 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
|
||||
Branch (b) was never available. **comfy-dev is holding a ~112 GB pull awaiting the
|
||||
operator's go**, not infra-ops'. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
||||
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
|
||||
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
|
||||
`/storetank`, so (b) was unavailable during the transition, not structurally).
|
||||
**TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order
|
||||
or the pool lands at ~90%; (2) retain vs reclaim irv-ml1's `/storetank` after the move —
|
||||
RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the
|
||||
stack; and under a no-spend striped destination it is the only redundancy). comfy-dev's
|
||||
~112 GB pull is landing now. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
||||
- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
|
||||
by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
|
||||
"trust the HTTP status, not `error_code`" still applies on running instances.
|
||||
@@ -157,7 +164,7 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
||||
|
||||
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
|
||||
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — branch (b) was never available because irv-ml1 keeps its eight services.** 99 MB/s measured; ~3.9 h. Also records the two-boxes confusion: the Ada box and the GX10 are DIFFERENT machines. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
||||
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). The R750xa's as-bought 2× 1.92 TB mirrored is ~1.74 TiB → arrival at **~90%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.49 TiB at ~45% with redundancy. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
||||
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
|
||||
- `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md`
|
||||
- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
|
||||
|
||||
Reference in New Issue
Block a user