From 5c88280f9fc3a21a288b3d0856c11efb90e57e67 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 1 Sep 2026 17:15:52 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20operator=20lea?= =?UTF-8?q?ning=206=20drives;=20records=20the=20layout=20analysis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six drives fills all eight bays, which turns a capacity top-up into a one-shot build decision. Recorded because the reasoning survives whatever he picks: - raidz2 over mirrors. Workload is large sequential reads of safetensors, ARC fronts it, SSD resilver has no seek penalty, and "expand two at a time" is meaningless once every bay is full. raidz2 survives any two failures; 4x mirrors dies to an unlucky pair. - Buy 8, not 6. A raidz vdev caps every member at the smallest, so the two as-bought 1.92 TB drives would cap all eight and put two used drives of unknown endurance inside the parity set. - Drive size is now the permanent ceiling. SAS/SATA backplane, all bays full, and the free-PCIe-slot inventory is still unpulled. Runway table included with an explicit caveat that the growth rate is projected off one acquisition batch, not measured. Memory-only; no version bump per the SemVer SKIP list. --- .../2026-09-01-ada-migration-branch-a.md | 52 ++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md index 83ba139..11f6691 100644 --- a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md +++ b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md @@ -154,6 +154,54 @@ other (1.75 + 1.82 = 3.57 TiB). The no-spend alternative — **striping the as-bought pair** for ~3.49 TiB with no redundancy — is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy. +## ⚠ OPERATOR LEANING 6 DRIVES (2026-09-01, "most likely I add 6 2TB SSDs") — NOT YET DECIDED + +Six fills **all eight bays**, which turns a capacity top-up into a one-shot, hard-to-reverse +build decision. Three things change: + +**1. Layout stops being obvious and becomes permanent.** raidz geometry is fixed at pool +creation — no conversion to mirrors, no widening, without destroy-and-rebuild. + + layout (8 drives) usable* full survives + 4x mirror vdevs 6.8 TiB 21.7% 1 per vdev (a both-halves loss kills the pool) + 8-wide raidz2 10.1 TiB 14.6% ANY 2 drives + * derated ~3-4% for parity padding + ZFS slop + +**Recommend raidz2.** The usual mirror arguments do not apply here: the workload is large +sequential reads of big safetensors (raidz's weakness is small random IOPS), 256 GB of ARC +fronts the metadata and hot checkpoints, SSD resilver has no seek penalty so the classic +raidz-resilver objection is void, and **"expand two drives at a time" is meaningless once all +eight bays are full.** raidz2 also survives ANY two failures where 4x mirrors dies to an +unlucky pair. Neither layout is throughput-limited — the HBA355i (PCIe 4.0 x8) and the +~6 GB/s backplane ceiling are both far above what checkpoint loading needs. + +**2. Buy 8, not 6 — a raidz vdev caps every member at the SMALLEST.** The two as-bought +1.92 TB Hynix drives would cap all eight at 1.92 TB (~480 GB thrown away) AND put two used +drives of unknown remaining write-endurance, from the reseller who already stripped this box, +inside the parity set. Two more drives makes the set uniform and demotes the 1.92s to cold +spares. Cheap insurance on a pool that cannot be re-laid-out later. + +**3. Drive SIZE is now the permanent ceiling.** The `321-BGKG` backplane is SAS/SATA only, so +with all bays full the ONLY later expansion is a PCIe NVMe add-in card — and the free-slot +inventory is still unpulled. Runway, measured in "days like the ~100 GiB acquisition day of +2026-09-01": + + 2x 2TB (the earlier rec) 3.46 TiB usable 42% ~13 days + 6x 2TB, mixed, raidz2 10.06 TiB usable 14.6% ~67 days + 8x 2TB, uniform, raidz2 10.48 TiB usable 14.0% ~71 days + 8x 4TB, uniform, raidz2 20.95 TiB usable 7.0% ~157 days + +⚠ The "days like today" rate is a **projection off a single deliberate acquisition batch**, +not a measured steady rate. Treat as an order-of-magnitude comparator only. + +**Non-issues, closed:** 8 SATA SSDs draw ~40 W against a ~1 kW box; HBA355i and backplane are +nowhere near limiting. + +**What 6-8 drives moots:** at ~14% the prune audit becomes pure hygiene with no capacity +stake at all. Retain on irv-ml1 stays recommended, but on **reproducibility alone** — the +redundancy argument for retain disappears under raidz2. + + ## Incidental — LTX 2.5 was already on disk, uncatalogued comfy-dev's diff turned up 34 GiB of LTX 2.5 (`ltx-2.5-22b-distilled-transformer` 20.0 GiB + @@ -197,7 +245,9 @@ comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent rea ## Open - **Cutover window** — operator's, not yet set. -- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered. +- **Drives for the R750xa** — operator leaning **6x 2 TB** (fills all 8 bays). Open: 6-vs-8 + (uniform set), 2 TB vs 4 TB (bays-full makes size the permanent ceiling), and raidz2 vs + mirrors (recommend raidz2). Not yet ordered. - **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended. - **comfy-dev's per-file prune audit** — offered, explicitly NOT gating the cutover, waiting on the operator wanting it. Their judgment, not infra-ops'.