memory: snapshot — operator leaning 6 drives; records the layout analysis
Six drives fills all eight bays, which turns a capacity top-up into a one-shot build decision. Recorded because the reasoning survives whatever he picks: - raidz2 over mirrors. Workload is large sequential reads of safetensors, ARC fronts it, SSD resilver has no seek penalty, and "expand two at a time" is meaningless once every bay is full. raidz2 survives any two failures; 4x mirrors dies to an unlucky pair. - Buy 8, not 6. A raidz vdev caps every member at the smallest, so the two as-bought 1.92 TB drives would cap all eight and put two used drives of unknown endurance inside the parity set. - Drive size is now the permanent ceiling. SAS/SATA backplane, all bays full, and the free-PCIe-slot inventory is still unpulled. Runway table included with an explicit caveat that the growth rate is projected off one acquisition batch, not measured. Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
@@ -154,6 +154,54 @@ other (1.75 + 1.82 = 3.57 TiB).
|
||||
The no-spend alternative — **striping the as-bought pair** for ~3.49 TiB with no redundancy —
|
||||
is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy.
|
||||
|
||||
## ⚠ OPERATOR LEANING 6 DRIVES (2026-09-01, "most likely I add 6 2TB SSDs") — NOT YET DECIDED
|
||||
|
||||
Six fills **all eight bays**, which turns a capacity top-up into a one-shot, hard-to-reverse
|
||||
build decision. Three things change:
|
||||
|
||||
**1. Layout stops being obvious and becomes permanent.** raidz geometry is fixed at pool
|
||||
creation — no conversion to mirrors, no widening, without destroy-and-rebuild.
|
||||
|
||||
layout (8 drives) usable* full survives
|
||||
4x mirror vdevs 6.8 TiB 21.7% 1 per vdev (a both-halves loss kills the pool)
|
||||
8-wide raidz2 10.1 TiB 14.6% ANY 2 drives
|
||||
* derated ~3-4% for parity padding + ZFS slop
|
||||
|
||||
**Recommend raidz2.** The usual mirror arguments do not apply here: the workload is large
|
||||
sequential reads of big safetensors (raidz's weakness is small random IOPS), 256 GB of ARC
|
||||
fronts the metadata and hot checkpoints, SSD resilver has no seek penalty so the classic
|
||||
raidz-resilver objection is void, and **"expand two drives at a time" is meaningless once all
|
||||
eight bays are full.** raidz2 also survives ANY two failures where 4x mirrors dies to an
|
||||
unlucky pair. Neither layout is throughput-limited — the HBA355i (PCIe 4.0 x8) and the
|
||||
~6 GB/s backplane ceiling are both far above what checkpoint loading needs.
|
||||
|
||||
**2. Buy 8, not 6 — a raidz vdev caps every member at the SMALLEST.** The two as-bought
|
||||
1.92 TB Hynix drives would cap all eight at 1.92 TB (~480 GB thrown away) AND put two used
|
||||
drives of unknown remaining write-endurance, from the reseller who already stripped this box,
|
||||
inside the parity set. Two more drives makes the set uniform and demotes the 1.92s to cold
|
||||
spares. Cheap insurance on a pool that cannot be re-laid-out later.
|
||||
|
||||
**3. Drive SIZE is now the permanent ceiling.** The `321-BGKG` backplane is SAS/SATA only, so
|
||||
with all bays full the ONLY later expansion is a PCIe NVMe add-in card — and the free-slot
|
||||
inventory is still unpulled. Runway, measured in "days like the ~100 GiB acquisition day of
|
||||
2026-09-01":
|
||||
|
||||
2x 2TB (the earlier rec) 3.46 TiB usable 42% ~13 days
|
||||
6x 2TB, mixed, raidz2 10.06 TiB usable 14.6% ~67 days
|
||||
8x 2TB, uniform, raidz2 10.48 TiB usable 14.0% ~71 days
|
||||
8x 4TB, uniform, raidz2 20.95 TiB usable 7.0% ~157 days
|
||||
|
||||
⚠ The "days like today" rate is a **projection off a single deliberate acquisition batch**,
|
||||
not a measured steady rate. Treat as an order-of-magnitude comparator only.
|
||||
|
||||
**Non-issues, closed:** 8 SATA SSDs draw ~40 W against a ~1 kW box; HBA355i and backplane are
|
||||
nowhere near limiting.
|
||||
|
||||
**What 6-8 drives moots:** at ~14% the prune audit becomes pure hygiene with no capacity
|
||||
stake at all. Retain on irv-ml1 stays recommended, but on **reproducibility alone** — the
|
||||
redundancy argument for retain disappears under raidz2.
|
||||
|
||||
|
||||
## Incidental — LTX 2.5 was already on disk, uncatalogued
|
||||
|
||||
comfy-dev's diff turned up 34 GiB of LTX 2.5 (`ltx-2.5-22b-distilled-transformer` 20.0 GiB +
|
||||
@@ -197,7 +245,9 @@ comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent rea
|
||||
## Open
|
||||
|
||||
- **Cutover window** — operator's, not yet set.
|
||||
- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered.
|
||||
- **Drives for the R750xa** — operator leaning **6x 2 TB** (fills all 8 bays). Open: 6-vs-8
|
||||
(uniform set), 2 TB vs 4 TB (bays-full makes size the permanent ceiling), and raidz2 vs
|
||||
mirrors (recommend raidz2). Not yet ordered.
|
||||
- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended.
|
||||
- **comfy-dev's per-file prune audit** — offered, explicitly NOT gating the cutover, waiting
|
||||
on the operator wanting it. Their judgment, not infra-ops'.
|
||||
|
||||
Reference in New Issue
Block a user