memory: snapshot — operator leaning 6 drives; records the layout analysis

Six drives fills all eight bays, which turns a capacity top-up into a
one-shot build decision. Recorded because the reasoning survives whatever
he picks:

- raidz2 over mirrors. Workload is large sequential reads of safetensors,
  ARC fronts it, SSD resilver has no seek penalty, and "expand two at a
  time" is meaningless once every bay is full. raidz2 survives any two
  failures; 4x mirrors dies to an unlucky pair.
- Buy 8, not 6. A raidz vdev caps every member at the smallest, so the
  two as-bought 1.92 TB drives would cap all eight and put two used
  drives of unknown endurance inside the parity set.
- Drive size is now the permanent ceiling. SAS/SATA backplane, all bays
  full, and the free-PCIe-slot inventory is still unpulled.

Runway table included with an explicit caveat that the growth rate is
projected off one acquisition batch, not measured.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 17:15:52 -07:00
parent 3180dc85fa
commit 5c88280f9f
@@ -154,6 +154,54 @@ other (1.75 + 1.82 = 3.57 TiB).
The no-spend alternative — **striping the as-bought pair** for ~3.49 TiB with no redundancy —
is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy.
## ⚠ OPERATOR LEANING 6 DRIVES (2026-09-01, "most likely I add 6 2TB SSDs") — NOT YET DECIDED
Six fills **all eight bays**, which turns a capacity top-up into a one-shot, hard-to-reverse
build decision. Three things change:
**1. Layout stops being obvious and becomes permanent.** raidz geometry is fixed at pool
creation — no conversion to mirrors, no widening, without destroy-and-rebuild.
layout (8 drives) usable* full survives
4x mirror vdevs 6.8 TiB 21.7% 1 per vdev (a both-halves loss kills the pool)
8-wide raidz2 10.1 TiB 14.6% ANY 2 drives
* derated ~3-4% for parity padding + ZFS slop
**Recommend raidz2.** The usual mirror arguments do not apply here: the workload is large
sequential reads of big safetensors (raidz's weakness is small random IOPS), 256 GB of ARC
fronts the metadata and hot checkpoints, SSD resilver has no seek penalty so the classic
raidz-resilver objection is void, and **"expand two drives at a time" is meaningless once all
eight bays are full.** raidz2 also survives ANY two failures where 4x mirrors dies to an
unlucky pair. Neither layout is throughput-limited — the HBA355i (PCIe 4.0 x8) and the
~6 GB/s backplane ceiling are both far above what checkpoint loading needs.
**2. Buy 8, not 6 — a raidz vdev caps every member at the SMALLEST.** The two as-bought
1.92 TB Hynix drives would cap all eight at 1.92 TB (~480 GB thrown away) AND put two used
drives of unknown remaining write-endurance, from the reseller who already stripped this box,
inside the parity set. Two more drives makes the set uniform and demotes the 1.92s to cold
spares. Cheap insurance on a pool that cannot be re-laid-out later.
**3. Drive SIZE is now the permanent ceiling.** The `321-BGKG` backplane is SAS/SATA only, so
with all bays full the ONLY later expansion is a PCIe NVMe add-in card — and the free-slot
inventory is still unpulled. Runway, measured in "days like the ~100 GiB acquisition day of
2026-09-01":
2x 2TB (the earlier rec) 3.46 TiB usable 42% ~13 days
6x 2TB, mixed, raidz2 10.06 TiB usable 14.6% ~67 days
8x 2TB, uniform, raidz2 10.48 TiB usable 14.0% ~71 days
8x 4TB, uniform, raidz2 20.95 TiB usable 7.0% ~157 days
⚠ The "days like today" rate is a **projection off a single deliberate acquisition batch**,
not a measured steady rate. Treat as an order-of-magnitude comparator only.
**Non-issues, closed:** 8 SATA SSDs draw ~40 W against a ~1 kW box; HBA355i and backplane are
nowhere near limiting.
**What 6-8 drives moots:** at ~14% the prune audit becomes pure hygiene with no capacity
stake at all. Retain on irv-ml1 stays recommended, but on **reproducibility alone** — the
redundancy argument for retain disappears under raidz2.
## Incidental — LTX 2.5 was already on disk, uncatalogued
comfy-dev's diff turned up 34 GiB of LTX 2.5 (`ltx-2.5-22b-distilled-transformer` 20.0 GiB +
@@ -197,7 +245,9 @@ comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent rea
## Open
- **Cutover window** — operator's, not yet set.
- **2x 2 TB SATA SSD for the R750xa** — surfaced to the operator, not yet ordered.
- **Drives for the R750xa** — operator leaning **6x 2 TB** (fills all 8 bays). Open: 6-vs-8
(uniform set), 2 TB vs 4 TB (bays-full makes size the permanent ceiling), and raidz2 vs
mirrors (recommend raidz2). Not yet ordered.
- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended.
- **comfy-dev's per-file prune audit** — offered, explicitly NOT gating the cutover, waiting
on the operator wanting it. Their judgment, not infra-ops'.