memory: snapshot — ARC sizing for the Ada box; no ramfs model tier

Operator asked whether 512 GB justifies a ramfs for hot models, then
self-corrected toward ARC with a 128 GB cap.

- No ramfs/tmpfs tier. ARC is the same cache done adaptively: no curated
  hot-list to rot, no boot-time copy-in, and the memory comes back under
  pressure. Neither tier beats the host-to-VRAM PCIe hop anyway, so the
  warm-load time is identical.
- 128 GB is below the OpenZFS Linux default of 50% of RAM, so doing
  nothing already gives 256 GB. Recommend ~320 GB, not past ~75%.
- Recorded the one honest argument for tmpfs: safetensors mmap gets
  double-buffered on ZFS-on-Linux, so a checkpoint can cost ~2x. Sized
  for, not architected around.
- Flagged that this is not the idle-VRAM-is-reserved case: arc_max is a
  ceiling on an elastic cache, not a preallocation.
- recordsize=1M cannot be won through zfs send, since recv reproduces the
  source's block structure. Not worth losing incremental send over; the
  128K cost on flash is metadata overhead, not throughput.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
2026-09-01 17:26:33 -07:00
parent 3410d578e3
commit 32fd1dbe4a
@@ -227,6 +227,53 @@ Three findings in one day, all from measurement, each beating a confident deriva
Same lesson as `docs/pfi/training-throughput-playbook.md` §4, on storage instead of training.
## Build-time: ARC sizing, and why NOT a ramfs/tmpfs model tier
Operator asked (2026-09-01) whether 512 GB justifies a ramfs for hot models, then self-
corrected to "or just let ZFS ARC handle it with a 128 GB pool or something." **ARC, yes.
128 GB, no — that is BELOW the Linux default.**
**Do not build a ramfs/tmpfs model tier.** ARC already is the RAM cache, and beats it:
- **ARC is elastic; tmpfs is a reservation.** A tmpfs holding a checkpoint costs that RAM
whether or not it is in use. ARC evicts cold data and hands the RAM back. On a box that
loads *different* models per workflow, adaptive wins outright.
- **tmpfs requires a curated hot-list that rots.** ARC decides from live access patterns and
is scan-resistant by construction (recency AND frequency).
- **`ramfs` specifically is dangerous** — no size limit, cannot swap, fills until OOM. If
ever used at all, `tmpfs`.
- **Not durable** — a 200 GB copy-in on every boot.
- **No speed to win.** Both are a memcpy; the floor is the host→VRAM PCIe 4.0 x16 hop
(~20 GB/s real) plus CPU-side deserialization, and neither tier can beat it:
cold, 2x SATA mirror 1.1 GB/s ~24 s for a 25 GB checkpoint
cold, 8-wide raidz2 3.3 GB/s ~9 s
warm in ARC ~12 GB/s ~3 s
tmpfs ~12 GB/s ~3 s <- identical, at the cost of curation
**The one honest argument FOR tmpfs:** safetensors loads via **mmap**, and on ZFS-on-Linux
mmap'd pages are double-buffered (ARC + page cache), so a 25 GB checkpoint can occupy ~50 GB.
tmpfs is single-copy. At 512 GB this is a capacity annoyance, not a reason to build a second
storage tier — size ARC for it instead.
**`zfs_arc_max`: recommend ~320 GB (62.5%).** OpenZFS on Linux already defaults to **50% of
RAM = 256 GB** here, so *doing nothing* is twice as good as the proposed 128 GB. A hot working
set of ~6 big checkpoints is ~150 GB, ~300 GB if mmap double-buffers — 128 GB would thrash
inside a single session. Do not go past ~75%: ARC shrink-under-pressure is laggy on Linux and
a memory-hungry ComfyUI on the same box can lose the race.
**This is NOT the [[idle VRAM is reserved, not waste]] case.** `gpu-memory-utilization`
pre-allocates a fixed pool; `zfs_arc_max` is a **ceiling on an elastic cache** that returns
memory under pressure. Raising the ARC cap reserves nothing. Do not apply the VRAM rule here.
**`recordsize=1M` cannot be won via `zfs send`.** Records are sized when written; `recv`
reproduces the source's block structure, so a 128K source (storetank is at the 128K default)
yields 128K records on the destination no matter what the property says. Getting 1M would mean
rsync instead of send — **not worth it**: the incremental-delta property at cutover is worth
far more, and on flash the 128K penalty is metadata overhead (~40 MB of ARC headers per 25 GB
file vs ~5 MB), not throughput. Set `recordsize=1M` only on datasets built fresh, not received.
Verify the recv behaviour at cutover rather than taking this on faith.
## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN
After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of