memory: snapshot — ARC sizing for the Ada box; no ramfs model tier
Operator asked whether 512 GB justifies a ramfs for hot models, then self-corrected toward ARC with a 128 GB cap. - No ramfs/tmpfs tier. ARC is the same cache done adaptively: no curated hot-list to rot, no boot-time copy-in, and the memory comes back under pressure. Neither tier beats the host-to-VRAM PCIe hop anyway, so the warm-load time is identical. - 128 GB is below the OpenZFS Linux default of 50% of RAM, so doing nothing already gives 256 GB. Recommend ~320 GB, not past ~75%. - Recorded the one honest argument for tmpfs: safetensors mmap gets double-buffered on ZFS-on-Linux, so a checkpoint can cost ~2x. Sized for, not architected around. - Flagged that this is not the idle-VRAM-is-reserved case: arc_max is a ceiling on an elastic cache, not a preallocation. - recordsize=1M cannot be won through zfs send, since recv reproduces the source's block structure. Not worth losing incremental send over; the 128K cost on flash is metadata overhead, not throughput. Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
@@ -227,6 +227,53 @@ Three findings in one day, all from measurement, each beating a confident deriva
|
||||
|
||||
Same lesson as `docs/pfi/training-throughput-playbook.md` §4, on storage instead of training.
|
||||
|
||||
## Build-time: ARC sizing, and why NOT a ramfs/tmpfs model tier
|
||||
|
||||
Operator asked (2026-09-01) whether 512 GB justifies a ramfs for hot models, then self-
|
||||
corrected to "or just let ZFS ARC handle it with a 128 GB pool or something." **ARC, yes.
|
||||
128 GB, no — that is BELOW the Linux default.**
|
||||
|
||||
**Do not build a ramfs/tmpfs model tier.** ARC already is the RAM cache, and beats it:
|
||||
|
||||
- **ARC is elastic; tmpfs is a reservation.** A tmpfs holding a checkpoint costs that RAM
|
||||
whether or not it is in use. ARC evicts cold data and hands the RAM back. On a box that
|
||||
loads *different* models per workflow, adaptive wins outright.
|
||||
- **tmpfs requires a curated hot-list that rots.** ARC decides from live access patterns and
|
||||
is scan-resistant by construction (recency AND frequency).
|
||||
- **`ramfs` specifically is dangerous** — no size limit, cannot swap, fills until OOM. If
|
||||
ever used at all, `tmpfs`.
|
||||
- **Not durable** — a 200 GB copy-in on every boot.
|
||||
- **No speed to win.** Both are a memcpy; the floor is the host→VRAM PCIe 4.0 x16 hop
|
||||
(~20 GB/s real) plus CPU-side deserialization, and neither tier can beat it:
|
||||
|
||||
cold, 2x SATA mirror 1.1 GB/s ~24 s for a 25 GB checkpoint
|
||||
cold, 8-wide raidz2 3.3 GB/s ~9 s
|
||||
warm in ARC ~12 GB/s ~3 s
|
||||
tmpfs ~12 GB/s ~3 s <- identical, at the cost of curation
|
||||
|
||||
⚠ **The one honest argument FOR tmpfs:** safetensors loads via **mmap**, and on ZFS-on-Linux
|
||||
mmap'd pages are double-buffered (ARC + page cache), so a 25 GB checkpoint can occupy ~50 GB.
|
||||
tmpfs is single-copy. At 512 GB this is a capacity annoyance, not a reason to build a second
|
||||
storage tier — size ARC for it instead.
|
||||
|
||||
**`zfs_arc_max`: recommend ~320 GB (62.5%).** OpenZFS on Linux already defaults to **50% of
|
||||
RAM = 256 GB** here, so *doing nothing* is twice as good as the proposed 128 GB. A hot working
|
||||
set of ~6 big checkpoints is ~150 GB, ~300 GB if mmap double-buffers — 128 GB would thrash
|
||||
inside a single session. Do not go past ~75%: ARC shrink-under-pressure is laggy on Linux and
|
||||
a memory-hungry ComfyUI on the same box can lose the race.
|
||||
|
||||
⚠ **This is NOT the [[idle VRAM is reserved, not waste]] case.** `gpu-memory-utilization`
|
||||
pre-allocates a fixed pool; `zfs_arc_max` is a **ceiling on an elastic cache** that returns
|
||||
memory under pressure. Raising the ARC cap reserves nothing. Do not apply the VRAM rule here.
|
||||
|
||||
⚠ **`recordsize=1M` cannot be won via `zfs send`.** Records are sized when written; `recv`
|
||||
reproduces the source's block structure, so a 128K source (storetank is at the 128K default)
|
||||
yields 128K records on the destination no matter what the property says. Getting 1M would mean
|
||||
rsync instead of send — **not worth it**: the incremental-delta property at cutover is worth
|
||||
far more, and on flash the 128K penalty is metadata overhead (~40 MB of ARC headers per 25 GB
|
||||
file vs ~5 MB), not throughput. Set `recordsize=1M` only on datasets built fresh, not received.
|
||||
Verify the recv behaviour at cutover rather than taking this on faith.
|
||||
|
||||
## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN
|
||||
|
||||
After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of
|
||||
|
||||
Reference in New Issue
Block a user