diff --git a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md index 11f6691..eb5fbca 100644 --- a/persistent-memory.d/2026-09-01-ada-migration-branch-a.md +++ b/persistent-memory.d/2026-09-01-ada-migration-branch-a.md @@ -227,6 +227,53 @@ Three findings in one day, all from measurement, each beating a confident deriva Same lesson as `docs/pfi/training-throughput-playbook.md` §4, on storage instead of training. +## Build-time: ARC sizing, and why NOT a ramfs/tmpfs model tier + +Operator asked (2026-09-01) whether 512 GB justifies a ramfs for hot models, then self- +corrected to "or just let ZFS ARC handle it with a 128 GB pool or something." **ARC, yes. +128 GB, no — that is BELOW the Linux default.** + +**Do not build a ramfs/tmpfs model tier.** ARC already is the RAM cache, and beats it: + +- **ARC is elastic; tmpfs is a reservation.** A tmpfs holding a checkpoint costs that RAM + whether or not it is in use. ARC evicts cold data and hands the RAM back. On a box that + loads *different* models per workflow, adaptive wins outright. +- **tmpfs requires a curated hot-list that rots.** ARC decides from live access patterns and + is scan-resistant by construction (recency AND frequency). +- **`ramfs` specifically is dangerous** — no size limit, cannot swap, fills until OOM. If + ever used at all, `tmpfs`. +- **Not durable** — a 200 GB copy-in on every boot. +- **No speed to win.** Both are a memcpy; the floor is the host→VRAM PCIe 4.0 x16 hop + (~20 GB/s real) plus CPU-side deserialization, and neither tier can beat it: + + cold, 2x SATA mirror 1.1 GB/s ~24 s for a 25 GB checkpoint + cold, 8-wide raidz2 3.3 GB/s ~9 s + warm in ARC ~12 GB/s ~3 s + tmpfs ~12 GB/s ~3 s <- identical, at the cost of curation + +⚠ **The one honest argument FOR tmpfs:** safetensors loads via **mmap**, and on ZFS-on-Linux +mmap'd pages are double-buffered (ARC + page cache), so a 25 GB checkpoint can occupy ~50 GB. +tmpfs is single-copy. At 512 GB this is a capacity annoyance, not a reason to build a second +storage tier — size ARC for it instead. + +**`zfs_arc_max`: recommend ~320 GB (62.5%).** OpenZFS on Linux already defaults to **50% of +RAM = 256 GB** here, so *doing nothing* is twice as good as the proposed 128 GB. A hot working +set of ~6 big checkpoints is ~150 GB, ~300 GB if mmap double-buffers — 128 GB would thrash +inside a single session. Do not go past ~75%: ARC shrink-under-pressure is laggy on Linux and +a memory-hungry ComfyUI on the same box can lose the race. + +⚠ **This is NOT the [[idle VRAM is reserved, not waste]] case.** `gpu-memory-utilization` +pre-allocates a fixed pool; `zfs_arc_max` is a **ceiling on an elastic cache** that returns +memory under pressure. Raising the ARC cap reserves nothing. Do not apply the VRAM rule here. + +⚠ **`recordsize=1M` cannot be won via `zfs send`.** Records are sized when written; `recv` +reproduces the source's block structure, so a 128K source (storetank is at the 128K default) +yields 128K records on the destination no matter what the property says. Getting 1M would mean +rsync instead of send — **not worth it**: the incremental-delta property at cutover is worth +far more, and on flash the 128K penalty is metadata overhead (~40 MB of ARC headers per 25 GB +file vs ~5 MB), not throughput. Set `recordsize=1M` only on datasets built fresh, not received. +Verify the recv behaviour at cutover rather than taking this on faith. + ## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of