Files
esh-pfi-infrastructure/persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md
T
vh e52def115c memory: snapshot — the ops log, and a day spent on instruments that report without looking
Archived 15 entries (Recent decisions 14, Tried and abandoned 1) oldest-first
to archival-memory.md; 5 held back on the open-deferred-work guard and 164 on
the 14-day guard, so the index stays over the soft cap at 477 lines. An
over-cap file that keeps live decisions beats a scannable one that lost a
belayed item.

Four new detail files cover the day: the ops log and its four self-inflicted
failure modes, the Booth's two dead controls and the four-iteration layout
probe, the Gitea org grant plus the dead claude-bot token that had been
misreporting permissions, and the disk triage that rescued a LoRA adapter from
a directory this box sweeps at three days.

lv-mccarthy's run outcome remains unverified after two days and is the first
line of the in-flight section and step 1 of the handoff.
2026-09-21 14:26:55 -07:00

2.4 KiB
Raw Blame History

[2026-09-21] nh3-dev disk triage: 27 GB reclaimed, and a LoRA adapter rescued from a directory the box sweeps every 3 days

Beszel alerted root >85%. Reclaimed 7 GB from regenerable caches (uv cache prune, npm, pip), then a deep dive found the real shape.

⚠⚠ The headline was not disk. /tmp held 19 GB of Claude Code session scratchpads, and inside one of them sat the babyyarros LoRA adapter — 252 MB, r=32/α=64 on Qwen3-4B-Instruct, with its loss series and provenance — existing nowhere else: not on /mnt/smithy, not under ~/development. /etc/tmpfiles.d/tmp.conf sets D /tmp … 3d, an admin file from 2026-07-18 that overrides the stock no-age rule, and the cleaner runs daily.

Rescued to /mnt/smithy/adapter-rescue/babyyarros-20260921, verified by content: sha256 63fda6cc… matching both the source and the adapter.sha256 recorded at training time.

⚠ Two corrections I made to myself during the dive, both worth keeping: I alarmed that shutterchute's RAW deliverables were 19 hours from deletion — their mtimes were that day, a live session working. And I suspected my own du/ find had reset the atime clock and manufactured the "0 would-remove" result; it had not (relatime, and an untouched comparison file still showed an old atime) — but it was right to check before trusting a number my own measurement could have created.

Operator-authorized deletions: 7.6 GB duplicate Qwen base shards, 12 GB models-staging/retro-diffusion (cold since 09 Aug), 7.5 GB splat-assets (cold since 15 Aug), 178 session dirs idle 7d+. 39 GB → 66 GB free, 84% → 72%.

⚠⚠ MY PRUNE DELETED AN ACTIVE SESSION'S DIRECTORY. -mtime +7 on a session dir is an unsound liveness test: a directory's mtime does not change when files are written into its subdirectories. dfacccde/ looked 7+ days idle while dfacccde/tasks/ was being written continuously. Cost: one lost tool output; recreated. All four of my safety assertions (right root, right depth, own session excluded, shutterchute excluded) passed — I checked the paths were right and never checked the liveness test was sound. Do not re-run that predicate. A correct version checks the deepest recent file, or cross-references running claude PIDs.

Also found: sudo -n requires a password as lkraven locally on nh3-dev, while ssh infra-ops@localhost has NOPASSWD.