1 Commits

Author SHA1 Message Date
vh f875f746b8 feat(playbooks): nh3-dev memory forensics — and the OOM hog is Claude Code
forseti asked for journald kernel persistence plus sysstat, on the
premise that nh3-dev's three OOM events in 14 days left no evidence.

The premise was wrong. journald has been persistent all along: 15,068
kernel entries in the 82-day previous boot and 351 OOM records across
retained boots, full task tables included. `journalctl -b -1 -k`
returned one entry because it ran as a user in neither adm nor
systemd-journal, and journalctl shows only your own messages in that
case. The same artifact produced the "journal stops at 05:36:08 with no
shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM
kills logged at 05:38, 05:40 and 05:42.

So the fix for "no evidence" is a group membership, not a logging
change: usermod -aG adm lkraven, which is the group Debian's journald
ACL names explicitly.

With the journal readable the attribution is already in it. The
versioned Claude Code binary lives at .local/share/claude/versions/,
so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as
are those named claude. Every one of the twelve largest resident
processes ever recorded on this box is a CC session, topping out at
18.4 GB. Everything else killed is 30-55 MB collateral, which clears
the althing daemons by measurement rather than by their own sampling.

sysstat and atop are added because the journal records the moment of
the kill, not the ramp, and names the victim rather than the winner.
atop was not requested and is the one that matters: with a dozen panes
open, only a per-process timeseries says which session was growing.

Not done: a cgroup cap on CC sessions. It is the real mitigation and it
would kill long-running sessions mid-work, so it goes to the operator.
2026-08-28 06:03:43 -07:00