vh
f875f746b8
forseti asked for journald kernel persistence plus sysstat, on the premise that nh3-dev's three OOM events in 14 days left no evidence. The premise was wrong. journald has been persistent all along: 15,068 kernel entries in the 82-day previous boot and 351 OOM records across retained boots, full task tables included. `journalctl -b -1 -k` returned one entry because it ran as a user in neither adm nor systemd-journal, and journalctl shows only your own messages in that case. The same artifact produced the "journal stops at 05:36:08 with no shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM kills logged at 05:38, 05:40 and 05:42. So the fix for "no evidence" is a group membership, not a logging change: usermod -aG adm lkraven, which is the group Debian's journald ACL names explicitly. With the journal readable the attribution is already in it. The versioned Claude Code binary lives at .local/share/claude/versions/, so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as are those named claude. Every one of the twelve largest resident processes ever recorded on this box is a CC session, topping out at 18.4 GB. Everything else killed is 30-55 MB collateral, which clears the althing daemons by measurement rather than by their own sampling. sysstat and atop are added because the journal records the moment of the kill, not the ramp, and names the victim rather than the winner. atop was not requested and is the one that matters: with a dozen panes open, only a per-process timeseries says which session was growing. Not done: a cgroup cap on CC sessions. It is the real mitigation and it would kill long-running sessions mid-work, so it goes to the operator.
3.8 KiB
[2026-08-28] nh3-dev's three OOM events attribute to Claude Code — and the "no evidence" was a permissions artifact
Three memory-exhaustion events in 14 days (08-14 00:15, 08-26 09:58, 08-28 05:36, interval halving). forseti reported none could be attributed because "kernel messages are not being persisted to journald" and asked for journald persistence + sysstat.
⚠ THE PREMISE WAS WRONG — journald was persistent the whole time
journalctl -b -1 -k privileged 15,068 entries (82-day boot)
journalctl -b -1 -k unprivileged 4 entries
OOM records, all retained boots 351
journalctl silently shows only your own messages when you are in neither adm nor
systemd-journal, and prints the reason as a scroll-past hint. Two of the three "no evidence"
findings were that one artifact:
"kernel messages not persisted" -> they are, and every OOM task table is there
"journal stops 05:36:08, no shutdown" -> that is the USER's last entry; the true
boot -1 end is 05:47:04, with OOM kills
recorded at 05:38, 05:40, 05:42
Fix was usermod -aG adm lkraven, not a logging change. Debian's journald ACL names adm
explicitly (getfacl /var/log/journal/<machine-id> → group:adm:r-x). Existing shells keep
their old group set — re-login, or sg adm -c '...', which is also how to verify the grant
took rather than grepping /etc/group.
⚠ THE HOG IS CLAUDE CODE
/home/lkraven/.local/share/claude/versions/2.1.220 is the versioned CC binary, so OOM victims
named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as are those named claude.
largest anon-rss ever recorded on this box
18,434,696 kB 2.1.177 18.4 GB
15,788,764 kB 2.1.220 15.8 GB
15,154,008 kB 2.1.220 15.2 GB
14,994,376 kB 2.1.168 15.0 GB
-> every one of the top TWELVE is a CC session
29 of the OOM victims are CC. Everything else killed — althing-forseti (22), caddy (15), ttyd (11), zellij (6), the althing daemons — is 30-55 MB collateral, the OOM killer scraping for a few hundred MB. The althing v2 daemons are cleared by measurement.
"claude is 408 MB each" is a YOUNG session. Mature ones measure 5.4-18.4 GB. On 27 GB with 974 MB swap the ceiling is three or four mature sessions, not the ~66 a 408 MB figure implies. Aug 28's task table: two CC at 5.4 GB + three zellij servers at 1.14 GB.
Instrumentation added (playbooks/nh3-dev-memory-forensics.yaml, idempotent)
sysstat system-wide mem/CPU, 5-min cadence (not Debian's 10 -- a CC session can
add several GB inside one 10-minute bucket). sar -r
atop PER-PROCESS, 60s, 7-day retention. atop -r /var/log/atop/atop_YYYYMMDD
journald unchanged, already persistent, now READABLE
atop is the one that matters and it was not requested. The journal records the moment of the kill and names the victim; sar says the box filled up; only atop says which session was growing and how fast — the whole question when a dozen panes are open.
Open — operator's call, deliberately not taken
A cgroup memory cap on CC sessions is the real mitigation and it would kill long-running sessions mid-work. Surfaced, not decided. Instrumentation makes event four diagnosable, not less likely.
Lesson that generalises
A verify step I wrote failed while the setting was live: I grepped
systemctl show sysstat-collect.timer for my own input *:00/05, but systemd normalises it to
*-*-* *:00/5:00. Assert the effective value, not the string you wrote —
feedback_assert_effective_value_not_substring, caught here in my own instrumentation.
Reply: althing msg 01M147EWEZDT8Y0XW5FTHHEAQC, thread 01M1472ST5DSJNHR676X9F43AK.