vh
f875f746b8
forseti asked for journald kernel persistence plus sysstat, on the premise that nh3-dev's three OOM events in 14 days left no evidence. The premise was wrong. journald has been persistent all along: 15,068 kernel entries in the 82-day previous boot and 351 OOM records across retained boots, full task tables included. `journalctl -b -1 -k` returned one entry because it ran as a user in neither adm nor systemd-journal, and journalctl shows only your own messages in that case. The same artifact produced the "journal stops at 05:36:08 with no shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM kills logged at 05:38, 05:40 and 05:42. So the fix for "no evidence" is a group membership, not a logging change: usermod -aG adm lkraven, which is the group Debian's journald ACL names explicitly. With the journal readable the attribution is already in it. The versioned Claude Code binary lives at .local/share/claude/versions/, so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as are those named claude. Every one of the twelve largest resident processes ever recorded on this box is a CC session, topping out at 18.4 GB. Everything else killed is 30-55 MB collateral, which clears the althing daemons by measurement rather than by their own sampling. sysstat and atop are added because the journal records the moment of the kill, not the ramp, and names the victim rather than the winner. atop was not requested and is the one that matters: with a dozen panes open, only a per-process timeseries says which session was growing. Not done: a cgroup cap on CC sessions. It is the real mitigation and it would kill long-running sessions mid-work, so it goes to the operator.
72 lines
3.8 KiB
Markdown
72 lines
3.8 KiB
Markdown
# `[2026-08-28]` nh3-dev's three OOM events attribute to Claude Code — and the "no evidence" was a permissions artifact
|
|
|
|
Three memory-exhaustion events in 14 days (08-14 00:15, 08-26 09:58, 08-28 05:36, interval
|
|
halving). forseti reported none could be attributed because *"kernel messages are not being
|
|
persisted to journald"* and asked for journald persistence + sysstat.
|
|
|
|
## ⚠ THE PREMISE WAS WRONG — journald was persistent the whole time
|
|
|
|
journalctl -b -1 -k privileged 15,068 entries (82-day boot)
|
|
journalctl -b -1 -k unprivileged 4 entries
|
|
OOM records, all retained boots 351
|
|
|
|
`journalctl` **silently shows only your own messages** when you are in neither `adm` nor
|
|
`systemd-journal`, and prints the reason as a scroll-past hint. Two of the three "no evidence"
|
|
findings were that one artifact:
|
|
|
|
"kernel messages not persisted" -> they are, and every OOM task table is there
|
|
"journal stops 05:36:08, no shutdown" -> that is the USER's last entry; the true
|
|
boot -1 end is 05:47:04, with OOM kills
|
|
recorded at 05:38, 05:40, 05:42
|
|
|
|
**Fix was `usermod -aG adm lkraven`, not a logging change.** Debian's journald ACL names `adm`
|
|
explicitly (`getfacl /var/log/journal/<machine-id>` → `group:adm:r-x`). Existing shells keep
|
|
their old group set — re-login, or `sg adm -c '...'`, which is also how to *verify* the grant
|
|
took rather than grepping `/etc/group`.
|
|
|
|
## ⚠ THE HOG IS CLAUDE CODE
|
|
|
|
`/home/lkraven/.local/share/claude/versions/2.1.220` is the versioned CC binary, so OOM victims
|
|
named `2.1.220` / `2.1.177` / `2.1.168` are CC sessions, as are those named `claude`.
|
|
|
|
largest anon-rss ever recorded on this box
|
|
18,434,696 kB 2.1.177 18.4 GB
|
|
15,788,764 kB 2.1.220 15.8 GB
|
|
15,154,008 kB 2.1.220 15.2 GB
|
|
14,994,376 kB 2.1.168 15.0 GB
|
|
-> every one of the top TWELVE is a CC session
|
|
|
|
29 of the OOM victims are CC. Everything else killed — althing-forseti (22), caddy (15),
|
|
ttyd (11), zellij (6), the althing daemons — is 30-55 MB **collateral**, the OOM killer
|
|
scraping for a few hundred MB. The althing v2 daemons are cleared by measurement.
|
|
|
|
**"claude is 408 MB each" is a YOUNG session.** Mature ones measure 5.4-18.4 GB. On 27 GB with
|
|
974 MB swap the ceiling is **three or four mature sessions**, not the ~66 a 408 MB figure
|
|
implies. Aug 28's task table: two CC at 5.4 GB + three zellij servers at 1.14 GB.
|
|
|
|
## Instrumentation added (`playbooks/nh3-dev-memory-forensics.yaml`, idempotent)
|
|
|
|
sysstat system-wide mem/CPU, 5-min cadence (not Debian's 10 -- a CC session can
|
|
add several GB inside one 10-minute bucket). sar -r
|
|
atop PER-PROCESS, 60s, 7-day retention. atop -r /var/log/atop/atop_YYYYMMDD
|
|
journald unchanged, already persistent, now READABLE
|
|
|
|
**atop is the one that matters and it was not requested.** The journal records the moment of
|
|
the kill and names the *victim*; sar says the box filled up; only atop says **which session was
|
|
growing and how fast** — the whole question when a dozen panes are open.
|
|
|
|
## Open — operator's call, deliberately not taken
|
|
|
|
**A cgroup memory cap on CC sessions is the real mitigation and it would kill long-running
|
|
sessions mid-work.** Surfaced, not decided. Instrumentation makes event four *diagnosable*, not
|
|
less likely.
|
|
|
|
## Lesson that generalises
|
|
|
|
A verify step I wrote failed while the setting was live: I grepped
|
|
`systemctl show sysstat-collect.timer` for my own input `*:00/05`, but systemd normalises it to
|
|
`*-*-* *:00/5:00`. **Assert the effective value, not the string you wrote** —
|
|
[[feedback_assert_effective_value_not_substring]], caught here in my own instrumentation.
|
|
|
|
Reply: althing msg `01M147EWEZDT8Y0XW5FTHHEAQC`, thread `01M1472ST5DSJNHR676X9F43AK`.
|