# `[2026-08-28]` nh3-dev's three OOM events attribute to Claude Code — and the "no evidence" was a permissions artifact

Three memory-exhaustion events in 14 days (08-14 00:15, 08-26 09:58, 08-28 05:36, interval
halving). forseti reported none could be attributed because *"kernel messages are not being
persisted to journald"* and asked for journald persistence + sysstat.

## ⚠ THE PREMISE WAS WRONG — journald was persistent the whole time

    journalctl -b -1 -k   privileged     15,068 entries (82-day boot)
    journalctl -b -1 -k   unprivileged        4 entries
    OOM records, all retained boots         351

`journalctl` **silently shows only your own messages** when you are in neither `adm` nor
`systemd-journal`, and prints the reason as a scroll-past hint. Two of the three "no evidence"
findings were that one artifact:

    "kernel messages not persisted"       -> they are, and every OOM task table is there
    "journal stops 05:36:08, no shutdown" -> that is the USER's last entry; the true
                                             boot -1 end is 05:47:04, with OOM kills
                                             recorded at 05:38, 05:40, 05:42

**Fix was `usermod -aG adm lkraven`, not a logging change.** Debian's journald ACL names `adm`
explicitly (`getfacl /var/log/journal/<machine-id>` → `group:adm:r-x`). Existing shells keep
their old group set — re-login, or `sg adm -c '...'`, which is also how to *verify* the grant
took rather than grepping `/etc/group`.

## ⚠ THE HOG IS CLAUDE CODE

`/home/lkraven/.local/share/claude/versions/2.1.220` is the versioned CC binary, so OOM victims
named `2.1.220` / `2.1.177` / `2.1.168` are CC sessions, as are those named `claude`.

    largest anon-rss ever recorded on this box
      18,434,696 kB  2.1.177   18.4 GB
      15,788,764 kB  2.1.220   15.8 GB
      15,154,008 kB  2.1.220   15.2 GB
      14,994,376 kB  2.1.168   15.0 GB
    -> every one of the top TWELVE is a CC session

29 of the OOM victims are CC. Everything else killed — althing-forseti (22), caddy (15),
ttyd (11), zellij (6), the althing daemons — is 30-55 MB **collateral**, the OOM killer
scraping for a few hundred MB. The althing v2 daemons are cleared by measurement.

**"claude is 408 MB each" is a YOUNG session.** Mature ones measure 5.4-18.4 GB. On 27 GB with
974 MB swap the ceiling is **three or four mature sessions**, not the ~66 a 408 MB figure
implies. Aug 28's task table: two CC at 5.4 GB + three zellij servers at 1.14 GB.

## Instrumentation added (`playbooks/nh3-dev-memory-forensics.yaml`, idempotent)

    sysstat   system-wide mem/CPU, 5-min cadence (not Debian's 10 -- a CC session can
              add several GB inside one 10-minute bucket).  sar -r
    atop      PER-PROCESS, 60s, 7-day retention.  atop -r /var/log/atop/atop_YYYYMMDD
    journald  unchanged, already persistent, now READABLE

**atop is the one that matters and it was not requested.** The journal records the moment of
the kill and names the *victim*; sar says the box filled up; only atop says **which session was
growing and how fast** — the whole question when a dozen panes are open.

## Open — operator's call, deliberately not taken

**A cgroup memory cap on CC sessions is the real mitigation and it would kill long-running
sessions mid-work.** Surfaced, not decided. Instrumentation makes event four *diagnosable*, not
less likely.

## Lesson that generalises

A verify step I wrote failed while the setting was live: I grepped
`systemctl show sysstat-collect.timer` for my own input `*:00/05`, but systemd normalises it to
`*-*-* *:00/5:00`. **Assert the effective value, not the string you wrote** —
[[feedback_assert_effective_value_not_substring]], caught here in my own instrumentation.

Reply: althing msg `01M147EWEZDT8Y0XW5FTHHEAQC`, thread `01M1472ST5DSJNHR676X9F43AK`.