17 lines
2.0 KiB
Markdown
17 lines
2.0 KiB
Markdown
# 2026-10-08 — heid's runaway sweep killed (Prime: "kill sweep immediately")
|
||
|
||
**Symptom.** nh3-dev's root disk flapped on the Beszel 85% line (2300 and 2329 10-07). The uv-cache prune (−6.7 GiB) bought minutes. `du` could not see the growth.
|
||
|
||
**Cause (found by infra-hermes, confirmed by infra-ops).** `heidnotes/sweep.py` under `heid/dispatch-log/work/01M4CC74W1HK46D1ZP9J26YSZK` was left running by a consult agent of svos-dev's U11 bug hunt, which had been relayed and closed at 1748. It swept the dispatch work dir recursively, and its output `heidnotes/final_sweep.txt` sat INSIDE that dir. So it read its own output and wrote it back out, forever: the tail was lines quoting `final_sweep.txt:66200017 …`. The work dir was then cleaned while the file was still open, so ~76 GB lived in a deleted-but-open file that only `/proc/<pid>/fd` showed.
|
||
|
||
**Two holders, not one.** infra-hermes saw PID 2332755; a second copy, PID 2368442 (its zsh wrapper 2368332, launched from the `heid` Claude session, pid 35804), held the same inode on fd 3. Killing the first freed nothing. Only when the second died did root go 86% → 66% (52 → 123 GB free).
|
||
|
||
**Lessons.**
|
||
- A deleted-but-open file IS readable through `/proc/<pid>/fd/<n>`. That is how the self-feeding loop was confirmed before the kill, turning "runaway, kill it" from a guess into a verdict.
|
||
- Before claiming that space will free, enumerate EVERY holder of the inode (walk `/proc/*/fd`), not just the first PID found.
|
||
- Kill by explicit PID. Matching `heidnotes/sweep.py` also caught the wrapper shell, which was harmless here, and `pgrep -f` self-matched the checking command.
|
||
|
||
**Fix at the source (heid `8bf2fe57`).** The sweep tool skips `heidnotes/` and any file over 20 MB. The consult-agent brief forbids leaving a process running after the report. heid is adding a check for lingering processes before each workdir clean. The same bug had hit a second agent at ~3.4 GB earlier that day, and that agent caught it itself.
|
||
|
||
Ops-log 2026-10-08T06:37:07Z `nh3-dev kill heid-sweep`.
|