memory: snapshot — 2026-10-08 patch night, nh3-svos a737a06 + post office live, albok 0.1.13/0.1.14, gateway glm-5.3, heid sweep incident; 16 entries archived
This commit is contained in:
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.
|
||||
|
||||
⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh).
|
||||
@@ -1,33 +0,0 @@
|
||||
# `[2026-09-23]` elway sudo uploads land root:root; fleet ownership audit; 33 files fixed
|
||||
|
||||
elway's sudo upload was scp-as-user then `sudo mv`, and mv keeps the owner, so
|
||||
every file it installed as root (systemd units, `/etc` configs, root-run
|
||||
scripts) landed owned by infra-ops or lkraven. A sudoers drop-in installed that
|
||||
way would break sudo outright. Commit `466f7aa`:
|
||||
|
||||
- sudo uploads chown to `root:root` by default; `upload.owner:` / `--owner`
|
||||
override; owner refused on non-sudo uploads.
|
||||
- chown+chmod act on the STAGED file, then one `mv` publishes it, so a failed
|
||||
chown cannot leave the live path mis-owned. A trap removes the staged file on
|
||||
every exit. A directory dest is refused before anything moves.
|
||||
- `mode` was spliced unquoted into the remote root shell; now octal-validated
|
||||
and quoted. `mode`/`owner` must be quoted YAML strings (bare `0644` → 420).
|
||||
- `preflight()` resolves every step before any remote action.
|
||||
- 24 unit tests; heid bug-hunt "Puck" (Gróa + seat) folded.
|
||||
|
||||
`scripts/fleet-ownership-audit.sh` (read-only; exit 0/4/5): tier A root-parsed
|
||||
paths (symlinks judged by target), tier X root-run unit Exec paths incl.
|
||||
drop-ins, tier B /opt summary. Unprivileged or partial hosts report INCOMPLETE,
|
||||
never clean; completion is nonce-marked. Positive controls on nh3-dev fired for
|
||||
each tier. **Six hosts have no infra-ops identity** (ana-wg, nh3-nas, pbs-ana,
|
||||
pbs-nh3, pfi-postgres, vm-esh-nas) and audit unprivileged.
|
||||
|
||||
**33 files chowned to root on 10 hosts** with Prime's approval (the first
|
||||
attempt was blocked by the permission classifier): 31 elway-placed (beszel
|
||||
agent trio on 6 hosts, esh-vm-db/ana-docker restic hooks and retry drop-ins,
|
||||
irv-ml1 units, nh3-dev alert bridge, fv-ml1 retired file) plus 2 non-elway
|
||||
root-exec holes (ana-docker `/usr/local/bin/resticprofile` owned by llmuser;
|
||||
esh-pve-nas `/usr/local/lib/libigdgmm.so.11.3.1343` uid 1000). Re-audit: 0 of
|
||||
the 33 remain; all affected services still active. Left as low-risk and
|
||||
reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1
|
||||
`btop` and a root uv-cache python.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move
|
||||
|
||||
**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`).
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-23]` headscale-ddns hardened
|
||||
|
||||
**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression).
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash
|
||||
|
||||
**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`.
|
||||
@@ -1,80 +0,0 @@
|
||||
# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1
|
||||
|
||||
Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done
|
||||
2204–2240 PT the same night, **with no esh-pve reboot**.
|
||||
|
||||
**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's
|
||||
`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied
|
||||
live by `playbooks/pve-nvidia-host.yaml`: nouveau was never loaded and the
|
||||
T400's vfio ids matched nothing, so nothing held the card. Our own
|
||||
`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}`
|
||||
**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired
|
||||
`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT
|
||||
rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM
|
||||
`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run`
|
||||
not an apt repo: one file on both sides makes the host-module/container-lib
|
||||
version match true by construction; apt would let an upgrade move one side.
|
||||
|
||||
**CT 110 `esh-ml1`** (`playbooks/gpu-lxc.yaml`): unprivileged Debian 12,
|
||||
nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool
|
||||
is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from
|
||||
the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit
|
||||
with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first
|
||||
try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per
|
||||
`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has
|
||||
no key/password yet). **Not in the vzdump job, on purpose** (rebuildable).
|
||||
DNS `esh-ml1.esh.internal` synced to all three resolvers.
|
||||
|
||||
**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013,
|
||||
same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each →
|
||||
4,823 MiB used of 16,380.
|
||||
|
||||
**Parity (the load-bearing measurement; embeddings are model-specific).** 11
|
||||
texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772,
|
||||
inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH
|
||||
0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity
|
||||
floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181,
|
||||
identical ranking.
|
||||
|
||||
**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment
|
||||
(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback
|
||||
proven with throwaway groups (created + deleted): primary refusing → ESH in
|
||||
+0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call,
|
||||
no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart
|
||||
production traffic is served by fv-ml1 (header `x-litellm-model-api-base`).
|
||||
`/health?model=` returned 503 with empty lists for both groups — not
|
||||
investigated. Gateway restart: liveliness back in ~52 s.
|
||||
|
||||
**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's
|
||||
pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no
|
||||
callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of
|
||||
`/model/info` found no other `10.250.50.54` targets.
|
||||
|
||||
**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):**
|
||||
`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore
|
||||
20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days**
|
||||
(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is
|
||||
an FV-outage failover for Worldtree + nevermore, not local service for ESH.
|
||||
|
||||
**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only
|
||||
ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
|
||||
Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
|
||||
untuned (router retries/cooldown).
|
||||
|
||||
**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).**
|
||||
Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is
|
||||
faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs
|
||||
~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8).
|
||||
Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would
|
||||
roughly double rerank latency for half the calls.
|
||||
|
||||
**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to
|
||||
evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference
|
||||
1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no
|
||||
Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside
|
||||
noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise
|
||||
→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is
|
||||
**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is
|
||||
**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30,
|
||||
~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the
|
||||
`huggingface/` provider. Both engines left running side by side pending Prime's call.
|
||||
@@ -1,40 +0,0 @@
|
||||
# `[2026-09-24]` esh-pve: VM 102 retired, a 14-day hung VFIO process found, T400 → RTX 2000 Ada
|
||||
|
||||
**VM 102 `esh-vm-workstation` retired permanently** (Prime: "not the machine I
|
||||
want running a virtual windows machine"). Final backup
|
||||
`pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` (verification ok) set
|
||||
**protected** so prune cannot take it; `qm destroy 102 --purge`.
|
||||
|
||||
⚠⚠ **Found: `kvm -id 102` had been hung in D state since the 2026-09-10 0332
|
||||
vzdump.** vzdump starts a stopped VM (here with the T400 passed through,
|
||||
`x-vga=1`) to read its disks; that process never exited. It held **16 GB of
|
||||
locked RAM**, the GPU, and two LVs, while `qm status 102` said `stopped` the
|
||||
whole time. Same class as the 2026-08-19 hard freeze: VFIO + this box. Only a
|
||||
reboot cleared it.
|
||||
|
||||
**GPU swap, Prime on site, 2119–2137.** Guests shut down cleanly; host powered
|
||||
off; the T400 was removed and an **NVIDIA RTX 2000 / 2000E Ada** installed
|
||||
(`01:00.0`, `10de:28b0`, subsystem `10de:1871`). Host back 2136:49; leftover
|
||||
LVs removed; all guests auto-started; esh-docker-vm NFS, traefik (0×403) and
|
||||
AdGuard verified. The new card has **no driver bound**; `/etc/modprobe.d` still
|
||||
lists the T400's vfio ids (`10de:1ff2,10de:10fa`), now matching nothing.
|
||||
Commits `719e3fa`, `7cbd4c3`.
|
||||
|
||||
**Decision (Prime, 2026-09-24): the card serves embedding + reranking as an
|
||||
LXC with the NVIDIA driver on the host — NOT a VFIO VM.** Rationale: both
|
||||
esh-pve hangs this year came from VFIO passthrough; LXC avoids it and pins no
|
||||
VM RAM. Accepted cost: a driver on the hypervisor, rebuilt on PVE kernel
|
||||
updates, host/container versions must match. **Implementation deferred to the
|
||||
next session** (tracking: this entry + `servers/esh-pve/README.md`).
|
||||
|
||||
**ESH single-route observation:** esh-scale (CT 108, ESH's only mesh subnet
|
||||
router) lives on esh-pve, so while esh-pve was down *all* of ESH looked dark
|
||||
from outside, including esh-pve-nas and vm-esh-nas, which were up. Untracked by
|
||||
operator choice so far — a second ESH route is an idea, not a decision.
|
||||
|
||||
**vPro:** the copper cable is in the AMT-capable port (**I226-LM**, `enp89s0`,
|
||||
MAC `58:47:ca:76:99:32`), verified at 2500 Mb/s; the I226-V (`enp88s0`) is
|
||||
not AMT-capable. Both links set back admin-down on the host. **MEBx
|
||||
provisioning deferred by Prime** ("some other time"): Ctrl+P at boot, enable
|
||||
manageability, static IP, KVM on with User Opt-in None, activate network
|
||||
access; MeshCentral should live on ana-docker/nh3-docker, never on esh-pve.
|
||||
@@ -1,32 +0,0 @@
|
||||
# `[2026-09-24]` pfi-gx10 "power on after AC loss" — UEFI patch applied, NOT validated
|
||||
|
||||
pfi-gx10 (ASUS Ascent GX10, BIOS `GX10DGX.0105.2026.0505.1153`, no BMC) stayed
|
||||
off after the NH3 outage. With Prime's approval (after the permission
|
||||
classifier first refused it as externally-sourced firmware code), bytes 3–4
|
||||
of the 81-byte UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9` were
|
||||
changed `0x1d,0x00 → 0x1c,0x01` via `efivar -w`; readback shows exactly those
|
||||
two bytes changed. Recipe: community tool `Fheodoor01/DGX-Spark---Power-on-boot-`
|
||||
(diff-derived from a power-on unit vs a stay-off unit; author-verified on
|
||||
GX10DGX.0103/.0104). **Byte 3's meaning is unknown**; the resulting pair
|
||||
matches factory power-on units.
|
||||
|
||||
**Original saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin`
|
||||
on the box and `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin`
|
||||
(sha256 `6d6f6f77…`, raw incl. the 4-byte attr header). Revert procedure in
|
||||
`servers/pfi-gx10/README.md`. Commit `e5197a3`.
|
||||
|
||||
⚠⚠ **An OS shutdown is NOT a test of this setting.** It means "start when
|
||||
power RETURNS after a cut"; a `poweroff` is supposed to stay off by design.
|
||||
The tool's README claims the two are indistinguishable — false here. I
|
||||
trusted it and ran two poweroff "tests" (1421, and 1435 after the reboot the
|
||||
README says the firmware needs); both stayed off, each costing Prime a trip to
|
||||
the button, and both prove nothing. Prime called it out directly.
|
||||
|
||||
**Status:** the box is OFF (the 1435 shutdown) with the patch in place, and
|
||||
Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone
|
||||
→ works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the
|
||||
button, revert from the backup, and set it in firmware via a PiKVM (he intends
|
||||
to add one with ATX control, which also covers remote power-on regardless).
|
||||
|
||||
**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and
|
||||
the box came up by itself at 1108:54 PT, with no button press. The patch works.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter
|
||||
|
||||
**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`).
|
||||
@@ -1,31 +0,0 @@
|
||||
# `[2026-09-24]` NH3 power outage — recovered; three latent gaps closed, one needs a hand
|
||||
|
||||
Power at NH3 dropped and returned; every NH3 box booted at 1233–1234 PT. The
|
||||
core recovered unaided (post office, headscale, egress proxy, DNS, WAN IP
|
||||
unchanged at `70.230.226.88`). Three things did not, each a latent config gap
|
||||
that only a site-wide power event exposes:
|
||||
|
||||
1. **pbs-nh3 (VM 105) had no `onboot` flag** and stayed down. Set `onboot=1`
|
||||
and started it; datastore NFS mounted (its line uses `bg`). `qm guest exec
|
||||
105` works (guest agent on) — the root path, since infra-ops is not
|
||||
provisioned there.
|
||||
2. **NFS boot race.** nh3-nas is the slowest box to serve NFS, so every plain
|
||||
fstab NFS line failed at boot (nh3-docker `/mnt/compose` `/mnt/backup`,
|
||||
nh3-dev `/mnt/backup`). nh3-dev's `/mnt/smithy` already had
|
||||
`x-systemd.automount` and self-healed on its next access (1234 failed, 1309
|
||||
mounted, no hand) — the existence proof. `playbooks/nh3-nfs-automount.yaml`
|
||||
brought the rest to that shape (`_netdev,nofail,x-systemd.automount,
|
||||
x-systemd.mount-timeout=30`), commit `1cbde50`. ⚠ nh3-dev's installer
|
||||
cdrom fstab line is a pre-existing `findmnt --verify` error; the playbook
|
||||
judges only errors a rewrite ADDS, after its first run aborted on it.
|
||||
3. **pfi-gx10 (bare metal, no BMC) did not power on** — ASUS ships "Restore AC
|
||||
Power Loss" = stay off. See `[[2026-09-24-gx10-ac-restore-patch]]`.
|
||||
|
||||
Also: **every Claude session on nh3-dev died** with the reboot except
|
||||
infra-ops, infra-hermes, jekyll. Relaunch is Prime's call (dev-launch).
|
||||
`nh3-laser` (VM 104) is on-demand and **stays off** by Prime's ruling;
|
||||
`servers/nh3-pve/README.md` now carries an expected-after-power-loss column per
|
||||
guest so the next triage does not flag it.
|
||||
|
||||
The headscale-ddns failures 1141–1202 were the WAN being down, correctly
|
||||
refused (no public v4); every run since 1247 reads `unchanged`.
|
||||
@@ -0,0 +1 @@
|
||||
- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed on nh3-albok at 1135 at albok-dev's ask 8413.** Rollback is NOT free: the first-start scan writes `sensitivity` sidecars, which 0.1.12 cannot load, so rolling back needs the strip snippet first. The census found `leaks` 8: six hold the shared all-agents key, two are false positives on a doc placeholder, nothing to rotate. → `stacks/albok-service/README.md`
|
||||
@@ -0,0 +1 @@
|
||||
- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3: sidecar listing, byte-less ingest, a `.browse/` view; detector 2) deployed at 1804 at albok-dev's ask 8524.** `leaks` went from 8 to 6 when the placeholder false positives cleared. Reindexes ran one per wing afterwards to render the view. ⚠ **OWED: the `.browse/` exposure test.** Pick a share surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on the server, and check that a fresh open through the client's cached listing FAILS. Nothing is shared until it passes; report to albok-dev. → `stacks/albok-service/README.md`
|
||||
@@ -0,0 +1 @@
|
||||
- `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml`
|
||||
@@ -0,0 +1 @@
|
||||
- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md`
|
||||
@@ -0,0 +1,16 @@
|
||||
# 2026-10-08 — heid's runaway sweep killed (Prime: "kill sweep immediately")
|
||||
|
||||
**Symptom.** nh3-dev's root disk flapped on the Beszel 85% line (2300 and 2329 10-07). The uv-cache prune (−6.7 GiB) bought minutes. `du` could not see the growth.
|
||||
|
||||
**Cause (found by infra-hermes, confirmed by infra-ops).** `heidnotes/sweep.py` under `heid/dispatch-log/work/01M4CC74W1HK46D1ZP9J26YSZK` was left running by a consult agent of svos-dev's U11 bug hunt, which had been relayed and closed at 1748. It swept the dispatch work dir recursively, and its output `heidnotes/final_sweep.txt` sat INSIDE that dir. So it read its own output and wrote it back out, forever: the tail was lines quoting `final_sweep.txt:66200017 …`. The work dir was then cleaned while the file was still open, so ~76 GB lived in a deleted-but-open file that only `/proc/<pid>/fd` showed.
|
||||
|
||||
**Two holders, not one.** infra-hermes saw PID 2332755; a second copy, PID 2368442 (its zsh wrapper 2368332, launched from the `heid` Claude session, pid 35804), held the same inode on fd 3. Killing the first freed nothing. Only when the second died did root go 86% → 66% (52 → 123 GB free).
|
||||
|
||||
**Lessons.**
|
||||
- A deleted-but-open file IS readable through `/proc/<pid>/fd/<n>`. That is how the self-feeding loop was confirmed before the kill, turning "runaway, kill it" from a guess into a verdict.
|
||||
- Before claiming that space will free, enumerate EVERY holder of the inode (walk `/proc/*/fd`), not just the first PID found.
|
||||
- Kill by explicit PID. Matching `heidnotes/sweep.py` also caught the wrapper shell, which was harmless here, and `pgrep -f` self-matched the checking command.
|
||||
|
||||
**Fix at the source (heid `8bf2fe57`).** The sweep tool skips `heidnotes/` and any file over 20 MB. The consult-agent brief forbids leaving a process running after the report. heid is adding a check for lingering processes before each workdir clean. The same bug had hit a second agent at ~3.4 GB earlier that day, and that agent caught it itself.
|
||||
|
||||
Ops-log 2026-10-08T06:37:07Z `nh3-dev kill heid-sweep`.
|
||||
@@ -0,0 +1 @@
|
||||
- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md`
|
||||
Reference in New Issue
Block a user