vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: run 3c had TWO launches — the third was an untimestamped report

Browse Source 
brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.

  #1  17:53:33 PDT  killed by the power loss at step 80/604
  #2  20:58:41 PDT  stopped deliberately at 21:07:40, healthy

The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.

Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.

Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.

Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
This commit is contained in:
vh
2026-08-27 10:59:29 -07:00
parent 98e7d4886a
commit 88d79375f7
3 changed files with 78 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md
+4 -1
View File
@@ -1,6 +1,9 @@
# `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back
Site-wide outage, ~90 minutes. **Operator-confirmed cause: a tripped power breaker**, not a
Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**,
boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now
pinned by run 3c's training log and the hourly cron that never ran — see
[[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a
fault and not the tunnel. The discriminator that established scope: `ana-srv1`
(38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the
NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned