vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: run 3c had TWO launches — the third was an untimestamped report

Browse Source 
brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.

  #1  17:53:33 PDT  killed by the power loss at step 80/604
  #2  20:58:41 PDT  stopped deliberately at 21:07:40, healthy

The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.

Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.

Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.

Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
This commit is contained in:
Vuong Hoang
2026-08-27 10:59:29 -07:00
parent 98e7d4886a
commit 88d79375f7
3 changed files with 78 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md
+4 -1
View File
@@ -1,6 +1,9 @@
# `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back
Site-wide outage, ~90 minutes. **Operator-confirmed cause: a tripped power breaker**, not a
Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**,
boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now
pinned by run 3c's training log and the hourly cron that never ran — see
[[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a
fault and not the tunnel. The discriminator that established scope: `ana-srv1`
(38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the
NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned
persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md
+71
View File
@@ -0,0 +1,71 @@
# `[2026-08-27]` There were TWO run-3c launches, not three — and the phantom third was my reporting
brokkr-smithy-dev asked, rather than guessed, how many times 3c was launched. Their reading was
three. **It was two**, and the third was an artifact of a state report I wrote without a
timestamp.
#1 launched 2026-08-26 17:53:33 PDT killed by the power loss at step 80/604
#2 launched 2026-08-26 20:58:41 PDT STOPPED BY ME at 21:07:40, healthy, on operator instruction
**3c did not "die twice". It died once and was stopped once.**
## The evidence, and where it lives
| fact | source |
|---|---|
| #1 launched 17:53:33 | `run-03c.log` ZFS **birth** time (`stat -c %w`) |
| #1 reached step 80/604 | `80/604 [18:15<2:23:38, 16.45s/it]` — read at 20:14, one minute after boot |
| #2's config rewritten 20:58:36 | `run-03c.json` mtime, same second as harness `0a6bd2e` (save_steps configurable) |
| #2's output tree is FRESH | `run-03c/` **birth** 20:58:41 — a new tree, not a resume; `checkpoints/` empty |
| #2 trained 21:01:25 → 21:07:34 | `batch-order.jsonl` mtime → `run-03c.log` mtime |
| **no #3** | nothing under `/tank/erp-tune` has an mtime after 21:07:34. The launcher redirects with `>`, so a relaunch truncates `run-03c.log` and rewrites `order-manifest.jsonl` + `batch-order.jsonl`. All three still carry launch-#2 stamps. |
`stat`'s **birth** time is what made this tractable — mtime alone cannot distinguish "truncated
and rewritten" from "created now", and `run-03c.log` was both (born 17:53:33, last written
21:07:34).
## The outage window, pinned to two minutes
train start #1 ≈17:56:30 (launch + 2m50s setup, measured on the relaunch)
step 80 at 18:15 elapsed ≈18:14:45
last journald entry, boot -1 17:54:38 (an ssh logout; the box was idle after, so it logged nothing)
the 18:17 hourly cron never ran
boot 0 20:13:14
**Power was lost between 18:14:45 and 18:17:00 PDT; ana-ml2 was down 1h58m** — not the "~90
minutes" recorded in [[2026-08-27-anaheim-breaker-and-onboot-gap]], which is corrected there.
⚠ The last journald entry of a hard-power-loss boot is **not** the moment of the loss — it is the
last time anything wanted to log. Here that was 20 minutes early. The training log's own progress
line was the tight bound; journald only bracketed it.
## Step 22 vs step 24
I reported "stopped at step 22". The log's last step is 24. My stop command read progress with
`tail -c 400 | tail -1` and the final flush at 21:07:34 carried 23 and 24 — written between my
read and the kill at 21:07:40. **24 is true; 22 was a ~20-second-stale tail.** Not evidence of a
different run.
## ⚠ THE DURABLE FINDING — an event report with no timestamp is a claim about "now"
At 23:03 I wrote *"Run 3c is STOPPED … GPU0 released: 300 W / 89.7 GB / 99% → 45 W / 2 MiB / 0%"*
in the present tense. That happened at 21:07:40. brokkr had a correctly-observed 22:46 snapshot
(no process, GPU idle, log last written 21:07) and my untimestamped report; **the only way to
reconcile them was to invent a launch between the two.**
Every fact in my message was accurate. It was unreadable *in sequence*. Same family as
[[feedback_confirm_reboot_by_observing_down]] and the two errors caught that night by checking
effect rather than exit code — the report looked current and nothing forced it to prove it.
**Put the wall clock on every state report.** Where a reading came from a command, quote the
command's own clock (`uptime`'s `21:07:27 up 54 min` sat in the same tool output as the 300 W
reading and would have settled this instantly).
## Second-order cost
brokkr's degraded-pool hypothesis was built on *"both 3c attempts have died on a DEGRADED pool"*
— two data points. There was one, and it has a sufficient cause that is not the pool. They
retracted the hypothesis as wrong about the cause; it was also **operating on an input I gave
them wrong**. A bad timestamp does not stay in your own artifact.
Reply: althing msg `01M125XZAB7DESMCD3751T7TTH`, thread `01M10CZSH1RPQSSAJNFZJDJMD5`.
persistent-memory.md
+3 -2
View File
@@ -108,9 +108,9 @@ no longer deployed sidecars here. See Recent decisions.)
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
## Current state / in-flight
_As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at step 22/604 after Anaheim tripped a power breaker.** Nothing is training._
_State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at last-logged step 24/604, killed deliberately at 2026-08-26 21:07:40 PDT after Anaheim tripped a power breaker.** Nothing is training._
- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker.
- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker. **Exactly TWO 3c launches, and only one of them died:** #1 17:53:33 → killed by the power loss at step 80/604; #2 20:58:41 → stopped BY ME at 21:07:40, healthy, on operator instruction. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- **🔴 `/tank` DEGRADED on ana-ml2 — a disk is genuinely gone**, 7 physical NVMe where the pool expects 8. raidz2, one parity disk spent, no data errors. **Operator is replacing it** (Supermicro AS-4125GS-TNRT2, PCIe hot-plug, should not need a power-down). → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
- **🟢 ADAPTERS BACKED UP OFF-SITE** — `nh3-nas:/volume1/smithy/erp-tune-adapter-backup/{run-01,run-02,run-03}`, sha256 verified at source, staging and rest (944 MB). They were single-copy mode-600 on the degraded pool. The four `merged-run03*` models are NOT backed up **by design** — all are derived from `run-03/adapter` by a documented verified merge, ~10 min each to regenerate from a 315 MB artifact that is now safe.
- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). `gen` stays up by instruction. Everything else on ana-ml2 is idle. Sheddable: `vllm-gen` 46 GB (the big one), embed 9.8 GB, coder 8.4 GB, rerank-a3 3.5 GB, reward 2.1 GB, scriberr.
@@ -125,6 +125,7 @@ _As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a
- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
- `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md`
- `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.**
- `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made.
- `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`.
- `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.