From 88d79375f729b4fcf0f527dedf5e045d1e7fc7e4 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 27 Aug 2026 10:59:29 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20run=203c=20had=20TWO=20launches=20?= =?UTF-8?q?=E2=80=94=20the=20third=20was=20an=20untimestamped=20report?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit brokkr-smithy-dev asked how many times 3c was launched rather than reconstructing it, and their reading was three. It was two. #1 17:53:33 PDT killed by the power loss at step 80/604 #2 20:58:41 PDT stopped deliberately at 21:07:40, healthy The phantom third came from a report I wrote at 23:03 narrating the 21:07 kill in the present tense with no timestamp. Every fact in it was accurate; it was unreadable in sequence against a correctly-observed 22:46 snapshot of an idle GPU. Evidence is ZFS birth times (a `>` redirect truncates the log but keeps its birth, so mtime alone cannot separate "rewritten" from "created"), plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a relaunch would have rewritten three files there. Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the downtime from "~90 minutes" to 1h58m: the last journald entry before a hard power loss is the last time anything wanted to log, not the moment of the loss, and here it was 20 minutes early. Corrects the in-flight header (step 22 -> last-logged step 24, stop deliberate) and its stale "as of" stamp. --- ...26-08-27-anaheim-breaker-and-onboot-gap.md | 5 +- ...08-27-run3c-launch-count-reconstruction.md | 71 +++++++++++++++++++ persistent-memory.md | 5 +- 3 files changed, 78 insertions(+), 3 deletions(-) create mode 100644 persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md diff --git a/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md b/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md index a443188..63fc7cc 100644 --- a/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md +++ b/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md @@ -1,6 +1,9 @@ # `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back -Site-wide outage, ~90 minutes. **Operator-confirmed cause: a tripped power breaker**, not a +Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**, +boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now +pinned by run 3c's training log and the hourly cron that never ran — see +[[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a fault and not the tunnel. The discriminator that established scope: `ana-srv1` (38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned diff --git a/persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md b/persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md new file mode 100644 index 0000000..b739fc8 --- /dev/null +++ b/persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md @@ -0,0 +1,71 @@ +# `[2026-08-27]` There were TWO run-3c launches, not three — and the phantom third was my reporting + +brokkr-smithy-dev asked, rather than guessed, how many times 3c was launched. Their reading was +three. **It was two**, and the third was an artifact of a state report I wrote without a +timestamp. + + #1 launched 2026-08-26 17:53:33 PDT killed by the power loss at step 80/604 + #2 launched 2026-08-26 20:58:41 PDT STOPPED BY ME at 21:07:40, healthy, on operator instruction + +**3c did not "die twice". It died once and was stopped once.** + +## The evidence, and where it lives + +| fact | source | +|---|---| +| #1 launched 17:53:33 | `run-03c.log` ZFS **birth** time (`stat -c %w`) | +| #1 reached step 80/604 | `80/604 [18:15<2:23:38, 16.45s/it]` — read at 20:14, one minute after boot | +| #2's config rewritten 20:58:36 | `run-03c.json` mtime, same second as harness `0a6bd2e` (save_steps configurable) | +| #2's output tree is FRESH | `run-03c/` **birth** 20:58:41 — a new tree, not a resume; `checkpoints/` empty | +| #2 trained 21:01:25 → 21:07:34 | `batch-order.jsonl` mtime → `run-03c.log` mtime | +| **no #3** | nothing under `/tank/erp-tune` has an mtime after 21:07:34. The launcher redirects with `>`, so a relaunch truncates `run-03c.log` and rewrites `order-manifest.jsonl` + `batch-order.jsonl`. All three still carry launch-#2 stamps. | + +`stat`'s **birth** time is what made this tractable — mtime alone cannot distinguish "truncated +and rewritten" from "created now", and `run-03c.log` was both (born 17:53:33, last written +21:07:34). + +## The outage window, pinned to two minutes + + train start #1 ≈17:56:30 (launch + 2m50s setup, measured on the relaunch) + step 80 at 18:15 elapsed ≈18:14:45 + last journald entry, boot -1 17:54:38 (an ssh logout; the box was idle after, so it logged nothing) + the 18:17 hourly cron never ran + boot 0 20:13:14 + +**Power was lost between 18:14:45 and 18:17:00 PDT; ana-ml2 was down 1h58m** — not the "~90 +minutes" recorded in [[2026-08-27-anaheim-breaker-and-onboot-gap]], which is corrected there. + +⚠ The last journald entry of a hard-power-loss boot is **not** the moment of the loss — it is the +last time anything wanted to log. Here that was 20 minutes early. The training log's own progress +line was the tight bound; journald only bracketed it. + +## Step 22 vs step 24 + +I reported "stopped at step 22". The log's last step is 24. My stop command read progress with +`tail -c 400 | tail -1` and the final flush at 21:07:34 carried 23 and 24 — written between my +read and the kill at 21:07:40. **24 is true; 22 was a ~20-second-stale tail.** Not evidence of a +different run. + +## ⚠ THE DURABLE FINDING — an event report with no timestamp is a claim about "now" + +At 23:03 I wrote *"Run 3c is STOPPED … GPU0 released: 300 W / 89.7 GB / 99% → 45 W / 2 MiB / 0%"* +in the present tense. That happened at 21:07:40. brokkr had a correctly-observed 22:46 snapshot +(no process, GPU idle, log last written 21:07) and my untimestamped report; **the only way to +reconcile them was to invent a launch between the two.** + +Every fact in my message was accurate. It was unreadable *in sequence*. Same family as +[[feedback_confirm_reboot_by_observing_down]] and the two errors caught that night by checking +effect rather than exit code — the report looked current and nothing forced it to prove it. + +**Put the wall clock on every state report.** Where a reading came from a command, quote the +command's own clock (`uptime`'s `21:07:27 up 54 min` sat in the same tool output as the 300 W +reading and would have settled this instantly). + +## Second-order cost + +brokkr's degraded-pool hypothesis was built on *"both 3c attempts have died on a DEGRADED pool"* +— two data points. There was one, and it has a sufficient cause that is not the pool. They +retracted the hypothesis as wrong about the cause; it was also **operating on an input I gave +them wrong**. A bad timestamp does not stay in your own artifact. + +Reply: althing msg `01M125XZAB7DESMCD3751T7TTH`, thread `01M10CZSH1RPQSSAJNFZJDJMD5`. diff --git a/persistent-memory.md b/persistent-memory.md index 58f20a1..13028f2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -108,9 +108,9 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at step 22/604 after Anaheim tripped a power breaker.** Nothing is training._ +_State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at last-logged step 24/604, killed deliberately at 2026-08-26 21:07:40 PDT after Anaheim tripped a power breaker.** Nothing is training._ -- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker. +- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker. **Exactly TWO 3c launches, and only one of them died:** #1 17:53:33 → killed by the power loss at step 80/604; #2 20:58:41 → stopped BY ME at 21:07:40, healthy, on operator instruction. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` - **🔴 `/tank` DEGRADED on ana-ml2 — a disk is genuinely gone**, 7 physical NVMe where the pool expects 8. raidz2, one parity disk spent, no data errors. **Operator is replacing it** (Supermicro AS-4125GS-TNRT2, PCIe hot-plug, should not need a power-down). → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` - **🟢 ADAPTERS BACKED UP OFF-SITE** — `nh3-nas:/volume1/smithy/erp-tune-adapter-backup/{run-01,run-02,run-03}`, sha256 verified at source, staging and rest (944 MB). They were single-copy mode-600 on the degraded pool. The four `merged-run03*` models are NOT backed up **by design** — all are derived from `run-03/adapter` by a documented verified merge, ~10 min each to regenerate from a 315 MB artifact that is now safe. - **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). `gen` stays up by instruction. Everything else on ana-ml2 is idle. Sheddable: `vllm-gen` 46 GB (the big one), embed 9.8 GB, coder 8.4 GB, rerank-a3 3.5 GB, reward 2.1 GB, scriberr. @@ -125,6 +125,7 @@ _As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a - `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` - `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md` - `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.** +- `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` - `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made. - `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`. - `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.