memory: run 3c had TWO launches — the third was an untimestamped report

brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.

  #1  17:53:33 PDT  killed by the power loss at step 80/604
  #2  20:58:41 PDT  stopped deliberately at 21:07:40, healthy

The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.

Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.

Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.

Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
This commit is contained in:
vh
2026-08-27 10:59:29 -07:00
parent 98e7d4886a
commit 88d79375f7
3 changed files with 78 additions and 3 deletions
+3 -2
View File
@@ -108,9 +108,9 @@ no longer deployed sidecars here. See Recent decisions.)
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
## Current state / in-flight
_As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at step 22/604 after Anaheim tripped a power breaker.** Nothing is training._
_State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at last-logged step 24/604, killed deliberately at 2026-08-26 21:07:40 PDT after Anaheim tripped a power breaker.** Nothing is training._
- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker.
- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker. **Exactly TWO 3c launches, and only one of them died:** #1 17:53:33 → killed by the power loss at step 80/604; #2 20:58:41 → stopped BY ME at 21:07:40, healthy, on operator instruction. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- **🔴 `/tank` DEGRADED on ana-ml2 — a disk is genuinely gone**, 7 physical NVMe where the pool expects 8. raidz2, one parity disk spent, no data errors. **Operator is replacing it** (Supermicro AS-4125GS-TNRT2, PCIe hot-plug, should not need a power-down). → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
- **🟢 ADAPTERS BACKED UP OFF-SITE** — `nh3-nas:/volume1/smithy/erp-tune-adapter-backup/{run-01,run-02,run-03}`, sha256 verified at source, staging and rest (944 MB). They were single-copy mode-600 on the degraded pool. The four `merged-run03*` models are NOT backed up **by design** — all are derived from `run-03/adapter` by a documented verified merge, ~10 min each to regenerate from a 315 MB artifact that is now safe.
- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). `gen` stays up by instruction. Everything else on ana-ml2 is idle. Sheddable: `vllm-gen` 46 GB (the big one), embed 9.8 GB, coder 8.4 GB, rerank-a3 3.5 GB, reward 2.1 GB, scriberr.
@@ -125,6 +125,7 @@ _As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a
- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
- `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md`
- `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.**
- `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made.
- `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`.
- `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.