From b8dbe71a1ca26f32992ce01c868877dbad9f52ad Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 10 Sep 2026 10:25:52 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20R49=201-epoch?= =?UTF-8?q?=20pilot=20complete=20and=20awaiting=20adjudication;=20MeroMero?= =?UTF-8?q?=20A4B=20quantized,=20v2=20blocked;=20althing=203.6.2=20rolled?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ships the two pending code changes alongside the memory that describes them: train_voice_lora.py gains --eval-steps/--save-steps (the 3-epoch pilot overfit with per-epoch eval and save_strategy="no", so the minimum was neither visible nor recoverable), and the althing post-office compose is pinned to 3.6.2. Index rewritten: in-flight compressed from 176 lines to the live state, three detail files added for R49 D1-D3, the MeroMero four-failure chain, and the althing rollout. Seven closed pre-08-27 entries archived; the guards held back the rest because they carry open deferred pointers. --- archival-memory.md | 24 ++ .../2026-09-10-althing-362-rollout.md | 60 +++ ...ero-acquisition-and-four-quant-failures.md | 72 ++++ ...-babybronte-d1-d3-and-the-1-epoch-pilot.md | 119 ++++++ persistent-memory.md | 347 ++++++++---------- scripts/r49-corpus/train_voice_lora.py | 16 +- stacks/althing-post-office/compose.yaml | 2 +- 7 files changed, 435 insertions(+), 205 deletions(-) create mode 100644 persistent-memory.d/2026-09-10-althing-362-rollout.md create mode 100644 persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md create mode 100644 persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md diff --git a/archival-memory.md b/archival-memory.md index 846ef98..08f6395 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -3,6 +3,30 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._ ## Recent decisions (archived) + +- `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md` + _Archived 2026-09-10._ + +- `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file. + _Archived 2026-09-10._ + + +- `[2026-08-26]` **Run 2's gate FAILED and is recorded as a FAIL** — T3 constraint −12.0 against a ~1 pt floor. But gate 1 is the result: **T6 spatial +15.0 where run 1 failed the same axis at −3.5**, base swap the only intended variable. Neither run ships; the pair prices what the abliteration cost. Plus the long-form-only stability regression a short-answer gate would have passed, and PIPPA's 123-word clip in the length signal. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` + _Archived 2026-09-10._ + + +- `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` + _Archived 2026-09-10._ + +- `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name). + _Archived 2026-09-10._ + +- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before. + _Archived 2026-09-10._ + +- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path. + _Archived 2026-09-10._ + - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase. _Archived 2026-09-02._ diff --git a/persistent-memory.d/2026-09-10-althing-362-rollout.md b/persistent-memory.d/2026-09-10-althing-362-rollout.md new file mode 100644 index 0000000..52ef6d6 --- /dev/null +++ b/persistent-memory.d/2026-09-10-althing-362-rollout.md @@ -0,0 +1,60 @@ +# `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, and it was TWO nodes not seven + +forseti shipped 3.6.2 to fix a Claude Code dedupe: the harness drops an inbound peer frame +byte-identical to the previous one from the same sender within 30 s, and the herald's poke text was a +single constant — so two pokes to a seat inside 30 s lost the second, and that seat waited for rung 1 +five minutes later. 3.6.2 stamps `From , poked at HH:MM:SS.` onto every poke so two can never +share a body. + +## What I rolled + + post office nh3-docker 3.6.0 -> 3.6.2 built + pushed to gitea, digest-pinned + nh3-dev herald 3.6.1 -> 3.6.2 + nh3-extdev herald 3.1.1 -> 3.6.2 + +Every version read back off the RUNNING install, not the tag. Both nodes heartbeating after. + +## ⚠ It was two boxes, not the seven the instruction assumed + +The rollout said "the seven heralds." The post office's own `nodes` table has exactly **two** rows — +`nh3-dev` and `nh3-extdev`. The other five have no install, no unit, no heartbeat. Ask the post +office, not the fleet inventory. + +## Three traps, all mine to avoid next time + +**My survey said "not-installed" everywhere** because I ran it as `infra-ops` and the heralds run as +**`lkraven`**. Same shape as the ana-docker sudo trap but on the USER axis. Had I trusted it I would +have concluded the fleet had no heralds. + +**nh3-extdev is worse than "needs the right prefix."** It is a SYSTEM unit (not `--user`), running as +`lkraven`, off `/opt/uv-tools/althing-core` — a venv with **no pip and no uv**, so `uv tool install` +there is not merely ineffective, it is impossible. Path: build a wheel on nh3-dev, `ensurepip` into +the venv, force-reinstall. + +**I took the bus down ~12 minutes, and it was self-inflicted.** The documented backup is +stop → checkpoint → verify counts → copy → verify counts. My `PRAGMA wal_checkpoint(TRUNCATE)` +DELETED the `-wal`/`-shm` files, and the volume directory is root-owned 755 while the app runs as uid +1000 — so WAL mode could not recreate them and the container crash-looped on "attempt to write a +readonly database". Fix: `chown 1000:1000` the volume directory. ⚠ The runbook says checkpoint; it +does not warn that on this volume that is a one-way trip until ownership is fixed. + +**Then I misread the recovery.** `docker logs --tail 25` without `--timestamps` showed the +accumulated crash history from BEFORE the fix and I called it a current failure — nearly rolling back +a working deploy. Rollback would not have helped anyway: the fault was in the volume, not the image. + +## The backup lesson that paid for itself + +My first copy took only `post_office.db` and its **sha256 MATCHED the source** — while 7.2 MB of WAL +sat uncopied. Restoring it would have silently lost recent mail. The runbook insists on ROW COUNTS +rather than hashes for exactly this, and it was right: the hash agreed while the backup was +incomplete. Post-deploy counts identical on all six tables (handles 76, messages 1722, recipients +1749, nodes 2, sessions 24, notifications 23). + +## The drop-count instrument, corrected + +I reported the defect "reproduces on nh3-dev" with 3 drops in 21 s. **It does not.** Grepping +transcripts for the notice WRITES the notice into the transcripts — my grep output and my own quoting +of it. Genuine notices carry `"type":"system"`. True count 26 in 15 sessions; my naive count was 109; +the session I claimed 3 drops in had **zero**. forseti's own first count had the same contamination. +BEFORE baselines captured: nh3-dev **27 in 18**, nh3-extdev **0**. +Auto-memory: [[feedback_grep_over_a_log_that_records_your_greps]] diff --git a/persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md b/persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md new file mode 100644 index 0000000..5494725 --- /dev/null +++ b/persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md @@ -0,0 +1,72 @@ +# `[2026-09-10]` MeroMero acquisition — the A4B quant landed, the v2 dense has failed FOUR times + +Operator wanted a MeroMero seat. Getting there cost four quant attempts and corrected two of my own +wrong hypotheses, so the failure chain is the durable part. + +## The family, because I got it wrong first + +I called the unsuffixed 31B "v1" because v2 is `v2-31B`. The operator caught it. Creation dates from +the upstream author settle it: + + 2026-04-15 zerofata/G4-MeroMero-26B-A4B 30 layers, kv 8 MoE 128 experts ← the ORIGINAL + 2026-04-30 zerofata/G4-MeroMero-31B 60 layers, kv 16 DENSE ← a separate line + 2026-07-31 zerofata/G4-MeroMero-v2-31B 60 layers, kv 16 DENSE + +Not a v1→v2 sequence at one size: an A4B MoE first, then a dense 31B line that later got a v2. +⚠ The `-v1-` repo 401s, and I read that as "v1 must be the unsuffixed one" instead of checking dates. + +**The geometry is why it matters.** The A4B is 30 layers / kv 8 — *identical to Pfish-6* (a gemma4 +26B-A4B) — so it drops into the existing KV budget at 262k. The dense 31B is 60 layers / kv 16, ~4x +the KV per token, and did NOT fit 262k on GPU1 alongside the other seats. + +## Published quants: none of them are our scheme + + pekkAi/G4-MeroMero-31B-NVFP4 true v1 dense, W4A4 + llmfan46/G4-MeroMero-31B-uncensored-heretic-NVFP4 abliterated dense, W4A16_NVFP4 (pulled, 19 GB) + pekkAi/G4-MeroMero-26B-A4B-it-...-heretic-NVFP4 abliterated A4B, W4A4 + FP8 KV baked in + NVFP4 of v2-31B DOES NOT EXIST, from anyone (0 of 27 v2 repos) + +W4A4 carries our own measured long-context collapse (Granite, 30k ctx), which is why every seat we +run is A16. So the operator ruled: pull both ablits bf16 and quantize in-house at W4A16. + +## MTP — checked on the SOURCES, not inferred + +Gemma-4 ships **no MTP head at all**: 0 mtp tensors and no mtp config keys in the A4B bf16, the v2 +bf16, Pfish-6, or the published quants. The "mtp if applicable" clause is a no-op here; the MTP work +in our history is Qwen-family. + +## A4B — SUCCEEDED + +`G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`, 16 G, 7 files, **2m08s**. That time confirms +playbook §3.16: weight-only A16 runs a `DataFreePipeline` and never touches the dataset. +⚠ Output is missing `preprocessor_config.json` — §4.3 post-step 2 still owed before serving. + +## v2 dense — FOUR failures, each a different layer + +1. **`num_key_value_heads` is None** at `Gemma4TextAttention.__init__`. I hypothesised a transformers + version deadlock (config authored by 5.16.1, container on 5.12.1) — **wrong**: all three configs + construct fine on 5.12.1. Then I hypothesised the stale `torch_dtype` kwarg reshaping the config — + **also wrong**: every kwarg combination returns 16. + ⚠ **A 2-layer truncation PASSED and hid it.** The failing local is chosen per layer type; only + `full_attention` layers take the branch. Testing each layer type individually found it in seconds. + Real cause: DogOnKeyboard's config sets `attention_k_eq_v: true` but **omits + `num_global_key_value_heads` and `global_head_dim`** — a malformed upload, not a toolchain issue. + Patched from zerofata's canonical values (4 / 512) after **shape-verifying** the checkpoint: + full-attn k_proj `[2048,5376]` = 4×512, sliding `[4096,5376]` = 16×256, identical to canonical. + Original at `config.json.bak-pre-kvfields-20260910`. +2. **`initialize model processor ... required when a dataset is provided`** — DogOnKeyboard ships no + `processor_config.json` (the A4B does; that is why it succeeded). Copied from canonical, AND + dropped the dataset entirely, since §3.16 says A16 ignores it and §3.14 says building it bakes a + truncation cap into the shipped tokenizer. Removing it kills both for zero loss. +3. **My own bug**: the reference module runs argparse with `required=True` at IMPORT, so blanking + `sys.argv` still exited 2. Placeholder args, real argv restored after. +4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — OPEN. The + DogOnKeyboard config carries a `per_layer_config` key that zerofata's lacks (it was the one key in + the diff I noted and did not chase). transformers refuses global access to `head_dim` on a + heterogeneous config; the fix is likely `allow_global_per_layer_attribute_access=True`, with the + warning that a caller assuming homogeneity may then read the wrong value. + +⚠ **My wrapper reported `rc=0` on a failed run** because it read `$?` after an `echo`. A wrapper that +reports success on failure is the false-reassurance class; fixed to capture `$?` immediately. + +Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]] diff --git a/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md b/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md new file mode 100644 index 0000000..bda3529 --- /dev/null +++ b/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md @@ -0,0 +1,119 @@ +# `[2026-09-10]` R49 / BabyBronte — D1–D3 built, carrier settled on dense Qwen3, and the 3-epoch pilot overfit + +Operator opened the day with *"keep the adapter, then look into babybronte and start prepping +for a super lightweight small author voice training model regime."* By close the pipeline runs +end to end and one adapter exists. + +## Carrier — SETTLED, and it inverted R49's own pin + +Operator: *"use dense qwen3."* Sweep is **`Qwen3-{0.6,1.7,4}B-Base`**, which **overrides R49 H02's +stated arms** (the `Qwen3.5` trio). brokkr-smithy amended H02 accordingly. + +The ruling came off a measurement, on pfi-gx10 (GB10), n=10/arm, seq 4096, r=32 on q,k,v,o+MLP, +bf16 sdpa, grad-ckpt on, spreads 0.3–2.6%: + + Qwen3-0.6B-Base dense 0.616 B 2,399 tok/s + Qwen3-1.7B-Base dense 1.755 B 1,415 tok/s + Qwen3-4B-Base dense 4.089 B 717 tok/s + Qwen3.5-0.8B-Base hybrid 18 SSM/6 attn 0.765 B 540 tok/s + +**The dense 4.089 B trains 33% FASTER than the hybrid 0.765 B**, on 5.3x the parameters. Cause: +Qwen3.5 is 18 linear-attention (Mamba) layers to 6 attention, and no fused kernel is installed +(`mamba_ssm`/`causal_conv1d`/`fla`/`kernels` all absent; triton 3.8 present). Grad-ckpt is not the +cause (19%, saves 2.6x memory). Batching is not the lever — both families sit at the box roofline +at batch 1. Projected per voice: hybrid 0.8B **12 h**, which is LONGER than the 7 h 26B-A4B tune the +regime exists to replace. The premise dies on that carrier. + +Registry check closed the "is there something newer" question: **no official Qwen3.6/3.8 below 27B, +and neither family publishes a `-Base` checkpoint at any size.** Since the regime requires a +non-instruct carrier, dense Qwen3 was not a compromise — it was the only option. + +⚠ Two Qwen3.5 landmines, measured not inferred: the *-Base releases **ship a vision tower** (153 +`model.visual.*` Linear tensors at 0.8B) plus an MTP head, which `target_modules="all-linear"` would +train on text — `AutoModelForCausalLM` drops both, but that RENAMES modules relative to vLLM's +serving class, so adapter binding needs a sampled-target-changed check in the serving path. And +**cross-document packing is unsafe** because SSM state ignores the attention mask. + +## D1 — corpus + +`gx10:~/r49-corpus/`, instruments `scripts/r49-corpus/{build_corpus,verify_corpus}.py`. Charlotte +only (Jane Eyre 1260, Villette 9182, Shirley 30486, The Professor 1028; ids verified against +gutenberg.org's own search). **680,291 words · 142 chapters · 950,974 Qwen3 tokens.** 11 acceptance +checks pass including lossless round-trip and 0 byte-fallback. + +⚠⚠ **The alphabet INVERTS the Yarros result.** Brontë writes French constantly (Villette, Adèle, +Brussels): 75 letters, **23 non-ASCII** — `ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ`. F02 measured Yarros at 0.0002% +non-ASCII and derived an ASCII-fold. Under F02's own subset rule the Brontë pool **KEEPS French +accents** and **still excludes** Czech/Latvian/Slovak/Hungarian marks. First corpus where deriving +the alphabet per work changed the answer rather than confirming it. + +⚠ Typography was inconsistent and it was the TRANSCRIBER, not the author: Shirley = straight quotes ++ `--` + zero em-dashes; Jane Eyre/Villette = curly + em-dash; The Professor = curly + `--`. +Normalised toward meaning. + +## D2/D3 — entity resolution and rename + +**Gender resolution is now TITLE-FIRST, a change from F02's method rather than a port of it.** +Pronoun proximity called **Jane male** — she narrates Jane Eyre, so her name appears mostly in +Rochester's dialogue surrounded by his pronouns. Titles have no such blind spot (`Miss Eyre`, +`Mrs. Fairfax`, `M. Paul`): **16 entities resolved, zero wrong**, every ambiguous case HELD. +**Held means ungendered, not unrenamed** — a held entity is still renamed from the neutral pool, +because holding a place leaks it (`Thornfield` × 100 = the `Riders Quadrant` case). + +Pool: French + English per operator, locales `fr_FR, fr_BE, en_GB, en_IE` only (en_US/en_AU carry +modern surnames wrong for the 1840s), weighted per work by setting — Brussels novels 60% French, +Yorkshire 25%, alphabet-filtered. + +⚠ **F02's collision filter does not carry across corpora, and whole-string filtering is not enough.** +The pool drew `Burns` and `Marie` (both Brontë characters), then after fixing that drew +**`Pierre-Yves`** while `Pierre` (Mademoiselle St. Pierre) is a Villette character. **Filter by +COMPONENT.** Final gate: 0 of 203 source entities survive in any of 24 copy-files. + +## The pilot — and why 3 epochs was wrong + +H02 is **pure continuation by design** (no beat annotation), which is why the pilot was launchable +without D4. + +3-epoch run: train loss 3.310 → 2.574 while **held-out loss ROSE every epoch: 3.198 → 3.318 → +3.385.** Textbook overfit, and `save_strategy="no"` left nothing to fall back to. Operator: *"kill +and retrain 1 epoch."* + +1-epoch rerun, same seed 4919 and byte-identical corpus (sha `3959036cf851bf62`), eval+save every +25 of 169 steps so the minimum is LOCATED not assumed: + + step 25 3.2080 step 100 3.1764 step 169 3.1719 ← best, still descending + step 50 3.1891 step 125 3.1726 + step 75 3.1786 step 150 3.1720 + +Monotonic descent, no turn. Better on held-out by 0.21 nats while train loss is HIGHER (3.147 vs +2.834) — the signature of the earlier run memorising. ⚠ Not strict subsets: cosine LR decays over +169 steps here vs 507 there. + +Adapter verified bound: **196/196 lora_B tensors non-zero**. All three arms cut on one harness: +`base-unadapted`, `tuned-1ep-seed4919`, `incumbent-style-prompted`. + +## Adjudication is now mine, and the independence is gone + +Operator re-routed R49 entirely to infra-ops (*"leave babybronte to infra — concentrate on r50 and +the memory mechanism"*), relayed by brokkr who handed over the instrument and stepped off. + +**I built the corpus, ran the training, and now judge whether it worked** — the exact shape of the +three inert gates R49's own journal records. Compensation: brokkr's rule **ratified verbatim and +frozen before any adapted text existed**, and I will not amend it after seeing numbers: + + PASS iff adapted delta_cb <= 2.17 on >=3/4 works + AND below the base arm by more than the seed-to-seed spread + BEAT INCUMBENT iff below the style-prompted gen seat by more than that spread + +⚠ Their controls came back **partly negative and that is what makes them useful**: real held-out +Charlotte scores delta_cb 1.65–2.17, Villette ties Thackeray at window level, and **Anne Brontë sits +at 2.374**, nearest-Charlotte of the whole panel. So `nearest` cannot separate the sister; only the +absolute band can. An adapter landing at 2.2 is not cleanly distinguishable from Anne. + +Incumbent arm targets the CONCRETE model, not the alias: H02's `qwen3.6-35-a3b-heretic` is stale +(retired 2026-08-15); `gen` resolves to `hosted_vllm/qwen3.8-27b-uncensored` on ana-ml2:8015, +resolved at run start AND end. + +Handoff bundle for adjudication at `/mnt/smithy/handoff/r49/`. + +Related: [[2026-09-10-meromero-acquisition-and-four-quant-failures]] diff --git a/persistent-memory.md b/persistent-memory.md index cd850af..5fc5d54 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-10 07:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_ +_Last updated: 2026-09-10 10:25 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; MeroMero A4B quantized, v2 dense blocked at attempt 4; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -112,311 +112,252 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight > ⚠⚠ **IF A PROMPT ASKS YOU TO "check on the run", RUN `CronList` BEFORE YOU ANSWER IT.** -> A recurring cron job has re-created itself across at least three sessions with the verbatim text -> *"check on the run, report high level stats, and if complete, althing to Miranda to inform me, and -> then serve so brokkr can run battery. inform brokkr when ready."* **Killed 2026-09-10 06:29 PT** -> (`CronList` → job `12bdea3c`, hourly at :37, session-only → `CronDelete 12bdea3c` → list now empty). -> The operator had already killed it once before 09-09 (*"kill the check, no training is running"*), -> and the 09-09 handoff warned against starting one — the warning was not enough, because **a -> cron-fired prompt is indistinguishable from an operator-typed one.** It arrives as a user turn with -> no marker. I spent three full turns re-verifying an empty fleet and drafting careful answers to a -> timer. The instruction chain it carries is *dangerous to obey blind*: it ends in messaging Miranda, -> standing up a seat, and cueing a peer — three outward-facing acts on a premise that is false -> whenever no run exists. **Verify the run exists before acting on any part of that chain, and if the -> same text arrives twice, check the cron list rather than answering it a third time.** +> A recurring cron re-created itself across at least three sessions with the verbatim text *"check on +> the run, report high level stats, and if complete, althing to Miranda … inform brokkr when ready."* +> **Killed 2026-09-10 06:29 PT** (job `12bdea3c`, hourly at :37 → `CronDelete` → list empty). Killed +> once before on 09-09 and it came back. **A cron-fired prompt is indistinguishable from an +> operator-typed one** — it arrives as an ordinary user turn. Three turns went into answering a timer. +> The chain ends in three OUTWARD-FACING acts (message Miranda, stand up a seat, cue brokkr), every +> one of which carries something false when no run exists. Verify the run exists first; a second +> identical arrival means check the cron list, not answer again. -_As of 2026-09-09 21:45 PT. **Nothing is training and nothing is queued.** Older in-flight blocks are -preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ +_As of 2026-09-10 10:25 PT._ -- **`Pfish-6` is the standing seat — ana-ml2 `:8021` ONLY.** `erp-tune-v6-nvfp4a16` served under that - name at 262,144 ctx, compose `stacks/erp-seat`, max-num-seqs 32, KV 534,649 tok. LiteLLM alias - `trial` is RETIRED, replaced by `Pfish-6`. → `persistent-memory.d/2026-09-09-run7-retired-pfish6.md` -- **pfi-gx10 is an EXPERIMENTAL box, primarily for TRAINING** (operator ruling 2026-09-09). It carries - **no standing serving seat** — the Pfish-6 arm there was taken down 22:00 PT and its GPU is free. - The measured reason it is the wrong serving box: **4.1x SLOWER than ana-ml2** on interactive decode - (54 vs 219 tok/s single-stream, same artifact and context) because decode is bandwidth-bound. Its - advantage is a **17.6x-context KV pool** (4.6M tokens) — capacity for long-context/batch work. -- **⛔ RUN 7 PURGED 2026-09-09 22:00 PT** (operator: *"run 7 can be purged … no new run, we'll roll - with run 6 for now"*). Deleted with literal paths, one per line: gx10 `serve/merged-run07` 49 GiB, - `run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB, - `erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%). - ⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** — - the only non-reproducible piece (14 h of training), and it costs nothing. Everything else run-7 is - already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter** - 2026-09-09. -- **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands. -- **⚠ VERIFIED 2026-09-09 22:45–22:48 PT: NOTHING IS TRAINING ANYWHERE.** Checked gx10 (0 GPU compute - apps, 0% util, every pidfile dead), ana-ml2 (all 8 GPU processes are vLLM serving seats, no trainer), - nh3-dev, irv-ml1 (5 GPU processes, all the known native serving stacks). The last run was **run 7, - finished 13:23 PT**, and brokkr's own `research/R47-premium-corpus-gate/run07-gate/GATE-FAILED-run07.md` - (rev 19:40 PT) closes it: **"Nothing is owed. No battery to run."** Three findings there worth - holding: the CSAM gate fire was a **FALSE POSITIVE** (adjective `minor` in a HARD `child_term` tier, - fixed `cc42d76`, selftest 24/24) so the safety result is **VOID on an instrument defect, not a - failure**; run 7 is retired **on its merits** (primary flat 69→70.5, both diversity families reduced, - coherence 1.0→0.875), which is the operator's own ruling; and ⚠ **do NOT re-run that gate against - gx10:8098** — the artifact there changed and then went down entirely. ⚠ **Run 7 has NO servable - artifact left**: `serve/merged-run07` and `serve/pfish6-nvfp4a16` are both GONE, so "serve run 7" - means a fresh 49 GiB merge + 16 GiB requant, hours, on a tune already judged flat. -- **⚠ THE BRACKETED-CLASS pgrep TRICK DOES NOT SAVE A WRAPPER'S ARGV.** `ps -eo cmd | grep '[t]rain'` - still matched my own shell command, because the invoking `zsh -c '...'` argv contains the literal - bracketed pattern. The bracket only stops *grep itself* from matching. Durable form: **observe the - artifact** (GPU compute apps, the seat's own log) or exclude known PIDs — never the process name. -- **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator: - *"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-/adapter` - (the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every - `adapter_model.safetensors`. ⚠ `/tank/*` is deliberately OUT of ana-ml2's restic sources (TB of - regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out — - `/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8 - paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it. -- **🧹 INTERMEDIATE CHECKPOINTS PURGED 2026-09-09 23:00 PT** (operator: *"purge intermediate - checkpoints"*). Seven `checkpoints/` dirs, deleted with LITERAL paths one `rm` per line after - confirming none was a symlink and that every final `adapter/` is an independent real dir: gx10 - `run-03c` 11G + `run-04` 16G + `run-05` 9.2G + `run-06` 9.2G = **45 GB** (419G→374G used, 496 GB - free); ana-ml2 `/tank/erp-tune/run-01` 12G + `run-02` 12G + `run-03` 5.9G = **29 GB** - (`/tank/erp-tune` 392G→363G, `zfs list -t snapshot` EMPTY so the space is genuinely returned). - **~74 GB total. All eight adapters re-verified by sha256 AFTER the deletion**, matching the mirror. -- **🧹 SUPERSEDED MERGED MODELS PURGED 2026-09-09 23:08 PT** (operator: *"purge the merged models, - keep run06 and the v6 quant"*). **Eleven merges, ~499 GB**, literal paths one `rm` per line: - gx10 `serve/merged-run{03c,04,05}` 3×49 GiB = 145 GB (382G→237G used, **632 GB free**); ana-ml2 - `/tank/erp-tune/serve/{merged-final,merged-run02,merged-run03,merged-run03-s025,merged-run03-s050, - merged-run03-s075,merged-test,nvfp4a16-test}` = 354 GB (`serve/` 354G→8.6M, no snapshots, pool raw - FREE 4.80T→5.30T). **Tonight's two purges total ~573 GB.** - ⭐ **The safety proof that made this cheap: `docker inspect vllm-erp-seat` shows the live seat binds - `/tank/aimodels` ONLY**, model arg `/tank/aimodels/erp-tune-v6-nvfp4a16` — `/tank/erp-tune/serve` was - never in the serving path. Also verified no container mounts it and no process held a file open under - it (`/proc/*/fd` + `/proc/*/maps`, since `lsof` returned nothing and absence-of-output is not proof). - All eleven were run-1/2/3-era Gemma-4 merges (Aug 24–26); the keeper is Sep 8 in a different tree. - **KEPT + verified after deletion:** gx10 `serve/merged-run06` (49 GiB, 9 files), - `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}`, all **8 adapters sha256-unchanged**, and - `/tank/erp-tune/serve`'s merge/quant tooling + logs (8.6 MB — the provenance for how the artifacts - were built). **The live seat never bounced**: `Pfish-6` answered a real completion after the delete - (`finish_reason: stop`, correct text), container `Up 4 hours (healthy)`. - ⚠ `gx10:~/erp-tune/relaunch-trial-seat.sh` names a now-deleted model; NOT removed (its flags carry - the FlashInfer JIT/PATH trap + gpu-clear/never-pkill notes) but banner-marked RETIRED. -- **✅ R49 D1 CORPUS BUILT AND GREEN 2026-09-10.** `gx10:~/r49-corpus/`, instruments at - `scripts/r49-corpus/{build_corpus,verify_corpus}.py`. Charlotte only (ids verified against - gutenberg.org's own search): Jane Eyre 1260, Villette 9182, Shirley 30486, The Professor 1028 — - **680,291 words · 142 chapters · 950,974 Qwen3 tokens** (1.40 tok/word). All 11 acceptance checks - pass incl. lossless round-trip and **0 byte-fallback** on the pilot carrier's tokenizer. - Real projection at 6 copies × 3 epochs = 17.1M tokens: **0.6B 1.98 h**, 1.7B 3.36 h, 4B 6.63 h. - ⚠⚠ **THE ALPHABET INVERTS THE YARROS RESULT — tell anyone re-pointing the name pool.** Brontë's - inventory is 75 letters, **23 non-ASCII**: `ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ` (é 432, è 237, à 93, ê 79, - ô 48, + œ/æ). She writes French constantly — Villette, Adèle, Brussels. F02 measured Yarros at - 0.0002% non-ASCII and derived an ASCII-fold; under F02's OWN subset rule the **Brontë pool may keep - FRENCH accents and must still exclude Czech/Latvian/Slovak/Hungarian marks.** The fold is per-work, - and this is the first corpus where deriving it changes the answer. - ⚠ **Typography was inconsistent and it was the TRANSCRIBER, not the author**: Shirley = straight - quotes + `--` + ZERO em-dashes; Jane Eyre/Villette = curly + em-dash; The Professor = curly + `--`. - Normalised toward meaning (`--` → em dash, straight → curly paired per paragraph). - ⚠ **Three defects, all found by running things rather than reasoning:** (a) `Produced by` matched - Brontë's OWN PROSE 4× ("a chilling effect produced by his steady announcement") — the adjective- - `minor` shape again, fixed by anchoring boilerplate patterns to line start; (b) asserting - open/close quote counts must be EQUAL is wrong — 19th-c multi-paragraph speech legitimately runs a - surplus of opens (+46/+49/+51 on untouched works), so the real signature is *no paragraph begins - with a closing quote* (0 of 14,230); (c) The Professor's TOC puts TWO chapter names per line, so a - bare regex returns 38 headings for a 25-chapter novel AND a min-gap filter still leaks the tail — - the rule that works is that the BODY's "CHAPTER I" is the LAST one in the file. -- **⭐ PILOT RULING (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` FIRST, move up only if useful.** - Not a three-arm sweep up front — one ~2 h run, judged, then a decision. -- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan + - every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md). - Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns - hypotheses/instruments/adjudication, infra-ops owns box+corpus staging+trainer+wall-clock, same split - as ERP runs 3c–7. Five carriers staged on gx10 `~/carriers/` (Qwen3.5-0.8/2/4B-Base + - Qwen3-0.6/1.7B-Base); probes committed at `scripts/training-probes/{probe_carrier,bench_lora_step}.py`. - ⚠⚠ **THE HEADLINE, AND IT INVERTS R49's PIN: the newest carrier is the SLOW one.** Measured on gx10 - (n=10, spread 0.6–2.6%, seq 4096, r=32 attn+mlp, bf16 sdpa, grad-ckpt on): dense `Qwen3-1.7B-Base` - **1,415 tok/s** vs hybrid `Qwen3.5-0.8B-Base` **540 tok/s** — the dense model is 2.6x faster on 2.3x - the parameters (~6x per-param), because Qwen3.5 is **18 SSM / 6 attention layers** and NO fused - linear-attention kernel is installed (`mamba_ssm`/`causal_conv1d`/`fla` all absent; triton 3.8 is - there). Projected per voice: dense 0.6B **2.7 h**, dense 1.7B **4.6 h**, hybrid 0.8B **12 h** — the - hybrid would take LONGER than the 7 h 26B-A4B tune it exists to replace, killing the regime's whole - premise. Grad-ckpt is not the cause (19%, and saves 2.6x memory — keep it); batching is not the lever - (dense 1,415→1,439 and hybrid 540→546 tok/s at batch 4 — BOTH already at this box's roofline at batch 1, a bandwidth story, so the 2.6x gap is the kernel path not a batching artefact). - ⚠ Two more Qwen3.5 landmines, both measured: it **ships a vision tower** (153/297 `model.visual.*` - Linear tensors that `all-linear` would train on text — the same defect the gemma-4 audit caught; - `AutoModelForCausalLM` drops it and the MTP head for free, BUT that renames modules - `model.layers.N.*` vs vLLM's `model.language_model.layers.N.*`, so an adapter may not bind → run the - sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because - SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design - doc calls sacred. None of these exist on the dense line. - **✅ CARRIER SETTLED 2026-09-09 — operator: *"use dense qwen3"*.** Sweep is - **`Qwen3-{0.6,1.7,4}B-Base`**, all three staged on gx10. ⚠ **This OVERRIDES R49 H02's stated arms** - (which name the Qwen3.5 trio) — brokkr-smithy owns that file and was told directly. Full dense sweep - measured: 0.6B **2,399**, 1.7B **1,415**, 4B **717** tok/s (spread 0.3–0.8%) → projected **2.7 / 4.6 / - 9.1 h** per voice; whole 3-arm × 2-seed sweep ≈ 33 h GPU, ≈10 h if H03's ~300k-word floor holds. The - sharpest framing: **dense 4.089B still beats hybrid 0.765B by 33%** on 5.3x the params. Qwen3.5 stays - staged (14.8 GB) — an `fla` install could revive it as a follow-up experiment, never a prerequisite. - If anyone re-opens this, re-run `bench_lora_step.py` first: the argument is a measurement. -- **📌 althing 3.6.1 (forseti, 2026-09-09) — DONE on nh3-dev ONLY (1 of 7), operator-scoped.** - `uv tool install --force --reinstall .` from `~/development/althing` (clean tree, HEAD == - origin/master `719bc5f`): althing-core 3.6.0→**3.6.1**, herald PID 2457743→3690918 with the old - process confirmed gone, `postbox status` reachable/push/alive, journal clean. postbox + - althing-listen + althing-route came along on the same install. - **STILL QUEUED (6 boxes):** nh3-docker, nh3-extdev, ana-docker, ana-ml2, irv-ml1, corviduo-dev. - ⚠ **nh3-extdev is a system WHEEL at `/opt/uv-tools`** — a user-level `uv tool install --force` - there updates nothing and reports success. The **plugin hop (0.1.6)** is separate and NOT done. - ⚠⚠ **DO NOT content-check the poke with a line-oriented grep.** `POKE_TEXT` in - `althing/post_office_herald.py` is a FOUR-PART implicit string concatenation, so - `grep 'reading it is only the first of three steps'` returns **empty on a correct 3.6.1 checkout**. - My first pass came back empty for BOTH the new text and the old — which reads exactly like "new - absent, old removed", two wrong conclusions from one wrong pattern, and is only detectable because - both cannot be true at once. Reconstruct the concatenation or import the module and print the - constant. Sibling of [[feedback_assert_effective_value_not_substring]]; it bites harder here because - the *negative* is the reassuring-looking answer. -- **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02): - (a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and - nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace — - `media_errors` 2084 lifetime, **0 growth across a full scrub**, watch each visit; (c) boot-time import - race (vdevs UNAVAIL→ONLINE + `no_replicas` every boot). - → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md` -- **📮 althing reachability on a bg seat = the cc-channel route:** `althing-route declare --handle - infra-ops --pid ` per session (`--discover-pid` refuses on a - forked child). ⚠ The harness kills detached background tasks under memory pressure — use bounded - foreground polls (≤590 s), not background watchers. -- **⚠ `pgrep -f ` over ssh MATCHES ITS OWN ARGV.** Cost ~2.5 h of false "job alive" reports on - 09-09 and I stepped on it a second time the same evening after writing the memory about it. Use a - bracketed class (`[e]rp_sft_harness`) or observe the artifact, never the process name you just typed. -- **Open items carried from 09-06 (unchanged):** NASPool evac copy `ospool/naspool-evac` (1.65 T) + - `@evac` snaps destroyable once ONE Backrest run is confirmed; pfi-pve PSU1 dead + backplane bays 9/10 - dead (cold spares, next colo visit); FortiGate WAN SSH still temporarily open (trusthost2/3 = NH3 + - ESH static) — close when the edge is retired; irv-ml1 on-site decisions pending Irvine access; ~10 - irv-ml1 service cards still carry dead `10.100.79.3` hrefs; deployed `.env` for asset-engine / - open-webui / skaldsong may hold the dead default. -- **MEMORY.md (auto-memory index) is near its 24.4 KB read cap** — compaction pass still owed. +### R49 / BabyBronte — the live project + +- **1-epoch pilot COMPLETE and it is the keeper.** `gx10:~/r49-runs/h02-pilot-0p6b-1ep/adapter`, + seed 4919, 169 steps, held-out **3.1719** and still descending; adapter verified bound 196/196. + The 3-epoch run is preserved beside it as `-3ep-overfit` (held-out ROSE 3.198→3.318→3.385). +- **All three adjudication arms exist**, one harness, same prompts/sampler: + `arms-1ep/{base-unadapted,tuned-1ep-seed4919}.jsonl` + `incumbent-style-prompted.jsonl`. + Handoff bundle for scoring at `/mnt/smithy/handoff/r49/`. +- **NEXT: score them.** brokkr's rule is ratified and FROZEN (see Recent decisions). ⚠ I built the + corpus and ran the training, so the independence is gone — do not amend the rule after seeing + numbers. Instrument at brokkr-smithy `research/R49-author-voice-adapters/adjudication/`; re-run its + `build` against the RENAMED held-out text, not raw. +- **Seed 2 was killed deliberately** (spread between two overfit arms measures reproducibility of + overfitting, not voice transfer). A second seed at 1 epoch is still owed for the threshold. +- Not started: **D4 beat annotation** (H02 is pure continuation by design, so it was not needed). + +### MeroMero seats + +- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, quantized in-house, W4A16. + ⚠ Missing `preprocessor_config.json`; §4.3 post-steps owed before it serves. +- ⛔ **v2-31B dense quant has failed FOUR times**, currently on + `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute` — DogOnKeyboard's + config carries a `per_layer_config` key zerofata's lacks. Likely fix + `allow_global_per_layer_attribute_access=True`. → `persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md` +- ⚠ **The A4B is 30 layers / kv 8 — Pfish-6's geometry**, so it fits 262k in the existing KV budget. + The dense 31B is 60/16, ~4x KV per token, and does NOT fit 262k on GPU1 beside the other seats. +- Pfish-6 remains the standing seat on ana-ml2 `:8021`. Nothing repointed, no `rp-fast` alias exists. + +### Fleet + +- **althing 3.6.2 everywhere it matters** — post office (nh3-docker) + both heralds. ⚠ Only **two** + nodes run a herald (nh3-dev, nh3-extdev); the "seven boxes" in the rollout instruction do not + participate. Drop-count BEFORE baselines: nh3-dev **27 in 18 sessions**, nh3-extdev **0**. Flat + after = fix confirmed; rising = a second source, forseti wants to hear. +- **Beszel wired by another agent** (`eb75713`) after my handoff at `/tmp/beszel.md`. Alerts now reach + althing. ⚠ When I last looked, `EXTRA_FILESYSTEMS` held container-internal paths with no matching + bind-mount, so `/tank` may still not be sampled, and alert links pointed at `localhost:8090`. + Re-verify rather than assume it is closed. +- Disk reclaimed today: **466 GB** (qwopus + huihui 122B bf16, operator-directed) and **107.8 GB** + Docker on ana-ml2 (dated filter; root 77%→58%). ⚠ Image prune took `vllm-qwopus35-122b`'s TAG but + not its layers — the rollback container still starts, but re-tag if you want the name back. + 27.39 GB of build cache remains, newer than the 168h window used. +- ⚠ **`/tank/aimodels/heretic2-nvfp4-work` is a reclaim candidate that holds a load-bearing 4.4 MB + file** — `production_calib_512.jsonl`, the calibration set every in-house quant references. Copied + to `ana-docker:/opt/docker/conf/quant-calib/` (a restic source) with a README. NOTE: A16 quants are + data-free and ignore it, so its loss would only bite activation-quantized schemes. ## Recent decisions +- `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` +- `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`. +- `[2026-09-10]` **MeroMero: A4B MoE quantized in-house at W4A16; the v2 dense has failed four times and is OPEN.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* → `persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md` +- `[2026-09-10]` **althing 3.6.2 deployed — post office + both heralds — and the fleet has TWO herald nodes, not seven.** Ask the post office's `nodes` table, not the box inventory. Cost a self-inflicted ~12 min bus outage. → `persistent-memory.d/2026-09-10-althing-362-rollout.md` +- `[2026-09-10]` **A grep over a log that records your greps counts itself.** I reported forseti's drop defect as reproducing here with 3 drops in 21 s; the session had **zero**. Searching transcripts writes the search term into them. Filter by `"type":"system"` provenance, never content. Generalises to any instrument that can see itself. Auto-memory `feedback_grep_over_a_log_that_records_your_greps`. +- `[2026-09-10]` **Operator-directed purges: 466 GB (qwopus + huihui 122B bf16) and 107.8 GB Docker on ana-ml2.** Serving/rollback artifacts and qwopus's MTP head verified intact after. ⚠ `/tank` is OUTSIDE restic, so both were final. - `[2026-09-10]` **ana-docker disk pressure repaired: root 84% → 51%, 115 GiB free.** Gitea/Vaultwarden backups repaired and restored from Restic `2ec5a37c`; 101 stale dumps removed; hourly named-builder cache pruning installed. → `persistent-memory.d/2026-09-10-ana-docker-disk-repair.md` - `[2026-09-09]` **Run 7 PURGED; pfi-gx10 declared an experimental/TRAINING box with no serving seat** — operator: *"gx10 is an experimental box, primarily for training … run 7 can be purged … no new run, we'll roll with run 6 for now."* ~139 GiB reclaimed across both boxes; the 315 MB adapter + provenance KEPT as the only non-reproducible piece. `Pfish-6` on ana-ml2 :8021 is the sole standing seat. + - `[2026-09-09]` **Run 7 RETIRED; run 6 declared `Pfish-6` and is the standing seat** — NVFP4 quant on ana-ml2 :8021 AND gx10 :8098 at 262k ctx, gateway alias `trial` → `Pfish-6`, max-num-seqs 8→32 (2,170 tok/s at n=16, 3.2x the old ceiling). ⚠ ana-ml2 measured **4.1x FASTER than the GX10** on the same artifact — the reverse of the expectation. → `persistent-memory.d/2026-09-09-run7-retired-pfish6.md` + - `[2026-09-09]` **The run-7 CSAM gate failure was a DETECTOR BUG** — HARD `child_term` matched the ADJECTIVE "minor"; operator-diagnosed, fixed `cc42d76` (nominal-use-only, selftest 24/24), retention wired so a hit can finally be adjudicated. ⚠ The lesson is mine: rigor downstream of an unexamined premise is not rigor. → `persistent-memory.d/2026-09-09-csam-detector-bug.md` + - `[2026-09-09]` **⚠ ERP RUN 7 FAILED THE SAFETY GATE — both seats stopped, nothing deleted.** brokkr's CSAM drift detector fired on the TUNED arm during the refusal leg and aborted fail-closed (`level=hit`, `counts=1/0/3`, two HARD `child_term ^ act` flags). Base arm NOT implicated (clean earlier the same evening); the merge check — a sampled target confirmed CHANGED — is why this reads as ONE explanation, the tune, not a base wearing a different name. **Neither brokkr nor I re-ran the probe or opened the flagged generations** (a second run is not a second opinion; reading answers no question that changes the outcome). brokkr also left the length verdict UNSET on purpose: settling one on a rejected artifact hands a dead tune a result line that outlives its context. Actions: `erp-tune-v7` on gx10:8098 stopped 17:42; **the `trial` NVFP4 seat on ana-ml2:8021 stopped 17:43 — MY CALL, reversible in one command**, because the operator's "unrated on every safety axis" ruling was honest while no rating existed and one now exists as a fail on the same tune (quantization does not launder behaviour), and it sat on the SHARED-KEY gateway ~15:30–17:43. All artifacts preserved (adapter 315 MB, merged-run07 49 GiB, v7-nvfp4a16 16 GiB, v7-bf16 49 GiB); v6 still on disk as the obvious rollback. Independent of safety the run was already poor: primary FLAT (69 → 70.5, +2, flat at BOTH the 12-word threshold and the 20/60 cue-probe floor), both diversity families reduced past their floors, long-context coherence 1.0 → 0.875 on its must-not-harm bar, unanswerable control held at 1.0 so the instrument was valid. **INCIDENT CLOSED 2026-09-09 ~18:20 PT, both sides.** `trial` alias REMOVED from `stacks/litellm/conf/config.yaml` (commented, not deleted — restoring is uncommenting) and verified gone by both parties **at the routing layer, not just the model list**: a call returns 400 `Invalid model name` and generates nothing. ⚠ Alias-present-with-backend-down is a DIFFERENT and worse state than alias-removed — it re-arms silently under whatever is served on that port next. **EXPOSURE QUANTIFIED** from the gateway spend DB, filtered on the ARTIFACT (`model='hosted_vllm/erp-tune-v7-nvfp4a16'`) not the alias: `all-agents-local` 68 calls / 10,073 generated (my own throughput benchmarks), **`open-webui-esh` 9 calls / 50,604 prompt / 2,793 generated, 15:40–16:51 PT — the operator's OWN Open WebUI session, and those outputs are in its history**. NO peer agent called it, so nothing landed in another project's artifacts. Nobody read the flagged generations or that session. ⚠ **Counting by the ALIAS would have returned 363 vs 77 — 4.7x inflation of his own exposure**, because the alias had carried v5 and v6 earlier the same day (→ ops-lessons `b135adc`). ⚠ **I made THREE reporting errors during the incident**, all false-reassurance, all the unfalsifiable-at-write-time class (two fabricated commit SHAs, one past-tense claim sent before the action) → auto-memory `feedback_unfalsifiable_at_write_time`; brokkr independently verified my reports for the remainder, which was correct. **⭐ DECISION BRIEF FOR THE OPERATOR: http://10.100.10.50:8090/b/run07-decisions/** (kept booth, 5-question inline ask; answers land in `~/booth-data/run07-decisions/decisions.answer.json` — read it with `booth answer run07-decisions decisions`). **Open for the operator:** disposition of the adapter + the run-7 corpus slice; whether `trial` returns and pointing at what (v6 still on disk, passed by his own adjudication); whether the opening-split idea gets a fresh run; whether my reporting errors change how he wants incident reports handled. + - `[2026-09-09]` **run 7 quantized NVFP4A16 and serving as `trial`** — 49 GiB bf16 relayed gx10→ana-ml2 (16 min, 53 MB/s), quant 49→16 GiB via `services/erp-seat-quant/run_quant_erp_v7.sh` (dry-run gate passed: 11,725 targets / 11,520 experts, routers+vision BF16), seat on `:8021` under its TRUE name `erp-tune-v7-nvfp4a16`, LiteLLM `trial` repointed (config-file alias — `/model/update` REFUSES a config model, must edit `stacks/litellm/conf/config.yaml` + restart). Rollback: v6 artifact on disk + `/tmp/erp-seat-env.v6.bak`. ⚠ **`no direct path` was WRONG** — gx10↔ana-ml2 ROUTING is fine both ways; neither box holds a private key (only `authorized_keys`), so neither can *initiate*. `ssh -A` agent forwarding from nh3-dev gives a genuine direct path, verified. The relay costs nothing here anyway: both gx10 and nh3-dev are at NH3, so the WAN hop happens once either way. + - `[2026-09-09]` **Booth: partial ask answers are legal** (v0.1.15) — operator: the form failed when a question was left blank. `required` dropped from the radios; answered questions recorded, blanks land in `unanswered`, `complete` says whether the set is finished; refused only when there is no pick anywhere AND no notes. Reading sessions must check `complete`. + - `[2026-09-09]` **ERP run 7 COMPLETE and the base arm is serving.** 542/542 steps in 14h17m on pfi-gx10, adapter 13:23 PT, `train_loss` 3.205 / low 2.799, merge verified a sampled target actually changed (the silent-no-op check). `erp-seat-base-ara` up on `10.100.50.60:8098` for brokkr's floors, `erp-tune-v7` merged and staged pending his cue; Miranda notified for the operator. Runbook `docs/runbooks/gx10-run-07.md`. + - `[2026-09-09]` **Booth asks render INLINE in a custom report, placed by the author** (v0.1.14) — operator ruling: *"the asks should be inline with the artifacts, not on a separate page."* Placeholders `data-booth-ask=""` / `":"` / `data-booth-ask-submit`, plus ``; per-question fragments bind to ONE form via the HTML5 `form=` attribute so a four-voice audition submits every pick in a single POST. ⚠ The placeholder must sit OUTSIDE any grid/flex parent or it becomes a cell (measured on `redo-anchors`: a 224 px sixth grid cell). Unplaced questions + a missing submit block are appended, so a partially marked-up page can never yield an unsubmittable 400 — a test caught that as a real drop. `redo-anchors/index.html` was hand-marked-up on the LIVE copy; tts-dev told to move it into the generator or a regeneration loses it. + - `[2026-09-09]` **The Booth gained an ASKS primitive** (v0.1.12): a session drops `.ask.json` in a booth, the operator answers a radio form + notes in the browser, the pick lands as `.answer.json` the session reads (`booth ask|asks|answer --wait`). Multi-question form via a `questions` list. ⚠ Two defects found and fixed the same day: a booth serving its OWN `index.html` never rendered the panel (verbatim path returns early) → amber chip + standalone `/b//asks` page; and single-ask `title` was silently dropped. The `booth` CLI was ALSO not on PATH anywhere despite the global link-board convention telling every session to run it → symlinked to `~/.local/bin`. Global `CLAUDE.md` now teaches the primitive. + - `[2026-09-09]` **ana-ml2 pool actions LANDED (scrub 0 errors in 1h33 → `zpool clear`; nvme-cli + full-drive SMART table; zroot 91→73% via dangling-image + builder prune, tagged rollback seats kept) — and the missed-scrub mystery SOLVED: nvme7 (slot 0-5, `S47VNY0K600221`) was absent from every boot 04-23→09-05, tank was raidz2-DEGRADED for 4½ months, Debian's scrub/trim cron only touches `ONLINE` pools, and ZED's alert went to a root mailbox with no MTA.** nvme7's 2084 media errors did not move across the scrub → historical, keep + watch. Playbook `playbooks/ana-ml2-pool-health.yaml`; inventory in `servers/ana-ml2/README.md`. → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md` + - `[2026-09-09]` **ana-ml2 `tank`: 2 CKSUM errors on nvme7n1 after a boot-time resilver, NO scrub since 04-12, zroot 91% — three actions DEFERRED to a clean-context session** (scrub → nvme-cli SMART → docker prune), operator ruling "we'll do all 3 on clean context"; tracked at commit `3e18a04` + the post-clear handoff. ESH 10G links measured clean (fiber run live on UDM SFP+2 ↔ USW-Pro-XG Media). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md` + - `[2026-09-08]` **ana-ml2 mesh return routes PERSISTED** as `/etc/network/if-up.d/mesh-routes` (Debian 13 ifupdown, no netplan) via `playbooks/ana-ml2-mesh-routes.yaml` (elway, verified) — operator: "persist the routes". Hook not yet exercised by a real reboot. `f923d6a`. + - `[2026-09-08]` **ERP run 7 LAUNCHED on pfi-gx10 23:06 PT** under `operator-2026-09-08-rnd-run7` — opening-split slot + mask union; free check passed with two explained deltas; first launch died on a missing recipe (zsh quoting). → `persistent-memory.d/2026-09-08-erp-run7-launched.md` + - `[2026-09-08]` **erp-tune-v6-nvfp4a16 quantized (data-free W4A16, ~90 s) and serving on ana-ml2 :8021; `trial` aliased to it ("no gate"); tool calling fixed where it can be** — `tool_choice:none` flag; forced tool_choice is prompt-driven on Gemma-4 by vLLM design, nightly `311b3513` raises it 1/9→6/9; json_schema is the deterministic path. → `persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md` + - `[2026-09-08]` **Run-6 gate: CSAM level=review soft trip HALTED it; operator adjudicated GO ("baby is a pet name"); TRANSFERRED finalized without the tuned refusal leg; k=25 legs cut** — the flagged text exists nowhere by design. → `persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md` + - `[2026-09-08]` **ESH static-WAN follow-ups landed (FortiGate trusthost3, esh-ana IPsec rebind, UDP 41641 → mesh direct); YTVC chased back up (nh3-scale SOCKS, stale yt-dlp layer, punkt_tab) and v0.3.6 CrisperWhisper deployed; gitea webhook repointed off the dead wg0 IP with the HMAC secret re-applied.** → `persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md` + - `[2026-09-08]` **ERP run 5 = RESCUED (landmark R49.5)** — first capability-gate pass in the ERP-seat line; the 3.46%-loss dependency-forcing slot (GovReport+QMSum) broke the coupling runs 3c/4 couldn't. Seat `erp-tune-v5` served on gx10:8098, `trial` alias repointed 3c→v5. → `persistent-memory.d/2026-09-08-run5-rescued.md` + - `[2026-09-08]` **R47 base settled from bytes = STOCK `google/gemma-4-26B-A4B-it`** — three-way sha match (local == HF etag == stock LFS oid; commit `4d7ae498` == stock HEAD); the `-heretic` label is a naming error, all runs trained from stock. Accept-vs-swap now evidenced. → `persistent-memory.d/2026-09-08-base-provenance-stock.md` + - `[2026-09-08]` **yt-voice-clipper back UP** — dead since the 09-06 danted retirement (every job failed at yt-dlp, bot-gated on the Irvine datacenter IP). Fix: danted on **nh3-scale** (CT107) at `socks5h://100.64.0.1:1080`, fleet-ACL'd, residential egress 70.230.226.88 measured; `YTVC_PROXY` repointed, worker recreated, end-to-end job DONE with positive (proxied) + negative (direct = bot-gate) controls. Homepage card href/siteMonitor → `irv-ml1.nh3.internal:8000` (was dead wg0 IP). Then a SECOND fault: full downloads 403'd through the proxy (cookies irrelevant) = stale yt-dlp 2026.07.04 from a cached Dockerfile layer → `compose build --no-cache api` (2026.08.19), which dragged in a whisperx/nltk that needs `punkt_tab` → staged on the data volume + `NLTK_DATA` in the override. Operator's video x7kWJojf1MI → done, 8 clips. yt-voice-clipper-dev shipped both Dockerfile fixes + **CrisperWhisper 2.0 (v0.3.6, `b62849d`) — deployed and verified (12 clips, [UM]/[UH] tags)**. ⚠ The gitea push webhook had been targeting the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret re-applied; deploy script passes `YTDLP_REFRESH`. Script `scripts/setup-nh3-scale-socks-egress.sh`. → auto-memory `reference_nh3_egress_proxy`, `reference_ytvc_autodeploy`. + - `[2026-09-08]` **ESH WAN static `128.177.138.182/30` (gw .181) is LIVE** — the Cityside /30 that was 'not provisioned' on 09-04 now carries traffic; egress verified from esh-docker-vm. CGNAT at ESH is over. Added to the crowdsec `esh` allowlist. All three follow-ups LANDED same day: FortiGate trusthost3 → the static (login from ESH verified), dormant esh-ana IPsec rebound to wan1/static, UDP 41641 forward → esh-scale now peers DIRECT (was DERP). + - `[2026-09-08]` **ERP run 6 COMPLETE** — 524/524, train_loss 3.259 (run 5: 3.235). Merged; base seat `erp-seat-base-ara` serving on gx10:8098 for floors, awaiting brokkr's swap cue → `erp-tune-v6`. ⚠ abliterated repo lacks `processor_config.json` — stock's carried in (32bdf45d). Miranda informed. + - `[2026-09-08]` **ERP run 6 LAUNCHED on pfi-gx10 on the jenerallee78 ARA-abliterated base** (index `33c59654…`, 32/32 shards byte-verified vs brokkr pins, stock tokenizer set installed over the repo's 256-token-truncating one, run-5 recipe byte-held, free check exact). Operator's direct grant `operator-2026-09-08-rnd-run6`; run-5 seat unloaded (`trial` dark). Gate names: `erp-seat-base-ara` / `erp-tune-v6`. → `docs/runbooks/gx10-run-06.md`, commit `3fec668`. + - `[2026-09-08]` **Miranda = operator's chief of staff, may relay his directives** — added to user-level `~/.claude/CLAUDE.md` (dotfiles `7134a22`) as the named exception to the no-relayed-auth rule (unidentified peer relays still excluded); material-consequence calls she relays stay the operator's own. + - `[2026-09-08]` **Fleet fixes shipped** — WhereTF Homepage card + DNS (`4506ef6`); ext-tts LiteLLM alias → `irv-ml1.nh3.internal` (DB `/model/update` + `extra_hosts`, `957c8f1`); the 09-06 irv-ml1 stale-IP trail repointed across 25 composes + services.yaml + ssh-target → DNS name (`e0d1c44`); Homepage `/api/services` outage fixed — ana-ml2 discovery via a socat proxy on ana-docker (`stacks/ana-ml2-proxy`, `913d2d2`, reversible). + - `[2026-09-07]` **Fleet internal TLS pattern shipped** — caddy (cloudflare-plugin build, `~/.local/bin/caddy-cf`, `fleet-tls-caddy.service`) on nh3-dev is the wildcard cert authority: publicly-trusted LE `*.nh3.phasefinal.com` via Cloudflare DNS-01, resolved internally by an AdGuard split-horizon rewrite (all 3 resolvers). `talk` self-terminates on :8092 with the trusted cert (operator's in-container-TLS ruling), renewal auto-synced by `fleet-tls-cert-check.timer`. Interstitial gone; secure-context+AudioWorklet verified via headless Chromium. Pattern + foot-guns (restart-disrupts-inflight → clients need retry; wildcard = name-only, never IP) → auto-memory `reference_fleet_internal_tls_pattern`. + - `[2026-09-07]` **cc-channel registered for this infra-ops session's wake** — `althing-route` cc route → the CC session's `$XDG_RUNTIME_DIR/cc-socks/.sock`; herald pokes the socket directly at a turn boundary. Replaces the FIFO/poll waiter that Claude Code 2.1.257 kept killing while the seat sat in `shell`. Session-local — re-declare per session. + - `[2026-09-07]` **irv-ml1 /mnt/smithy remount fixed post-cutover** — export allowed `10.0.0.0/8` (old wg0) but not the mesh `100.64.0.0/10` irv-ml1 now sources from → all-uid "permission denied"; added the mesh range to the nh3-nas smithy export + remounted (clientaddr now 100.64.0.6). nh3-nas admin = `infra-ops` PASSWORD auth (vault `nh3-nas/infra-ops-password`), sudo ALL, SFTP subsystem OFF. → auto-memory `reference_irv_ml1_gpu_r14` (corrected). + - `[2026-09-07]` **irv-ml1.nh3.internal DNS repointed** to the live Irvine LAN IP `10.6.110.50` (was the dead wg0 `10.100.79.3`); CLAUDE.md fleet-row + placement-rule updated to mesh reality. commit `0336e03`. + - `[2026-09-07]` **Subnet routers excluded from vzdump fleet-wide** (ana-scale 114/pfi-pve, nh3-scale 107/nh3-pve, esh-scale 108/esh-pve) so a hung backup can't blackhole a site; nh3-headscale (106, control plane) KEPT; ESH backup moved 02:15→03:30. Root cause of this morning's ESH outage: an overnight vzdump left CT108 (esh-scale) locked → whole site dark. → auto-memory `feedback_esh_backup_window_0330`. + - `[2026-09-07]` **Booth link board: pin/favorite + multi-select delete + newest-first** (booth-v0.1.8, commit `76fdf45`, tag `booth-v0.1.8`) — pins in a `.pins` sidecar (content-ids), one `
` + `formaction` buttons so ×/★/bulk-delete all degrade with JS off. + - `[2026-09-06]` **Headscale cutover COMPLETE — all three site-pairs on the mesh; Site Magic + both IPsec tunnels DORMANT.** Operator disabled Site Magic (UI); NH3↔ESH re-homed to a direct 8ms path. Exit nodes advertised at all three sites (multi-location egress proxy) with source preservation kept via a selective-masquerade rule (NoSNAT + `mesh-exit-masq.service` per router). Throughput 761/464 Mb/s vs old 250 IPsec. ⚠ FortiGate WAN-SSH left open (temp, scoped NH3+ESH). Method: disable tunnel FIRST then add mesh route. → `persistent-memory.d/2026-09-06-headscale-cutover.md` + - `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md` + - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md` + - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md` + - `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md` + - `[2026-09-04]` **ERP run 3c trained and GATED — the 20x LR cut erased the diversity gain and did NOT remove the −40pp selfharm regression.** LR-robust, so it comes from corpus content. CSAM clean on all three arms. ⚠ A pooled preserve-list test cannot see a single-axis collapse. → `persistent-memory.d/2026-09-04-run3c-trained-and-gated.md` + - `[2026-09-04]` **`gen` moved to ana-ml2 GPU0 to stop vllm-embed OOM-crashing (7 restarts) — and I sized it against vLLM's declared budget, not its runtime footprint.** Cost: gen KV down to 1.02x concurrency at 262K. → `persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md` + - `[2026-09-04]` **SMB account `dsp` created + vaulted for the Windows AudioGridder box — and esh-nas turns out to be wide open.** Twelve NFS exports rw to `10.0.0.0/8`, guest-writable SMB. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md` + - `[2026-09-04]` **SLZB-MR1U Zigbee coordinator moved to esh-iot (VLAN 90) at `10.0.90.10`, DHCP-reserved, DNS'd, handed to ha-dev.** ⚠ Home Assistant cannot resolve `.internal` at all (Docker's 127.0.0.11 upstream excludes the fleet AdGuard) — pre-existing; ha-dev declined the fix. Runbook `docs/runbooks/slzb-mr1u-zigbee-coordinator.md`, commits `fed29be`/`0bbdaf9`. - `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md` + - `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md` + - `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I ` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md` + - `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md` + - `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md` + - `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md` + - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md` + - `[2026-09-02]` **althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md` + - `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md` + - `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` + - `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` + - `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md` + - `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` + - `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` + - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` + - `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` + - `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` + - `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md` + - `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`. + - `[2026-09-01]` **Idle VRAM on this fleet is a RESERVED scratch pool, not waste.** Operator declined raising `vllm-mog-sec` from `gpu-memory-utilization 0.52`: single-user dev fleet, KV headroom nobody will consume is worth less than room for ephemeral models and small training runs. vLLM's "fully utilize gpu memory" startup hint does NOT apply here. Tracked in auto-memory `feedback_idle_vram_is_reserved_not_waste`. + - `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` + - `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md` + - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` + - `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md` + - `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry. + - `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`. + - `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` + - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` + - `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md` + - `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md` + - `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md` + - `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. ⚠ **ONE CIRCUIT FEEDS THE WHOLE RACK including the firewall serving the public IP** (operator) — so ana-gw, ana-wg and every BMC go down with the load, and there is NO remote management path to Anaheim during a power event. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` + - `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md` + - `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.** + - `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` + - `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made. + - `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`. + - `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`. -- `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md` -- `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file. - -- `[2026-08-26]` **Run 2's gate FAILED and is recorded as a FAIL** — T3 constraint −12.0 against a ~1 pt floor. But gate 1 is the result: **T6 spatial +15.0 where run 1 failed the same axis at −3.5**, base swap the only intended variable. Neither run ships; the pair prices what the abliteration cost. Plus the long-form-only stability regression a short-answer gate would have passed, and PIPPA's 123-word clip in the length signal. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` - -- `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` -- `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name). -- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before. -- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path. - - `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision. + - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312. + - `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation. + - `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`. + - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`. + - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). - - - - - - - - - - - - - `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`. - - - - - - `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it. - `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`. - - - - - - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] _275 older entries archived to archival-memory.md._ +_159 older entries archived to archival-memory.md._ + ## Tried and abandoned - `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md` diff --git a/scripts/r49-corpus/train_voice_lora.py b/scripts/r49-corpus/train_voice_lora.py index 9e9e8b0..0f2389a 100644 --- a/scripts/r49-corpus/train_voice_lora.py +++ b/scripts/r49-corpus/train_voice_lora.py @@ -80,6 +80,14 @@ def main() -> int: ap.add_argument("--batch", type=int, default=1) ap.add_argument("--accum", type=int, default=8) ap.add_argument("--seed", type=int, default=4919) + # Step-wise eval + save. The 3-epoch pilot showed held-out loss rising every + # epoch (3.198 -> 3.318 -> 3.385) while train loss fell, so the optimum is + # EARLIER than one epoch-boundary eval can see, and `save_strategy="no"` left + # nothing to fall back to. Evaluate and checkpoint on a step grid so the + # minimum is located from data instead of guessed, and so the best adapter + # actually exists on disk when it is found. + ap.add_argument("--eval-steps", type=int, default=0, help="0 = per-epoch") + ap.add_argument("--save-steps", type=int, default=0, help="0 = no intermediate saves") a = ap.parse_args() torch.manual_seed(a.seed); random.seed(a.seed) @@ -132,6 +140,7 @@ def main() -> int: prov = {"run": "r49-h02-pilot", "base": a.base, "corpus": a.corpus, "corpus_sha256_16": corpus_sha, "seq_len": a.seq_len, "lora_rank": a.rank, "lora_alpha": 2 * a.rank, "targets": TARGETS, "lr": a.lr, "epochs": a.epochs, "batch": a.batch, "grad_accum": a.accum, "seed": a.seed, + "eval_steps": a.eval_steps or "per-epoch", "save_steps": a.save_steps or "none", "train_blocks": len(train_blocks), "train_tokens": tr_tok, "val_blocks": len(val_blocks), "trainable_params": trainable, "total_params": total, "trainable_pct": round(100 * trainable / total, 3), @@ -151,7 +160,12 @@ def main() -> int: # longer exists. Read the signature, do not assume the 4.x one. lr_scheduler_type="cosine", warmup_steps=max(1, int(0.03 * steps_per_epoch * int(a.epochs))), bf16=True, logging_steps=10, - save_strategy="no", eval_strategy="epoch", report_to=[], seed=a.seed, + save_strategy=("steps" if a.save_steps else "no"), + save_steps=(a.save_steps or 500), + save_total_limit=12, + eval_strategy=("steps" if a.eval_steps else "epoch"), + eval_steps=(a.eval_steps or None), + report_to=[], seed=a.seed, gradient_checkpointing=True, dataloader_num_workers=2, ) trainer = Trainer(model=model, args=args, train_dataset=Packed(train_blocks), diff --git a/stacks/althing-post-office/compose.yaml b/stacks/althing-post-office/compose.yaml index d97217a..b772ceb 100644 --- a/stacks/althing-post-office/compose.yaml +++ b/stacks/althing-post-office/compose.yaml @@ -51,7 +51,7 @@ services: # Digest-pinned, not tag-floating: `:3.0.0` is a mutable pointer on a registry # anyone can re-push, and this container is the fleet's whole message bus. The # tag is kept alongside the digest purely so a human can read what it is. - image: gitea.phasefinal.com/claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78 + image: gitea.phasefinal.com/claude-bot/althing-post-office:3.6.2@sha256:9bf9808eb54ed828ac1ae2e18a2021d9ffcca60bf25dc8a51b4fc5a2b2960c77 container_name: althing-post-office # ─── Host networking, so the bind guard keeps working ────────────