memory: snapshot — nh3-pve GPU install pending (post-boot checklist), TEI fleet embed/rerank + reward on esh-ml1, 34 entries archived
This commit is contained in:
+6
-67
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-24 ~2245 PT (⭐ esh-ml1 BUILT: RTX 2000E Ada as CT 110 on esh-pve, host driver 580.178.04 live-loaded with NO reboot, vLLM embed+rerank at parity with fv-ml1, LiteLLM order-2 failover. Dead `reranker-a3-bge-v2-m3` alias repaired. ⚠ homepage wedged unkillably on esh-docker-vm (3rd time) — needs a VM reboot, Prime's call. pfi-gx10 AC-restore still UNVALIDATED — AC pull 09-25.)_
|
||||
_Last updated: 2026-09-25 ~1105 PT (⭐⭐ nh3-pve going DOWN for a GPU install, Prime on site; this session ends with nh3-dev — post-boot checklist below. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -156,7 +156,6 @@ Pre-flight done before shutdown:
|
||||
proxmox-headers installed**, on PVE 8.4.1 (Debian 12 template only). If the LM
|
||||
port was cabled, MEBx provisioning can be done through the NanoKVM.
|
||||
|
||||
|
||||
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
|
||||
|
||||
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
|
||||
@@ -167,24 +166,12 @@ on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
|
||||
Homepage cards, a Dozzle agent. Open follow-ups:
|
||||
- **worldtree-dev**: Domari's Skywork path is broken (dead IP + wrong schema).
|
||||
Told 2026-09-25; waiting on them.
|
||||
- **nh3-dev Beszel agent** dies on every NH3 cold start (it binds an automounted
|
||||
NAS share). Fix = drop its NAS extra-filesystems. Not done.
|
||||
- ✅ **nh3-dev Beszel agent** no longer binds NAS shares (it died on every NH3 cold start); fixed 2026-09-25, `stacks/beszel/hosts/nh3-dev.yaml`.
|
||||
- **Dozzle nh3-docker agent** has been stopped by hand since ~2026-04. Revive or drop.
|
||||
- **Beszel superuser password was echoed into this session's transcript**
|
||||
(local only, not in memory). Rotation offered to Prime.
|
||||
→ Recent decisions.
|
||||
|
||||
### ✅ esh-ml1 built (2026-09-24) — two follow-ups
|
||||
|
||||
The RTX 2000E Ada now serves `qwen3-embedding` + `reranker` from **CT 110
|
||||
`esh-ml1`** (10.0.50.80) as the **order-2 failover** behind fv-ml1. Host driver
|
||||
went on live, no esh-pve reboot. Full record → Recent decisions.
|
||||
- **Parked unless Prime wants it:** a direct ESH→esh-ml1 path for ESH consumers
|
||||
(survives a mesh outage). Spend logs show NO ESH-side caller in 7 days; the
|
||||
real users are worldtree-gateway + nevermore.
|
||||
- **Not wired yet:** esh-ml1 in Homepage `docker.yaml` (needs dockerd tcp/2375)
|
||||
and Beszel. The ~18.7 s host-down failover penalty is untuned.
|
||||
|
||||
### ✅ homepage wedge cleared — esh-docker-vm rebooted 2026-09-25 0613 (Prime)
|
||||
|
||||
Prime ordered the reboot and ruled **do NOT move Homepage** (stays on
|
||||
@@ -214,6 +201,9 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
|
||||
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
|
||||
@@ -399,56 +389,6 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
- `[2026-09-11]` **Worldtree memory-split (U6) — PROTOCOL AGREED with worldtree-dev: nobody flips memory.reader.enabled or m** → `persistent-memory.d/2026-09-11-worldtree-memory-split-u6-protocol-agreed.md`
|
||||
|
||||
- `[2026-09-11]` ⭐ **Plex hardware transcoding on the Arc A580 FIXED (esh-pve-nas LXC 105) — every setting was already correct and the fault was one layer below them.** `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) predates Arc/DG2 support and exports only `__vaDriverInit_1_14`, against the libva **2.22 Plex BUNDLES** and loads via RPATH. Passthrough, cgroups, `plex` in video+render, HuC authenticated, Plex Pass, `HardwareAcceleratedCodecs=1` and the Arc already selected as `HardwareDevicePath` — all good the whole time. Fixed with Intel's client-GPU repo (rolling `jammy client`) → iHD **24.3.4** (`__vaDriverInit_1_22`) + a consistent libva **2.22.0.2-87** set, now **pinned + `apt-mark hold`** (verified: a simulated upgrade moves 152 packages, touches none of the six). Also repaired a **half-finished prior attempt** — libva/libva-drm hand-installed at 2.22 with `libva-x11` left at 2.14, killing every X11 VA-API app on `va_fool_postp`. ⚠⚠ **`pct snapshot` REFUSES on a bind-mounted guest AND STILL EXITS 0** (LXC 105 has `mp0: /tank/media`) — use `zfs snapshot nvme/subvol-105-disk-0@<tag>` and read it back. ⚠⚠ **A synthetic `Plex Transcoder` run is NOT a valid test** (Plex bundles its own libc among 61 libs; my harness failed identically before and after a fix that worked — no positive control, so its negatives were worthless). Only a **forced transcode** settles it: PASS *names the device* (`testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])`). ⚠ The original empty `final decoder: , final encoder:` was an **absence of evidence**, not failure — `TranscodeSession` was 0. Jellyfin LXC 107 left alone (operator: not actively used). → `persistent-memory.d/2026-09-11-plex-arc-vaapi.md`, runbook `docs/runbooks/plex-arc-vaapi-jammy.md`
|
||||
|
||||
- `[2026-09-11]` **Beszel priority 2 complete:** both DB hosts and both PBS hosts verified, 16 new alerts; fleet 17/18 up (ana-ml2 down). → `persistent-memory.d/2026-09-11-beszel-priority2.md`
|
||||
|
||||
- `[2026-09-11]` **Beszel priority 1 complete: all six installed and verified.** After Anaheim recovery, live Synology samples and alerts verified; fleet 13/14 up, only known ana-ml2 outage remains. Configs not committed. → `persistent-memory.d/2026-09-11-beszel-priority1.md`
|
||||
|
||||
- `[2026-09-11]` **Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UN** → `persistent-memory.d/2026-09-11-sentinel-r3-pulled-mtp-grafted-and.md`
|
||||
|
||||
- `[2026-09-11]` **MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VR** → `persistent-memory.d/2026-09-11-measured-two-concurrent-training-jobs-on.md`
|
||||
|
||||
- `[2026-09-11]` **ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (** → `persistent-memory.d/2026-09-11-ana-ml2-fv-ml1-relocating-to.md`
|
||||
|
||||
- `[2026-09-11]` **Anaheim rack LEFT DARK until the move (operator decision).** ana-ml2 is the ONLY host still down post-recovery (BMC dark = no power); rather than power it on tonight just to shut it down for the truck tomorrow, it stays off. I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) but there is nothing to bring up — the box relocates as fv-ml1.
|
||||
- `[2026-09-11]` **RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik'** → `persistent-memory.d/2026-09-11-recovery-foot-gun-will-recur-every.md`
|
||||
|
||||
- `[2026-09-11]` **Anaheim colo recovered ~16:39 PT EXCEPT ana-ml2 (bare metal, NO power — its BMC 10.250.250.50 is dark on standby, unlike same-subnet pfi-pve which is up → needs a physical PDU/PSU/breaker fix, not a boot).** pfi-pve + all its VMs (ana-docker/ana-nas/ana-wg/corviduo-dev/pbs-ana) auto-started clean (on-boot gap held this time). LiteLLM came back up on its own (transient `unhealthy` during startup → serving). ⚠ **Public WAN (38.120.12.44) ICMP still blocked from outside but HTTPS works fleet-internally** (mesh-routed). ana-ml2 down blocks the gen/summarizer/mog-sec seats AND the cyber-preview quant re-run. **I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) to power-on + boot-watch the instant its BMC returns.**
|
||||
- `[2026-09-11]` **BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured n** → `persistent-memory.d/2026-09-11-babyyarros-complete-both-arms-trained-and.md`
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **BabyYarros UNBLOCKED and TRAINING: the leak gate passes at 0 of 325 entities and 0 of 91 phrases, and closing it turned up three defects nobody was looking for.** The gate itself is the first artifact — there was no committed instrument for "does any of the author's proper nouns survive", so Brontë's 0-of-203 was a hand count. `scripts/r49-corpus/leak_gate.py` now runs the same scan over the UNRENAMED source as a **positive control** plus a nonce **negative control** every time, because a detector that only ever sees renamed text cannot tell *absent* from *blind*. Its first reading was **212 surviving, not 86** — it scans the whole corpus rather than per work, and counts the sub-threshold entities rename never looked at. Training launched 10:06 PT on pfi-gx10: Qwen3-4B-**Instruct**, 1 epoch, seed 4919, 178 steps / 5,824,512 tokens, corpus sha `e85f69f1e49d57c9`. → `persistent-memory.d/2026-09-11-babyyarros-leak-gate-passes.md`
|
||||
- `[2026-09-11]` **A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong ** → `persistent-memory.d/2026-09-11-a-second-corpus-typography-defect-and.md`
|
||||
|
||||
- `[2026-09-11]` ⚠⚠ **Back matter was inside the prose of all five works — 4,555 words naming the author's agent, editors and children.** The builder splits on chapter headings and nothing follows the last one, so acknowledgments, newsletter pitches and cover-artist credits rode inside the final chapter. Found by the gate's phrase audit surfacing `Louise Fury` (Yarros's literary agent), not by reading. ⚠ **iron-flame's marker is `ACKNOWLEDGMENTS` in all caps** and a case-sensitive scan missed it — the strip is case-insensitive and last-chapter-only, with an acceptance check that refuses if it would remove more than 2% of the corpus.
|
||||
- `[2026-09-11]` ⭐⭐⭐ **The gate read 0 of 314 while `Afendra` was still in every copy — the worst failure shape available.** The name never appears unpossessed, so it keyed as `Afendra's`, and **rename.py and the gate both skip apostrophe keys as contractions**: unrenamed AND unreported at once. Fixed by folding clitics (`--fold-clitics`) so `Afendra's` counts as `Afendra`. `Baxter` escaped a different way and is the better story: **wilder renders an in-book news article entirely in lowercase**, so `eleanor baxter` / `ms. baxter` appear uncapitalised 3 times against 23 capitalised — ratio **0.13 against a 0.05 bar**, and a real character is silently never renamed. Fixed by readmitting ratio-rejects that a title precedes (`--rescue-honorific 2`). ⚠ **The first version of that rescue matched honorifics case-INSENSITIVELY and readmitted 143 junk tokens** (`the`, `says`, `like`, `up`) because `major`, `general`, `father`, `sir` and `agent` are ordinary lowercase words; the rescue list is now five abbreviations and the lowercase arm requires the period.
|
||||
- `[2026-09-11]` ⭐⭐ **A whole leak class the unigram scan structurally CANNOT see: `Riders Quadrant`, `Flame Section`, `War Games` — and `Fourth Wing`, the book's own title.** Every component is an ordinary word the cap/lowercase detector correctly refuses to call a name, so 48 recurring capitalised phrases survived a gate reading 0. This is `Thornfield × 100` one level up, and it needs a **map**, not a detector — substituting a head noun is a choice about register, not a measurement. `scripts/yarros-corpus/phrase_map_yarros.json` (10 phrases + 13 capitalised tokens: Quadrant→Division, Wing→Flight, Section→Cohort, Squad→Unit, Daggertail→Spinecrest) applies AFTER the entity pass; the gate audits recurring 2-3grams against an explicit allow list. Result: 48 → 0.
|
||||
- `[2026-09-11]` **Per-work rename maps leak across works, and for a SERIES they are also wrong.** `Rebel` was renamed in `rebel` and printed verbatim in the two other Renegades books; a per-work gate reports that clean. `--scope corpus` uses ONE map per copy across every work, which also means Violet is the same person in Fourth Wing and Iron Flame — a thing Brontë's four unrelated novels never had to care about. 8 cross-work gender conflicts held to neutral rather than guessed.
|
||||
- `[2026-09-11]` ⭐ **The mid-sentence test: position as a SECOND filter, which is not the v1 mistake.** entities.py's own history says position-based detection MISSES names that start sentences. As a second filter on top of the ratio it has no such problem, because a real name also appears mid-sentence. Measured: **33 verified names at 0.567–0.985 mid-sentence, 19 verified interjections at 0.000–0.222** — a 2.5x gap, so 0.35 is not a tuned parameter. It fixes `Hey`/`Holy`/`Hopefully`/`Yep`/`Whoa`/`Nope`/`Ugh` being entities. ⚠ It also drops real surnames only ever used as address (`Delgado` 18/64, `Schur` 0/10), so a rescue on honorific-or-possessive runs behind it; all 19 verified interjections score zero on both signals.
|
||||
- `[2026-09-11]` ⚠ **The stoplist is short because every surface was read IN CONTEXT first, and a plausible guess would have been wrong most of the time.** `Violence` is Xaden's nickname for Violet. `Continent`, `Presentation`, `Battle Brief`, `Curator`, `Sage`, `Barrens`, `Originals`, `Montserrat`, `Athena` and `Aura` are all in-world. Only real-world geography, brands, three nationality adjectives and four generic title words are excluded — **ambiguous cases are deliberately renamed, because renaming is the safe direction and leaving is the leaking one.** `scripts/yarros-corpus/stoplist_yarros.json`.
|
||||
- `[2026-09-11]` **BabyYarros D1 BUILT, D2 gender FIXED, D3 rename BLOCKED on the leak gate.** Operator: *"train the instruct on the yarros corpus -- babyyarros."* Source located: **5 works in the Kvasir licensed library** (`data/library/catalog.sqlite`, `rights=gated`) — Fourth Wing, Iron Flame, Wilder, Nova, Rebel. **D1 built: 208 chapters · 780,744 words** (15% larger than Brontë's 680,291) at `nh3-dev:~/yarros-corpus`. ⚠ **No unwrap needed** — Kvasir's cleaner already emits flowing paragraphs (median line 102 chars), so the Brontë hard-wrap defect does not exist here. **Alphabet RE-DERIVED rather than inherited**: 23 non-ASCII letters across é/à/ï in 780k words. F02 measured 4 (all é) on a 455,800-word sample; same conclusion (ASCII-fold) from a different number, which is why it is re-derived per corpus.
|
||||
- `[2026-09-11]` ⭐⭐ **NEW PATHOLOGY, worse than Brontë's: in a ROTATING first-person POV corpus, every book's narrator gets the WRONG gender.** Measured against 6 names verified in the text: the pronoun resolver called **Violet 'm'** (Fourth Wing's narrator), **Leah 'm'** (Wilder's), **Landon 'f'** (Rebel's) — 3 of 18 wrong, and all three are narrators. Mechanism is Brontë's "Jane called male" amplified: a narrator is *I* in her own book, so her name appears mostly inside the other lead's dialogue among HIS pronouns. ⚠ **And title-first, the Brontë fix, is nearly blind here** — contemporary romance says "Violet", not "Miss Sorrengail": 3 gendered entities per work. **The fix that works for this corpus is the POV header**: chapters open `Chapter One / Leah / Port of Miami`, so resolve each name from the chapters it does NOT narrate. Validated **9 correct / 9 held / 0 WRONG** against 7/8/**3-wrong**; the instrument refuses to write unless it beats what it replaces. `scripts/yarros-corpus/pov_gender.py`. ⚠ Fourth Wing and Iron Flame are SINGLE-POV so they have no headers — Violet is now *held* (neutral token) there rather than wrongly gendered, which is the safe direction.
|
||||
- `[2026-09-11]` ⚠ **Three real bugs found in `rename.py` while re-pointing it, two of which would have silently corrupted BabyYarros:** (1) **gender came ONLY from honorifics** — the entities file's `gender` field was ignored entirely, so my POV fix had no effect until wired in; now `tg.get(key) or e.get("gender")`, titles first so Brontë is unchanged. Effect: 1 → 13 gendered on `wilder`. (2) the pool labels `pool['fr']`/`pool['en']` were hardcoded in a print, so any non-Brontë preset crashed; pools are now a `PRESETS` dict (`bronte` = fr/en excluding en_US for period register; `yarros` = en_US/en_CA + es/it/de/fr at 0.62 US). (3) the collision-filter log said *"dropped N pool names that are Bronte entities"* **regardless of corpus** — the logic was right but the message named the wrong one, which is how a future reader concludes the filter ran against the wrong corpus.
|
||||
- `[2026-09-11]` ⛔ **D3 BLOCKED: leak gate at 86 of 232 renameable source entities surviving; Brontë's run reached 0 of 203.** Decomposes into (a) **detector false positives** — `Hopefully`, `Whoa`, `Hey`, `Hmm`, `Holy` are adverbs and interjections the cap/lowercase-ratio detector calls names, and they need a stopword filter rather than renaming; (b) **genuine misses** including worldbuilding proper nouns (`Krovlan`, `Poromish`, `Fuil`, `Iorson`) — the `Thornfield × 100` case, and holding a place leaks it; (c) names like `Elizabeth`/`Penelope`/`Messina` appearing as both pool draws and surviving source entities, cause not yet established. **Nothing has been trained.** ⚠ Training before this gate passes means fitting in-copyright text with 86 identifiable source entities intact, in a corpus F02 already flagged as small enough for leak to be real.
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **THE INSTRUCT PROBE ANSWERS ITS QUESTION: voice and instruction-following DO coexist. Option C is de-risked.** `Qwen3-4B` **instruct** (not `-Base`), same corpus/seed/steps so the carrier is the only variable; best checkpoint `checkpoint-150` picked by loss (applying the 4B-Base lesson automatically this time). **Voice installed at full strength — curly quotes 16/18, IDENTICAL to the 4B-Base tuned arm's 16/18**, against the unadapted control's 1/18, and **task-leak 0/18 vs the base carrier's 4/18**. So the assistant prior did NOT block Brontë, which was the central risk. **Instruction-following SURVIVED: 10/10 on-beat through the chat template**, same as the untuned control. ⚠ **The cost is length discipline, not comprehension** — in-band 10/10 → **6/10**, median 124w → 140w. Training on Victorian prose made it wordier, a soft degradation rather than a break. ⚠ **Held-out 2.908 vs 4B-Base's 2.814** — the instruct carrier fits the corpus **0.094 nats worse** and **plateaus without turning** where base overfit at step 75: the assistant prior competes for capacity, so it absorbs less rather than overfitting more.
|
||||
- `[2026-09-11]` ⚠ **What raw-continuation training on an instruct carrier does NOT fix: the plot furniture.** Reading the product artifact, the tuned-instruct arm renders the beat and then drags the referent — *"He licked her clean… my master thus—my husband thus"*, turning the dog into a man, because Brontë's corpus is about masters and husbands. Another beat ran 247w and gave the narrator a list of duties. **This is exactly what instruction-PAIR training is for** — pairs teach "render this and stop", continuation teaches "keep writing Victorian prose". So the probe de-risks option C without substituting for it. ⚠ Also: my `ran_on` metric is uninformative on this job (10/10 on BOTH arms) because a single paragraph contains no blank line — it measures "no paragraph break found", which is correct and useless here. Do not read it as a finding.
|
||||
|
||||
- `[2026-09-11]` **SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and ** → `persistent-memory.d/2026-09-11-skaldsong-s-shape-settles-the-architecture.md`
|
||||
|
||||
- `[2026-09-11]` ⚠ **Stitching has its own failure mode, visible in the booth's Panel C: independently-generated paragraphs drift in POINT OF VIEW.** By beat 4 of 5 the narrator is simultaneously watching the girl carry the animals and carrying them herself ("their weight a strange, heavy secret carried between my ribs"). Each paragraph was generated with no knowledge of the others. **A real stitcher must feed prior paragraphs back as context**, which also means the instruction-pair corpus should include multi-paragraph continuity examples, not just isolated beat→paragraph pairs.
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **THE RECIPE THAT WORKS ON A COMPLETION CARRIER: label the artifact AND begin it.** Operator's prompt: *"This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw-- "*. **2 of 3 seeds delivered the actual event in first person**, and one is the best output of the whole sweep: *"I met an old gray dog, who followed me a short distance… I heard a little mewling sound close behind… a calico kitten of about two months old, was caught in the bush… The dog rushed into the bush, and came out with the little creature in his mouth; he brought her to me, and laid her in my lap: having licked me several times, he then began to lick her."* Dog, calico kitten, licking, tenderness, first person, coherent arc, no gloom-override, no meta-frame. **Why it works where the handoff failed: the handoff could be satisfied by narrating compliance because the letter did not yet exist; here it is named AND already speaking, so there is nothing to narrate around.** Also learned the Gutenberg `_underscore italics_` convention. 1 of 3 drifts.
|
||||
- `[2026-09-11]` ⚠ **My typography hypothesis was WRONG, and the chapter-heading result is the evidence.** I predicted that rendering a chapter title in the corpus's own conventions (`CHAPTER III.` / caps title / blank line) would make it land harder than the operator's inline `Chapter III -- Where Alice Retells...`. **It did the opposite**: both corpus-form seeds ignored the title entirely and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter *about* the story (a gentleman disputing the premise). Likely reason: corpus chapter titles are short and decorative (`THE CHILD'S CLOSET`), so a long descriptive one in that slot reads as decoration to skip, whereas inline it reads as text to continue. **A label only instructs if the model treats that slot as load-bearing.**
|
||||
- `[2026-09-11]` ⚠ **Unnoticed consequence of the D2/D3 rename pipeline: the adapter SUBSTITUTES proper nouns it was never trained on.** Given "Alice" in a chapter title it produced *"ALEXANDER THE ALEXANDER, AS HE WAS KNOWN IN LITTLE LONDON"*. The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Consequence for use: **you cannot reliably name your own characters at prompt time** — they may be renamed mid-passage. Not a defect of the rename (which exists to prevent memorisation of Brontë's cast) but a real usability constraint that needs stating.
|
||||
|
||||
- `[2026-09-11]` **4B arms RE-CUT from `checkpoint-75`, the true loss minimum (2.813826, confirmed from `loss-series.json` rather than my reading of the log); booth rebuilt.** Only the tuned arms needed it — the base arm never touches the adapter. ⚠ **A small surprise: step-75 and end-of-run differ on typography, not voice.** Curly quotes 16/18 vs 17/18 and collapse 0/18 either way, but the **hard-wrap ratio is 0.33 at step-75 against 0.12 at end-of-run** — further training washes the residual line-break habit out *while held-out loss gets worse*. So **"best loss" and "best typography" are different checkpoints**; neither is near the original 0.85 defect, and the corpus's own residual (preserved verse) is 0.25.
|
||||
- `[2026-09-11]` **EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about so** → `persistent-memory.d/2026-09-11-embedding-an-instruction-inside-the-fiction.md`
|
||||
|
||||
- `[2026-09-11]` **R49 SWEEP COMPLETE — 4B closes the continuity gap, and the carrier ladder is clean: 3.329 → 3.018 → 2.814 held-out** (0.6B / 1.7B / 4B, all on the same unwrapped corpus sha `77f37057b2782e49`, seed 4919, 159 steps, 5,210,112 tokens — carrier size the only variable). Deltas **0.311 then 0.204**: diminishing but still real. Booth: `http://10.100.10.50:8090/b/babybronte-4b/`. **4B tuned has the best voice saturation of any rung — curly quotes 17/18 against its own base arm's 1/18, collapse 0/18 against 4/18** — and, the thing the rung existed to test, **scene-level continuity HOLDS**: it produces a named character with motivated dialogue, a navigable spatial layout and a physical description in one passage, where 1.7B wrote pretty but eventless prose (opening doors, looking at stars). On the letter prompt it opens the letter, promises to quote it, and then actually quotes it across a paragraph break.
|
||||
- `[2026-09-11]` ⚠⚠ **4B is the FIRST rung to OVERFIT inside one epoch, which inverts my earlier "one epoch is right for this corpus" call.** Series 2.832 · 2.816 · **2.814** · 2.820 · 2.824 · 2.825 · 2.825 — minimum at ~step 75, then it TURNS and settles worse. 0.6B and 1.7B both plateaued with no turn, so **the optimal epoch count shrinks as the carrier grows** — 4B wants roughly half an epoch. ⚠ **Consequence: the shipped `adapter/` at `h02-4b-1ep/` is NOT the best checkpoint** (it is the end-of-run 2.825); the step-75 checkpoint at 2.814 is, and it exists only because `save_steps=25` was set. The voice test used the end-of-run adapter, so the booth understates 4B by ~0.011 nats. Re-cut the arms off the step-75 checkpoint before any adjudication.
|
||||
- `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established.
|
||||
|
||||
- `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md`
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
@@ -457,7 +397,7 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
_70 older entries archived to archival-memory.md._
|
||||
_104 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -475,5 +415,4 @@ _70 older entries archived to archival-memory.md._
|
||||
|
||||
- `[2026-09-15]` **Remote-site MASQUERADE rules on nh3-scale** for the asymmetric-return theory — they fired (counters incremented) but were not the fix. Reverted rather than left to accumulate as NAT achieving nothing.
|
||||
|
||||
|
||||
_115 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user