vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — lv-mccarthy through D3 on gx10, SFT pairs next (a mccarthy register must be written first)

Browse Source
This commit is contained in:
vh
2026-09-17 10:46:44 -07:00
parent 5ddb0472e4
commit 4dce0d0a43
3 changed files with 227 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+64 -84
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-17 ~08:45 PT (lv-hemingway SHIPPED; floor rule pairwise, which retroactively passes lv-bronte; lv-mccarthy D1 built and proceeding; lv-krakauer PARKED — research is not a voice.)_
_Last updated: 2026-09-17 ~09:00 PT (lv-mccarthy through D3 on gx10 — leak gate PASSED, val split larger than Hemingway's; SFT pairs are next and need a `mccarthy` register written first. lv-krakauer parked.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,99 +115,84 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-17 ~01:30 PT._
_As of 2026-09-17 ~09:00 PT._
### SHIPPED — `lv-hemingway` on ckpt850, the line's first clean voice pass
### NEXT: build the SFT pairs for `lv-mccarthy`
Live on `vllm-voices` (fv-ml1 GPU 0 :8027) beside `voices-base`, `lv-yarros`, `lv-bronte`.
**VOICE +0.413 delta_cb at 6.4x its floor** — closes 73.8% of the achievable span, and clears
the OLD all-arms floor too, so the verdict does not lean on the rule change. **DAMAGE clean.**
⚠ **MEMORISATION is the axis to read**: 0.07 hit-rate against **held-out Hemingway's own
0.01** — ~7x the author's self-collision rate — but all 19 matched runs were read and every
one is stock dialogue capped at **9 words**, no proper noun, no plot. Elevated rate, zero
protectable content; in-copyright author, so the fair-use call is the operator's.
Everything upstream is done and gated. `~/lv-mccarthy/` on **pfi-gx10** holds `corpus-clean/`
(167 units, 584,716 words, entities-final.json), `corpus-renamed/` (6 copies, 1,002 records,
leak gate PASSED) and `scripts/`.
The pairs step, mirroring the Hemingway run:
```
gx10:~/lv-mccarthy
scripts/yarros-corpus/build_sft_pairs.py \
--corpus corpus-renamed/copies --split train --copy copy0 \
--register <NEW: no mccarthy register exists yet> \
--source-entities corpus-clean/entities-final.json \
--context-frac 1.0 --drop-leading-heading --out pairs/pairs-full.jsonl
```
⚠ **`--register mccarthy` DOES NOT EXIST YET** — `build_sft_pairs.py`'s REGISTERS map has
yarros/bronte/hemingway only. It must be written before the run: it is the system prompt the
beat-generator uses AND the one the trainer and the eval are bound to.
⚠ **Pass `--source-entities`** — it exists now and Hemingway's pairs predate it. This is the
beat-contamination filter; running without it is how 70 of 7,094 Hemingway beats ended up
naming Santiago, Catherine and Rinaldi.
⚠ Pairs generation runs against the `gen` seat through LiteLLM (`10.250.50.70:4000`), took
**2,759 s (~46 min)** for Hemingway's 7,094 pairs, and the seat was confirmed up.
Then `--split val` for the val pairs, then `train_pairs_lora.py`.
### SHIPPED — `lv-hemingway` on ckpt850, live on `vllm-voices` (fv-ml1 GPU 0 :8027)
Beside `voices-base`, `lv-yarros`, `lv-bronte`. VOICE +0.413 delta_cb at 6.4x its floor,
closing 73.8% of the achievable span — the strongest in the line, and it clears the OLD
all-arms floor too, so it does not lean on the rule change.
⚠ MEMORISATION is the axis to read: 0.07 hit-rate against **held-out Hemingway's own 0.01**,
so ~7x the author's self-collision rate — but all 19 matched runs were READ and every one is
stock dialogue capped at 9 words, no proper noun, no plot.
→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
⚠ **ckpt850, NOT the loss minimum at 1750** — the two are indistinguishable on voice (0.072
gap vs a 0.113 floor) so the tiebreak fell to the resolving axes, and 850 wins all of them
(2.3x tighter spread, lower memorisation, less ran-on, half an epoch less overfit).
**The two-epoch recipe is now 0 for 2.** Read the curve; prefer the earlier tied checkpoint.
### BUILT — `lv-mccarthy` and `lv-krakauer` D1 corpora (operator: "both", 2026-09-17)
| corpus | units | words | notes |
|---|---|---|---|
| **mccarthy** | 167 | **584,756** | `~/mccarthy-corpus`, 6 novels, both guards PASS |
| **krakauer** | 126 | **422,880** | `~/krakauer-corpus`, 4 works, ALL NON-FICTION |
Shared blocker solved first: `scripts/r49-corpus/split_units.py` picks a unit mode by SIZE
not count (the inherited "most units" rule gave Cities of the Plain 4 units of 22,312w — the
book's PARTS), and falls back to paragraph-block sections for works with no divisions.
→ `persistent-memory.d/2026-09-17-mccarthy-krakauer-d1.md`
⚠ **McCarthy's voice IS missing punctuation** — 0.0 quote marks/10k against Hemingway's 838,
and `dont`/`aint`/`wont`. `repair_typography.py` MUST NOT be run on that corpus; the builder
runs no normalisation and asserts the quote density so a future change fails the build.
⚠ **And delta_cb is a char-bigram metric**, so an adapter that learns only "emit no quotes"
will pass the voice axis without learning a sentence. **Pre-register a punctuation-normalised
secondary read before gating lv-mccarthy.**
### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17). henge id **82**
### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17), henge id **82**
*"he's a great writer because of his research, not because he has a strong identifiable
voice."* ⭐ **The reason is about the AUTHOR, not the data, and it reframes the selection
criterion for the whole line:** ask whether there IS a voice worth adapting before any
corpus work. That question was never asked here — corpus mechanics were investigated first.
→ auto-memory `feedback_voice_worth_adapting_before_corpus`.
voice."* Corpus (126 units, 422,880 words) and builder stay committed and re-runnable.
The corpus (126 units, 422,880 words) and its builder stay committed and re-runnable. What
would unpark it is a re-extraction preserving indentation and italics, which fixes the
unmeasurable-quoted-material problem but **not** the operator's objection.
### OPEN for the operator
### OPEN for the operator — two measured defects, neither acted on
1. **The althing route-declaring SessionStart hook is NOT installed on nh3-dev**, so any
relaunched pane silently drops to pull mode and stops receiving mail. althing's own skill
documents the hook; this box does not have it. Installing it means editing
`~/.claude/settings.json`. Raised, not answered.
2. **Hemingway's train beats: 70 of 7,094 (0.96%) name characters the rename removed.**
`audit_pairs_sourcenames.py --filter-out` yields a verified-clean 7,024-pair set in one
command; retrain ~3h17 + re-gate ~1h40. Operator said **ship stands** — recorded, not acted on.
3. **130 of 946 Hemingway entity-map surfaces are probably not names** and were renamed anyway.
Same ruling: ship stands.
1. **Hemingway's TRAIN beats carry the source-name leak: 70 of 7,094 (0.96%)** (Santiago x16,
Catherine x7, Rinaldi x3 ...), responses 0 of 7,294, **val 0 of 200 so the gate itself is
unconfounded**. `audit_pairs_sourcenames.py --filter-out` yields a verified-clean
7,024-pair set in one command. **Retrain ~3h17 + re-gate ~1h40, unattended.**
2. **130 of 946 entity-map surfaces are probably not names** and were renamed anyway
(`African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado`), plus **16 bare initials
incl. `C` at 274 occurrences** — 1,616 instances, 0.162% of corpus words.
`audit_entity_map.py` finds them; they need READING, not auto-removal, because
`the Widow` / `the Informer` are genuine epithet-names that should be renamed.
### ESH is on Verizon failover — Cityside Fiber failed twice on 2026-09-16
Both would be fixed in one pass if a corpus rebuild happens. Neither blocks anything today.
ESH egress `97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both
with 7-day expiries set 2026-09-16. ⚠ When those expire or the CGNAT egress rotates, ESH loses
colo access. Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark.
### ESH is on Verizon failover — Cityside Fiber failed TWICE tonight
### Older deferred set, unchanged
19:09–20:15 and again from ~01:06. The operator switched WAN1 to DHCP to get service back at all
and has a ticket in to restore the static `128.177.138.182/30`. ESH egress is currently
`97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both with 7-day
expiries set 2026-09-16 ~19:31 and ~01:xx. ⚠ **When those expire, or when the CGNAT egress
rotates, ESH loses colo access** — that is the false-ban class that blackholed the site before.
Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark.
AI-tab Dormant regrouping (**belayed**), `nconnect=8` on `/mnt/smithy` (**declined in scope**),
fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, id 75**).
### OPEN, operator's call, nothing blocked
- **Henge id 79** (`catalog-all-robotics-bits-and-bobs`) was filed under source
`eshpfi-management` by the skill's repo inference; it is a personal idea. Re-park with
`--personal` if wanted.
- **xAI billing** — did the stopped HTTP A/B bill on top of the coding plan? Needs the
operator's console; this fleet holds no xAI credential.
- **A `BabyYarros → lv-yarros` pointer** in memory so historical entries stay findable.
- Older deferred set, unchanged: AI-tab Dormant regrouping (**belayed**), `nconnect=8` on
`/mnt/smithy` (**declined in scope**), fused MoE kernel path (**parked, id 47**), TTS-stack
move to fv-ml1 (**parked, id 75**).
⚠ **fv-ml1 GPU 0 is at 96.0 of 97.9 GB.** Adding `lv-bronte` cost nothing measurable (96092 →
96090 MiB) because a LoRA rides inside the existing seat — but a new SEAT still needs a
placement decision. GPU 3 is a HELD RESERVE.
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified
before this session began. ⚠ **Do NOT commit them** — untouched and deliberately left alone.
⚠ **Uncommitted and NOT mine:** `graphify-out/GRAPH_REPORT.md`, `scripts/seat-inventory.py`,
`stacks/searxng/{README.md,conf/searxng-settings.yml}`, `configs/esh-scale/`. Leave them alone.
## Recent decisions
- `[2026-09-17]` ⭐⭐ **A unit splitter must choose by SIZE, not count — and the val split scales with WORK COUNT, not corpus size.** `scripts/r49-corpus/split_units.py` + a multi-index `--holdout-chapter`. The inherited most-units rule gave Cities of the Plain 4 units of 22,312w (the book's PARTS); the single-index holdout would have given McCarthy a Brontë-class 18k-word val reference on a 588k corpus. Both fixed, both caught by controls. → `persistent-memory.d/2026-09-17-mccarthy-d1-d3.md`
- `[2026-09-17]` ⭐ **lv-mccarthy D1–D3 complete on gx10, leak gate PASSED (0 of 75 renameable, 0 of 37 sub-threshold, both controls green).** Three McCarthy-specific calls, each forced by a measurement: corpus-scoped rename (the Border Trilogy shares 9 surfaces across books), a new `mccarthy` name preset (Hemingway's carries it_IT/fr_FR and McCarthy writes neither), and `--min-cap 5` to match the entity map's floor — the first gate run failed with 45 survivors purely because rename's floor was 8 and the map's was 5. → `persistent-memory.d/2026-09-17-mccarthy-d1-d3.md`
- `[2026-09-17]` ⭐⭐ **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.** Operator: *"he's a great writer because of his research, not because he has a strong identifiable voice."* I surveyed, built the corpus, measured containment and fixed three stripping defects before anyone asked the question that decided it. henge id 82. → auto-memory `feedback_voice_worth_adapting_before_corpus`.
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared at a different minute. A relaunched pane therefore drops to pull mode and stops receiving mail. Raised with the operator, not acted on, because installing it edits `~/.claude/settings.json` and a peer's question is the wrong authorisation for a config change. Tracked at althing thread `01M2R0KPPE85SVJ96KSQ3YKKQP`.
- `[2026-09-17]` **Hemingway ships as-is: operator ruled "ship stands" on both measured corpus defects** — the 0.96% beat contamination and the 130 non-name entity-map surfaces. `audit_pairs_sourcenames.py --filter-out` and `audit_entity_map.py` exist and are the instruments if that is ever revisited; neither was run against the shipped adapter.
- `[2026-09-17]` ⭐⭐⭐ **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.** Operator: *"he's a great writer because of his research, not because he has a strong identifiable voice."* A voice adapter is worth its ~6 hours only if the target has a prose signature a reader could pick out blind; Krakauer's excellence is reporting, which an adapter cannot carry. I surveyed, built, measured containment and fixed three stripping defects before anyone asked the question that decided it. **For each candidate, say what the voice IS in one sentence and how it shows up in char-bigram space, before the first catalogue query.** henge id 82. → auto-memory `feedback_voice_worth_adapting_before_corpus`.
- `[2026-09-17]` ⭐⭐ **A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters".** `scripts/r49-corpus/split_units.py`: a marker mode qualifies only if its median unit is inside [600, 12000] AND no unit holds half the work; among qualifying modes PRIORITY breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title), and paragraph-block sections are the fallback for works with no divisions. ⭐ **Both rules exist because a control caught them**: scoring by "median closest to target" chose `caps-title` (6 units, one holding **97%** of the book) over True at First Light's real 20 chapters, because a median cannot see that distribution and a max bound can. Positive control: 8/10 Hemingway works reproduce the shipped mode and count exactly. Negative control: 40,000 words with no blank lines → 1 unit, refuses to fabricate divisions. Commit `705fa3a`.
@@ -456,11 +441,6 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md`
- `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
@@ -469,7 +449,7 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
_10 older entries archived to archival-memory.md._
_11 older entries archived to archival-memory.md._
_5 older entries archived to archival-memory.md._