memory: snapshot — lv-mccarthy through D3 on gx10, SFT pairs next (a mccarthy register must be written first)
This commit is contained in:
@@ -4,6 +4,10 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
- `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
|
||||
_Archived 2026-09-17._
|
||||
|
||||
|
||||
- `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
|
||||
_Archived 2026-09-17._
|
||||
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
# `[2026-09-17]` lv-mccarthy D1→D3 — built, gated, and every stage caught a defect in the stage before it
|
||||
|
||||
**`~/lv-mccarthy/` on pfi-gx10.** `corpus-clean/` (167 units, 584,716 words),
|
||||
`corpus-renamed/` (6 copies, 1,002 records), `scripts/`. Commits `705fa3a` `f3bf3ca`
|
||||
`0fa68cb` `5aa10bf` `5ddb047`.
|
||||
|
||||
```
|
||||
leak gate 0 of 75 renameable and 0 of 37 sub-threshold survive in any copy
|
||||
positive control 108/108 surfaces found in the unrenamed source
|
||||
negative control nonce absent from both trees
|
||||
```
|
||||
|
||||
## The shared splitter: choose by SIZE, not by count
|
||||
|
||||
`scripts/r49-corpus/split_units.py`. The inherited rule was "most units above a floor", which
|
||||
is wrong for any book whose markers are PARTS:
|
||||
|
||||
```
|
||||
Cities of the Plain 4 roman marks -> 4 units, median 22,312w
|
||||
The Crossing 4 roman marks -> 4 units, median 37,310w
|
||||
```
|
||||
|
||||
Four beats one, so it won, and the old guard only fired at exactly one unit. Now: a mode
|
||||
qualifies only if its median unit is inside **[600, 12000]** AND no unit holds half the work;
|
||||
among qualifying modes **priority** breaks the tie (contents > chapter-word > roman >
|
||||
bare-numeral > caps-title). Works with no divisions fall back to **paragraph-block sections**.
|
||||
|
||||
⭐⭐ **The first version of that rule was WORSE than what it replaced, and a control caught
|
||||
it.** Scoring by "median closest to target" chose `caps-title` over the real chapters of
|
||||
Hemingway's *True at First Light*:
|
||||
|
||||
```
|
||||
bare-numeral 20 units median 5,337w max 11,155 <- the book's own chapters
|
||||
caps-title 6 units median 777w max 113,886 <- median looked BETTER
|
||||
```
|
||||
|
||||
Five stray all-caps lines gave five tiny units beside **one holding 97% of the book**. A median
|
||||
cannot see that distribution; a max bound can. Controls green both ways afterwards: 8/10
|
||||
Hemingway works reproduce the shipped mode and count exactly, and 40,000 words with no blank
|
||||
lines returns **1 unit** rather than fabricating sections.
|
||||
|
||||
⚠ The Hemingway builder is deliberately NOT repointed at this module — its corpus is shipped
|
||||
and its sha is pinned by a live adapter.
|
||||
|
||||
## D1: the job was protecting a style that reads as damage
|
||||
|
||||
```
|
||||
quote marks 0.0 per 10k (Hemingway 838)
|
||||
apostrophes 123 per 10k (Hemingway 241) `dont` `aint` `wont` `didnt`
|
||||
```
|
||||
|
||||
⚠⚠ **`repair_typography.py` MUST NOT be run on this corpus.** It normalises "toward what the
|
||||
text does" and would put the quotation marks back. The builder runs no normalisation and then
|
||||
**asserts** the quote density, so a future well-meaning change fails the build.
|
||||
|
||||
⚠ **AND IT MAKES THE VOICE GATE EASY TO PASS FOR THE WRONG REASON.** `voice_distance.py` is
|
||||
Burrows's Delta over CHARACTER BIGRAMS. An adapter that learns only "emit no quotation marks"
|
||||
moves delta_cb a long way without having learned a sentence. **Pre-register a
|
||||
punctuation-normalised secondary read before gating lv-mccarthy.** Tracked in the builder
|
||||
docstring, commit `f3bf3ca`.
|
||||
|
||||
Also: two truncated catalogue rows dropped for complete mobi siblings; all 15 cross-work
|
||||
containment pairs measured (worst **0.10%**); back matter in 4 of 6 works carrying the author's
|
||||
name 26 times → **0**; alphabet re-derived at 1,411 non-ASCII letters across 14 Spanish forms.
|
||||
|
||||
⚠ The back-matter strip runs **BEFORE** the split for McCarthy, inverting the Hemingway order:
|
||||
Blood Meridian and The Crossing end with a dumped table of contents made of bare roman numerals
|
||||
on their own lines — the exact shape of a chapter marker.
|
||||
|
||||
## D2 caught a D1 defect: three small-caps manglings
|
||||
|
||||
The entity map returned `E`, `H`, `T`, `K` as renameable entities with 17–33 capitalised
|
||||
occurrences each — the `G` class from Hemingway, where `G` was about to be renamed to a surname
|
||||
248 times. Reading them showed the extractor mangled small-caps openings three ways:
|
||||
|
||||
```
|
||||
1. SPLIT INITIAL `T HE HOUSE was built` -> `The house was built` 32 cases
|
||||
2. UNMARKED RUN `THEY STOOD in the doorway` -> `They stood in the doorway` 88 cases
|
||||
3. LOST INITIAL `HE CANDLEFLAME` -> `THE CANDLEFLAME` 1 case
|
||||
```
|
||||
|
||||
Rule 1 requires a FOLLOWING all-caps word, so `A TV was playing` and `A Mexican was changing`
|
||||
are untouched. Rule 2's `[a-z]` lookahead is what makes it safe — a genuine shout or sign is
|
||||
not followed mid-sentence by lowercase. All 23 distinct first words of the 88 were checked.
|
||||
|
||||
⚠⚠ **A fourth "fix" was nearly shipped that would have CORRUPTED the text.** `HEY RODE` →
|
||||
`THEY RODE` looked right from a survey of the BUILT corpus. The raw master has `THEY RODE`
|
||||
intact, twice — `HEY RODE` matched as a SUBSTRING, and the unanchored replace produced
|
||||
`TTHEY RODE`, which rule 2 then lowercased to `Tthey rode`. Caught by the count assertion
|
||||
(expected 1, replaced 2) and settled by reading the master. ⚠ My first corruption check also
|
||||
missed it, searching for `TTHEY` when the pipeline had already lowercased it — **check the
|
||||
shape the pipeline emits, not the shape you imagined.**
|
||||
|
||||
## D2's own gates: and `audit_stoplist` was scanning its own rationale
|
||||
|
||||
⚠⚠ **A defect in `audit_stoplist.py`, latent for every corpus before this one.** It built its
|
||||
surface set from every list value in the stoplist JSON — including `_why`, which by convention
|
||||
is a LIST OF PROSE LINES. Its empty separator line matched the honorific pattern **139 times**,
|
||||
printing a flag with no surface name above the one real catch. Now skips `_`-prefixed keys.
|
||||
|
||||
That real catch was a contradiction **inside my own file**: `Franklin` sat in the geography list
|
||||
(the old name for El Paso) while the same file's note recorded *"I'm here to see Mr Franklin"*,
|
||||
a lawyer in All the Pretty Horses. A second self-inflicted one: a speculative A–Z fragments list
|
||||
stoplisted `I` and `A`, and `Sir I dont think I can do that` duly tripped the audit. It is now
|
||||
the four letters actually measured as entities.
|
||||
|
||||
Everything ambiguous was read in context: **Socorro is the ranch cook, not the New Mexico
|
||||
town**; Niño, Keno and Redbo are HORSES (renameable, the `Inglés` precedent); Yaqui and Gilenos
|
||||
are real peoples; Hashknives is a real cattle outfit; Hearst, Trias, Huerta and Madero are real
|
||||
historical figures on the page under their own names.
|
||||
|
||||
Final: 123 map surfaces, 124-surface stoplist, `entities.py` 27/27 controls, both audits PASS.
|
||||
|
||||
## The human gender pass is an auditable file
|
||||
|
||||
The honorific/window resolver scored **21 correct / 3 held / 1 WRONG** against a 26-name
|
||||
control; the base-rate proximity resolver built for Hemingway scored 18/6/1 and **its own guard
|
||||
correctly REFUSED to write**. So the incumbent stands and four entries are fixed by hand in
|
||||
`gender_overrides_mccarthy.json`, each carrying its evidence.
|
||||
|
||||
⚠ All four are female and all four look male-dominated in raw counts, because this corpus runs
|
||||
**29,144 male pronouns to 5,036 female — a base rate of 85.3% male**. Carla Jean Moss at
|
||||
31m/21f would be 44m/8f at that rate; 21 against an expected 8 is decisive. Same arithmetic that
|
||||
recovered Pilar and Brett on Hemingway. Alfonsa was in the control and is correctly absent from
|
||||
the map at 4 occurrences, below the min-count floor — an error in the control, not the pipeline.
|
||||
|
||||
`apply_gender_overrides.py` refuses twice: a name absent from the map is an error rather than a
|
||||
silent no-op, and overruling a gender the detector holds needs an explicit `"correcting": true`
|
||||
so it cannot look like filling a held entity in a diff.
|
||||
|
||||
## D3: three calls, and the holdout fix that matters most
|
||||
|
||||
1. **`--scope corpus`**, not the per-work default. Nine surfaces appear in more than one work —
|
||||
Parham (The Crossing + Cities of the Plain), Grady and Cole (All the Pretty Horses + Cities
|
||||
of the Plain), Socorro, Héctor. A per-work map gives John Grady a different invented name in
|
||||
each novel, turning one character into two.
|
||||
2. **A new `mccarthy` preset.** Hemingway's romance pool carries `it_IT` and `fr_FR` for his
|
||||
Italian and French casts; McCarthy writes neither language. `en_GB` goes for the same reason.
|
||||
`en_US` + `es_MX`/`es_ES` at an even share.
|
||||
3. **`--min-cap 5` to match the entity map's floor.** The first gate run FAILED with 45
|
||||
survivors: `entities.py` admits cap ≥ 5 while `rename.py` renamed only cap ≥ 8, so every
|
||||
entity between sat in the map, was never renamed, and counted as a leak. Hemingway never hit
|
||||
it because its map had `sub_threshold_total: 0`.
|
||||
|
||||
⭐ **`--holdout-chapter` NOW TAKES A LIST.** The val split is one chapter index per work, so its
|
||||
SIZE is set by how many WORKS a corpus has, not how many words:
|
||||
|
||||
```
|
||||
Hemingway 10 works -> 9 val units -> 36,563 words/copy -> gate DECISIVE
|
||||
Brontë 4 works -> 4 val units -> 17,043 words/copy -> gate MARGINAL
|
||||
McCarthy 6 works -> 6 val units -> ~18,000 would have been Brontë's end
|
||||
```
|
||||
|
||||
Holding out chapters **7 and 17** gives **11 units and 40,653 words per copy — larger than
|
||||
Hemingway's** — for 7% of the corpus, on a corpus 40% smaller than his. No amount of corpus size
|
||||
fixes a val split that scales with work count.
|
||||
|
||||
Related: [[2026-09-17-lv-hemingway-gate]], [[2026-09-17-mccarthy-krakauer-d1]],
|
||||
[[2026-09-17-lv-bronte-gate]].
|
||||
+64
-84
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-17 ~08:45 PT (lv-hemingway SHIPPED; floor rule pairwise, which retroactively passes lv-bronte; lv-mccarthy D1 built and proceeding; lv-krakauer PARKED — research is not a voice.)_
|
||||
_Last updated: 2026-09-17 ~09:00 PT (lv-mccarthy through D3 on gx10 — leak gate PASSED, val split larger than Hemingway's; SFT pairs are next and need a `mccarthy` register written first. lv-krakauer parked.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,99 +115,84 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-17 ~01:30 PT._
|
||||
_As of 2026-09-17 ~09:00 PT._
|
||||
|
||||
### SHIPPED — `lv-hemingway` on ckpt850, the line's first clean voice pass
|
||||
### NEXT: build the SFT pairs for `lv-mccarthy`
|
||||
|
||||
Live on `vllm-voices` (fv-ml1 GPU 0 :8027) beside `voices-base`, `lv-yarros`, `lv-bronte`.
|
||||
**VOICE +0.413 delta_cb at 6.4x its floor** — closes 73.8% of the achievable span, and clears
|
||||
the OLD all-arms floor too, so the verdict does not lean on the rule change. **DAMAGE clean.**
|
||||
⚠ **MEMORISATION is the axis to read**: 0.07 hit-rate against **held-out Hemingway's own
|
||||
0.01** — ~7x the author's self-collision rate — but all 19 matched runs were read and every
|
||||
one is stock dialogue capped at **9 words**, no proper noun, no plot. Elevated rate, zero
|
||||
protectable content; in-copyright author, so the fair-use call is the operator's.
|
||||
Everything upstream is done and gated. `~/lv-mccarthy/` on **pfi-gx10** holds `corpus-clean/`
|
||||
(167 units, 584,716 words, entities-final.json), `corpus-renamed/` (6 copies, 1,002 records,
|
||||
leak gate PASSED) and `scripts/`.
|
||||
|
||||
The pairs step, mirroring the Hemingway run:
|
||||
|
||||
```
|
||||
gx10:~/lv-mccarthy
|
||||
scripts/yarros-corpus/build_sft_pairs.py \
|
||||
--corpus corpus-renamed/copies --split train --copy copy0 \
|
||||
--register <NEW: no mccarthy register exists yet> \
|
||||
--source-entities corpus-clean/entities-final.json \
|
||||
--context-frac 1.0 --drop-leading-heading --out pairs/pairs-full.jsonl
|
||||
```
|
||||
|
||||
⚠ **`--register mccarthy` DOES NOT EXIST YET** — `build_sft_pairs.py`'s REGISTERS map has
|
||||
yarros/bronte/hemingway only. It must be written before the run: it is the system prompt the
|
||||
beat-generator uses AND the one the trainer and the eval are bound to.
|
||||
⚠ **Pass `--source-entities`** — it exists now and Hemingway's pairs predate it. This is the
|
||||
beat-contamination filter; running without it is how 70 of 7,094 Hemingway beats ended up
|
||||
naming Santiago, Catherine and Rinaldi.
|
||||
⚠ Pairs generation runs against the `gen` seat through LiteLLM (`10.250.50.70:4000`), took
|
||||
**2,759 s (~46 min)** for Hemingway's 7,094 pairs, and the seat was confirmed up.
|
||||
Then `--split val` for the val pairs, then `train_pairs_lora.py`.
|
||||
|
||||
### SHIPPED — `lv-hemingway` on ckpt850, live on `vllm-voices` (fv-ml1 GPU 0 :8027)
|
||||
|
||||
Beside `voices-base`, `lv-yarros`, `lv-bronte`. VOICE +0.413 delta_cb at 6.4x its floor,
|
||||
closing 73.8% of the achievable span — the strongest in the line, and it clears the OLD
|
||||
all-arms floor too, so it does not lean on the rule change.
|
||||
⚠ MEMORISATION is the axis to read: 0.07 hit-rate against **held-out Hemingway's own 0.01**,
|
||||
so ~7x the author's self-collision rate — but all 19 matched runs were READ and every one is
|
||||
stock dialogue capped at 9 words, no proper noun, no plot.
|
||||
→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
|
||||
|
||||
⚠ **ckpt850, NOT the loss minimum at 1750** — the two are indistinguishable on voice (0.072
|
||||
gap vs a 0.113 floor) so the tiebreak fell to the resolving axes, and 850 wins all of them
|
||||
(2.3x tighter spread, lower memorisation, less ran-on, half an epoch less overfit).
|
||||
**The two-epoch recipe is now 0 for 2.** Read the curve; prefer the earlier tied checkpoint.
|
||||
|
||||
### BUILT — `lv-mccarthy` and `lv-krakauer` D1 corpora (operator: "both", 2026-09-17)
|
||||
|
||||
| corpus | units | words | notes |
|
||||
|---|---|---|---|
|
||||
| **mccarthy** | 167 | **584,756** | `~/mccarthy-corpus`, 6 novels, both guards PASS |
|
||||
| **krakauer** | 126 | **422,880** | `~/krakauer-corpus`, 4 works, ALL NON-FICTION |
|
||||
|
||||
Shared blocker solved first: `scripts/r49-corpus/split_units.py` picks a unit mode by SIZE
|
||||
not count (the inherited "most units" rule gave Cities of the Plain 4 units of 22,312w — the
|
||||
book's PARTS), and falls back to paragraph-block sections for works with no divisions.
|
||||
→ `persistent-memory.d/2026-09-17-mccarthy-krakauer-d1.md`
|
||||
|
||||
⚠ **McCarthy's voice IS missing punctuation** — 0.0 quote marks/10k against Hemingway's 838,
|
||||
and `dont`/`aint`/`wont`. `repair_typography.py` MUST NOT be run on that corpus; the builder
|
||||
runs no normalisation and asserts the quote density so a future change fails the build.
|
||||
⚠ **And delta_cb is a char-bigram metric**, so an adapter that learns only "emit no quotes"
|
||||
will pass the voice axis without learning a sentence. **Pre-register a punctuation-normalised
|
||||
secondary read before gating lv-mccarthy.**
|
||||
|
||||
### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17). henge id **82**
|
||||
### ⛔ PARKED — `lv-krakauer` (operator, 2026-09-17), henge id **82**
|
||||
|
||||
*"he's a great writer because of his research, not because he has a strong identifiable
|
||||
voice."* ⭐ **The reason is about the AUTHOR, not the data, and it reframes the selection
|
||||
criterion for the whole line:** ask whether there IS a voice worth adapting before any
|
||||
corpus work. That question was never asked here — corpus mechanics were investigated first.
|
||||
→ auto-memory `feedback_voice_worth_adapting_before_corpus`.
|
||||
voice."* Corpus (126 units, 422,880 words) and builder stay committed and re-runnable.
|
||||
|
||||
The corpus (126 units, 422,880 words) and its builder stay committed and re-runnable. What
|
||||
would unpark it is a re-extraction preserving indentation and italics, which fixes the
|
||||
unmeasurable-quoted-material problem but **not** the operator's objection.
|
||||
### OPEN for the operator
|
||||
|
||||
### OPEN for the operator — two measured defects, neither acted on
|
||||
1. **The althing route-declaring SessionStart hook is NOT installed on nh3-dev**, so any
|
||||
relaunched pane silently drops to pull mode and stops receiving mail. althing's own skill
|
||||
documents the hook; this box does not have it. Installing it means editing
|
||||
`~/.claude/settings.json`. Raised, not answered.
|
||||
2. **Hemingway's train beats: 70 of 7,094 (0.96%) name characters the rename removed.**
|
||||
`audit_pairs_sourcenames.py --filter-out` yields a verified-clean 7,024-pair set in one
|
||||
command; retrain ~3h17 + re-gate ~1h40. Operator said **ship stands** — recorded, not acted on.
|
||||
3. **130 of 946 Hemingway entity-map surfaces are probably not names** and were renamed anyway.
|
||||
Same ruling: ship stands.
|
||||
|
||||
1. **Hemingway's TRAIN beats carry the source-name leak: 70 of 7,094 (0.96%)** (Santiago x16,
|
||||
Catherine x7, Rinaldi x3 ...), responses 0 of 7,294, **val 0 of 200 so the gate itself is
|
||||
unconfounded**. `audit_pairs_sourcenames.py --filter-out` yields a verified-clean
|
||||
7,024-pair set in one command. **Retrain ~3h17 + re-gate ~1h40, unattended.**
|
||||
2. **130 of 946 entity-map surfaces are probably not names** and were renamed anyway
|
||||
(`African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado`), plus **16 bare initials
|
||||
incl. `C` at 274 occurrences** — 1,616 instances, 0.162% of corpus words.
|
||||
`audit_entity_map.py` finds them; they need READING, not auto-removal, because
|
||||
`the Widow` / `the Informer` are genuine epithet-names that should be renamed.
|
||||
### ESH is on Verizon failover — Cityside Fiber failed twice on 2026-09-16
|
||||
|
||||
Both would be fixed in one pass if a corpus rebuild happens. Neither blocks anything today.
|
||||
ESH egress `97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both
|
||||
with 7-day expiries set 2026-09-16. ⚠ When those expire or the CGNAT egress rotates, ESH loses
|
||||
colo access. Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark.
|
||||
|
||||
### ESH is on Verizon failover — Cityside Fiber failed TWICE tonight
|
||||
### Older deferred set, unchanged
|
||||
|
||||
19:09–20:15 and again from ~01:06. The operator switched WAN1 to DHCP to get service back at all
|
||||
and has a ticket in to restore the static `128.177.138.182/30`. ESH egress is currently
|
||||
`97.190.18.88`; crowdsec's `esh` allowlist carries it plus `23.164.40.174`, both with 7-day
|
||||
expiries set 2026-09-16 ~19:31 and ~01:xx. ⚠ **When those expire, or when the CGNAT egress
|
||||
rotates, ESH loses colo access** — that is the false-ban class that blackholed the site before.
|
||||
Check `curl -s4 ifconfig.me` from esh-docker-vm first if ESH goes dark.
|
||||
AI-tab Dormant regrouping (**belayed**), `nconnect=8` on `/mnt/smithy` (**declined in scope**),
|
||||
fused MoE kernel path (**parked, id 47**), TTS-stack move to fv-ml1 (**parked, id 75**).
|
||||
|
||||
### OPEN, operator's call, nothing blocked
|
||||
|
||||
- **Henge id 79** (`catalog-all-robotics-bits-and-bobs`) was filed under source
|
||||
`eshpfi-management` by the skill's repo inference; it is a personal idea. Re-park with
|
||||
`--personal` if wanted.
|
||||
- **xAI billing** — did the stopped HTTP A/B bill on top of the coding plan? Needs the
|
||||
operator's console; this fleet holds no xAI credential.
|
||||
- **A `BabyYarros → lv-yarros` pointer** in memory so historical entries stay findable.
|
||||
- Older deferred set, unchanged: AI-tab Dormant regrouping (**belayed**), `nconnect=8` on
|
||||
`/mnt/smithy` (**declined in scope**), fused MoE kernel path (**parked, id 47**), TTS-stack
|
||||
move to fv-ml1 (**parked, id 75**).
|
||||
|
||||
⚠ **fv-ml1 GPU 0 is at 96.0 of 97.9 GB.** Adding `lv-bronte` cost nothing measurable (96092 →
|
||||
96090 MiB) because a LoRA rides inside the existing seat — but a new SEAT still needs a
|
||||
placement decision. GPU 3 is a HELD RESERVE.
|
||||
|
||||
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified
|
||||
before this session began. ⚠ **Do NOT commit them** — untouched and deliberately left alone.
|
||||
⚠ **Uncommitted and NOT mine:** `graphify-out/GRAPH_REPORT.md`, `scripts/seat-inventory.py`,
|
||||
`stacks/searxng/{README.md,conf/searxng-settings.yml}`, `configs/esh-scale/`. Leave them alone.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-17]` ⭐⭐ **A unit splitter must choose by SIZE, not count — and the val split scales with WORK COUNT, not corpus size.** `scripts/r49-corpus/split_units.py` + a multi-index `--holdout-chapter`. The inherited most-units rule gave Cities of the Plain 4 units of 22,312w (the book's PARTS); the single-index holdout would have given McCarthy a Brontë-class 18k-word val reference on a 588k corpus. Both fixed, both caught by controls. → `persistent-memory.d/2026-09-17-mccarthy-d1-d3.md`
|
||||
- `[2026-09-17]` ⭐ **lv-mccarthy D1–D3 complete on gx10, leak gate PASSED (0 of 75 renameable, 0 of 37 sub-threshold, both controls green).** Three McCarthy-specific calls, each forced by a measurement: corpus-scoped rename (the Border Trilogy shares 9 surfaces across books), a new `mccarthy` name preset (Hemingway's carries it_IT/fr_FR and McCarthy writes neither), and `--min-cap 5` to match the entity map's floor — the first gate run failed with 45 survivors purely because rename's floor was 8 and the map's was 5. → `persistent-memory.d/2026-09-17-mccarthy-d1-d3.md`
|
||||
- `[2026-09-17]` ⭐⭐ **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.** Operator: *"he's a great writer because of his research, not because he has a strong identifiable voice."* I surveyed, built the corpus, measured containment and fixed three stripping defects before anyone asked the question that decided it. henge id 82. → auto-memory `feedback_voice_worth_adapting_before_corpus`.
|
||||
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared at a different minute. A relaunched pane therefore drops to pull mode and stops receiving mail. Raised with the operator, not acted on, because installing it edits `~/.claude/settings.json` and a peer's question is the wrong authorisation for a config change. Tracked at althing thread `01M2R0KPPE85SVJ96KSQ3YKKQP`.
|
||||
- `[2026-09-17]` **Hemingway ships as-is: operator ruled "ship stands" on both measured corpus defects** — the 0.96% beat contamination and the 130 non-name entity-map surfaces. `audit_pairs_sourcenames.py --filter-out` and `audit_entity_map.py` exist and are the instruments if that is ever revisited; neither was run against the shipped adapter.
|
||||
|
||||
- `[2026-09-17]` ⭐⭐⭐ **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean.** Operator: *"he's a great writer because of his research, not because he has a strong identifiable voice."* A voice adapter is worth its ~6 hours only if the target has a prose signature a reader could pick out blind; Krakauer's excellence is reporting, which an adapter cannot carry. I surveyed, built, measured containment and fixed three stripping defects before anyone asked the question that decided it. **For each candidate, say what the voice IS in one sentence and how it shows up in char-bigram space, before the first catalogue query.** henge id 82. → auto-memory `feedback_voice_worth_adapting_before_corpus`.
|
||||
|
||||
- `[2026-09-17]` ⭐⭐ **A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters".** `scripts/r49-corpus/split_units.py`: a marker mode qualifies only if its median unit is inside [600, 12000] AND no unit holds half the work; among qualifying modes PRIORITY breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title), and paragraph-block sections are the fallback for works with no divisions. ⭐ **Both rules exist because a control caught them**: scoring by "median closest to target" chose `caps-title` (6 units, one holding **97%** of the book) over True at First Light's real 20 chapters, because a median cannot see that distribution and a max bound can. Positive control: 8/10 Hemingway works reproduce the shipped mode and count exactly. Negative control: 40,000 words with no blank lines → 1 unit, refuses to fabricate divisions. Commit `705fa3a`.
|
||||
@@ -456,11 +441,6 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
|
||||
- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md`
|
||||
|
||||
|
||||
- `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
|
||||
|
||||
|
||||
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
|
||||
|
||||
@@ -469,7 +449,7 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
_10 older entries archived to archival-memory.md._
|
||||
_11 older entries archived to archival-memory.md._
|
||||
|
||||
_5 older entries archived to archival-memory.md._
|
||||
|
||||
|
||||
Reference in New Issue
Block a user