In-flight rewritten for the next goal. lv-hemingway is TRAINED and nothing else
has been done to it: ship candidate is checkpoint-1750 (ep 1.97, eval 2.2783),
the end-of-run adapter is 0.0763 worse, and the v2 gate has not been run. Every
instrument it needs was parameterised during the lv-bronte run tonight and the
in-flight section names all four with their traps.
New detail files:
2026-09-17-lv-bronte-gate.md shipped, voice axis failed, why anyway
2026-09-17-beat-contamination-leak.md the leak the corpus gate cannot see
2026-09-17-esh-fiber-outages.md two Cityside failures, rotation fragility
Also commits the memorization_check.py parameterisation, which was left
uncommitted: its hardcoded Yarros defaults would have compared a Hemingway arm
against the Yarros corpus and reported a meaningless clean zero.
Auto-archival: index was 415 lines pre-run, over the 300 cap. Only five entries
cleared the 14-day age guard, and three of those carry open deferred pointers
(fused MoE park 47, nconnect=8, AI-tab belayed) and are referenced by in-flight.
A fourth — every CI job on pfi-fleet runs as root on ana-docker — is a live
security property rather than settled history, so it is held back deliberately.
One entry archived. The file stays over cap, which is the guard working: an
over-cap file that keeps live decisions beats a scannable one that lost them.
@@ -4,6 +4,9 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
## Recent decisions (archived)
## Recent decisions (archived)
- `[2026-09-02]`**`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
_Archived 2026-09-17._
- `[2026-09-01]`**Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
- `[2026-09-01]`**Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
_Last updated: 2026-09-16 ~16:15 PT (lv-yarros SHIPPED — instruction-pair SFT beats raw-text on voice; voices-seat live on fv-ml1 GPU 0 :8027 with measured 24.3% LoRA cost; lv-hemingway corpus gated and training, ~1h out; Grok token broker built then SHELVED by the keep-the-jail ruling.)_
_Last updated: 2026-09-17 ~01:30 PT (lv-bronte SHIPPED with a FAILED voice axis on the record; a beat-contamination leak class the corpus gate cannot see, found and patched; audit_stoplist added; next goal is landing lv-hemingway, which is trained but un-gated.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,43 +115,74 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
## Current state / in-flight
_As of 2026-09-16 ~16:15 PT._
_As of 2026-09-17 ~01:30 PT._
### One job in flight: `lv-hemingway` training on pfi-gx10, ~1 hour out.
### Next goal: land `lv-hemingway`. It is TRAINED; nothing else has been done to it.
built during the lv-bronte run tonight — reuse it rather than rebuilding:
**OPEN, operator's call, nothing blocked:**
1.`scripts/r49-corpus/build_beat_fixture.py --pairs <hemingway val pairs> --out beats-hemingway-30.json --sidecar ... -n 30` — refuses any split but `val`.
- **Brontë has never been through the automated leak gate** — its "0 of 203" was a HAND COUNT and
2.`scripts/r49-corpus/gen_beats_chat_yarros.py --system-from <pairs>.provenance.json` — **the `--system-from` flag is mandatory**; the harness's hardcoded SYS is Yarros's and driving a Hemingway arm with it confounds the carrier change with a prompt change.
the gate did not exist yet. On Yarros the same instrument read **212 surviving where a hand count
3.`scripts/yarros-corpus/memorization_check.py --eval-dir ... --corpus ... --glob 'beats5.*.jsonl'` — **now takes paths**; its old hardcoded Yarros defaults would have compared a Hemingway arm against the Yarros corpus and reported a meaningless clean zero.
said 86**. Its gated corpus survives at `gx10:~/r49-corpus-renamed-unwrapped/` and is
4.`scripts/r49-corpus/voice_distance.py <renamed-corpus> <eval-dir>` — needs `voice.<arm>.jsonl` files with a `continuation` field, and identifies the control by the substring **`unadapted`** in the arm name. Adapt with a script like `gx10:~/lv-bronte/voice-prep.py`.
pair-buildable; `lv-bronte` exists only as RAW-TEXT arms, i.e. the arm that never cleared its own
control. Re-gate before building pairs, or accept the hand count.
⚠ **Hemingway's pairs were built BEFORE the `--source-entities` beat filter existed.** Measured
- **xAI billing** — did the stopped HTTP A/B bill on top of the coding plan? Needs the operator's
on Brontë: 1.8% of generated beats named the author's real characters, because the beat-writing
console access; this fleet holds no xAI credential.
model recognises the book and restores canonical names — a leak the corpus gate structurally
- **A `BabyYarros → lv-yarros` pointer** in memory, so historical entries stay findable under the
cannot see. Exposure scales with how famous the book is. **Re-verify the Hemingway pairs against
new name. Historical entries were deliberately left as dated records.
its entity map before trusting them**, and regenerate with `--source-entities` if they leak.
⚠ **Do not assume the two-epoch recipe.** It held for Yarros and Hemingway and did NOT hold for
Brontë. Read the eval curve; Hemingway's minimum genuinely is at 1750, which is already known.
### SHIPPED tonight — `lv-bronte`, with a FAILED voice axis on the record
Live on `vllm-voices` (fv-ml1 GPU 0 :8027) as `lv-bronte` beside `voices-base` and `lv-yarros`.
Checkpoint-475. **It did not pass its voice gate** (+0.193 delta_cb against a 0.251 measured
floor); shipped because it is additive, reversible, and clean on the safety axis (8-gram overlap
0.00, identical to the never-saw-it control) on a public-domain corpus. The caveat is written
into the commit, the compose file, and a README beside the adapter on NFS.
`/mnt/smithy` (**declined in scope**), fused MoE kernel path (**parked, id 47**), TTS-stack move
`/mnt/smithy` (**declined in scope**), fused MoE kernel path (**parked, id 47**), TTS-stack
to fv-ml1 (**parked, id 75**).
move to fv-ml1 (**parked, id 75**).
⚠ **fv-ml1 GPU 0 is at 96.0 of 97.9 GB.**GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 a HELD RESERVE. The
⚠ **fv-ml1 GPU 0 is at 96.0 of 97.9 GB.**Adding `lv-bronte` cost nothing measurable (96092 →
next seat needing room on fv-ml1 requires a placement decision.
96090 MiB) because a LoRA rides inside the existing seat — but a new SEAT still needs a
placement decision. GPU 3 is a HELD RESERVE.
**Uncommitted:**`graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified
**Uncommitted:**`graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified
before this session began. ⚠ **Do NOT commit them** — untouched and deliberately left alone.
before this session began. ⚠ **Do NOT commit them** — untouched and deliberately left alone.
## Recent decisions
## Recent decisions
-`[2026-09-17]` ⭐⭐ **A leak class the corpus gate structurally CANNOT see: the beat-writing model recognises the book and restores the author's real character names.** 1.8% of Brontë beats named Rochester/Jane/Brocklehurst while 0 responses did. Worst for public-domain classics; Yarros and Hemingway's clean runs are NOT evidence they are immune. Patched as a `sourcename` reject + `--source-entities`. → `persistent-memory.d/2026-09-17-beat-contamination-leak.md`
-`[2026-09-17]` ⭐ **A stoplist entry is an assertion the leak gate can no longer check** — stoplisting removes a surface from the entity map, so a wrongly stoplisted CHARACTER is an undetectable leak. Three were wrong on Brontë (Leaven, Pierrot, Samuel); `scripts/r49-corpus/audit_stoplist.py` finds them by honorific and now gates the pipeline. Commit `8bb7686`.
-`[2026-09-17]`**ESH: Cityside Fiber failed TWICE (19:09 and ~01:06); operator switched WAN1 to DHCP to restore service and has a ticket for the static.** crowdsec `esh` allowlist carries both failover egresses with 7-day expiries — the rotation-fragility is live. → `persistent-memory.d/2026-09-17-esh-fiber-outages.md`
-`[2026-09-17]` ⭐⭐ **lv-bronte SHIPPED on voices-seat (ckpt475) DESPITE failing the v2 VOICE axis — additive, reversible, safety-axis clean.** Both candidates closed 48–52% of the achievable distance to Brontë but +0.193/+0.210 sit under a 0.251 noise floor set by ONE outlier seed in the arm not being shipped; cause is structural (81 val pairs vs Hemingway's 200) and not cheaply fixable. ckpt475 is the pick if it ships. The two-epoch recipe did NOT transfer. → `persistent-memory.d/2026-09-17-lv-bronte-gate.md`
-`[2026-09-17]` ⭐⭐ **lv-bronte SHIPPED on voices-seat (ckpt475) DESPITE failing the v2 VOICE axis — additive, reversible, safety-axis clean.** Both candidates closed 48–52% of the achievable distance to Brontë but +0.193/+0.210 sit under a 0.251 noise floor set by ONE outlier seed in the arm not being shipped; cause is structural (81 val pairs vs Hemingway's 200) and not cheaply fixable. ckpt475 is the pick if it ships. The two-epoch recipe did NOT transfer. → `persistent-memory.d/2026-09-17-lv-bronte-gate.md`
-`[2026-09-16]` ⭐⭐ **Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor).**`lv-yarros` shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → `persistent-memory.d/2026-09-16-lv-voices-line.md`
-`[2026-09-16]` ⭐⭐ **Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor).**`lv-yarros` shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → `persistent-memory.d/2026-09-16-lv-voices-line.md`
-`[2026-09-16]` ⭐ **voices-seat live: one carrier, N `lv-<author>` LoRA adapters, hot-swap measured at 0.24 s.** LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; `--gpu-memory-utilization` is a request against TOTAL VRAM and only a pinned KV makes it predictive. → `persistent-memory.d/2026-09-16-voices-seat-lora.md`
-`[2026-09-16]` ⭐ **voices-seat live: one carrier, N `lv-<author>` LoRA adapters, hot-swap measured at 0.24 s.** LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; `--gpu-memory-utilization` is a request against TOTAL VRAM and only a pinned KV makes it predictive. → `persistent-memory.d/2026-09-16-voices-seat-lora.md`
@@ -382,9 +413,6 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
-`[2026-09-03]`**nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md`
-`[2026-09-03]`**nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md`
-`[2026-09-02]`**`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
-`[2026-09-02]`**Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
-`[2026-09-02]`**Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
@@ -398,7 +426,7 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
-`[2026-08-19]`**AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
-`[2026-08-19]`**AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
_9 older entries archived to archival-memory.md._
_10 older entries archived to archival-memory.md._
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
