memory: snapshot — nh3-pve GPU install pending (post-boot checklist), TEI fleet embed/rerank + reward on esh-ml1, 34 entries archived
This commit is contained in:
@@ -10130,3 +10130,384 @@ next colo visit (Dell R750xs 12×3.5" SAS backplane; front-half teardown), then
|
||||
`docs/runbooks/pfi-pve-naspool-rebuild.md`. Booth: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/
|
||||
_Archived 2026-09-24._
|
||||
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
# `[2026-09-11]` Plex never hardware-transcoded on the Arc, and every setting said it should
|
||||
|
||||
Operator: *"I believe plex is running but I guess it's using cpu quicksync — anything to
|
||||
gain by moving it to a gpu?"* The premise had a fold in it (**Quick Sync *is* a GPU** —
|
||||
it's the iGPU's media engine), and the real answer was that the GPU was already wired up
|
||||
and Plex had been unable to use it.
|
||||
|
||||
## The configuration was correct the entire time
|
||||
|
||||
Plex is **LXC 105 (`vm-plex`) on esh-pve-nas**, 10.0.50.56. Verified good *before* any
|
||||
change: Arc A580 present with DMC/GuC/**HuC authenticated for all workloads**; both
|
||||
render nodes bind-mounted into the LXC with cgroup allows; `plex` in `video(44)` +
|
||||
`render(104)`; Plex 1.43.2; lifetime Plex Pass; `HardwareAcceleratedCodecs=1`; and
|
||||
`HardwareDevicePath` **already pointed at the Arc** (`…@0000:03:00.0`), not the iGPU.
|
||||
|
||||
⚠ **`HardwareAcceleratedCodecs` is ABSENT from `Preferences.xml` when it is ENABLED** —
|
||||
Plex only persists non-defaults. I read its absence as "off" and was wrong; the API
|
||||
(`GET /:/prefs`) reported `1`. Read Plex settings from the API, never the file.
|
||||
|
||||
## Root cause, one layer below every setting
|
||||
|
||||
`intel-media-va-driver` **22.3.1 (Apr 2023, stock jammy)** — predates Arc/DG2 support
|
||||
entirely and exports only `__vaDriverInit_1_14`, against the libva **2.22** that Plex
|
||||
**bundles** and loads via RPATH (`/usr/lib/plexmediaserver/lib/libva.so.2`), not the
|
||||
system one.
|
||||
|
||||
⚠ And a **half-finished prior attempt at this same fix** was sitting there: libva and
|
||||
libva-drm hand-installed at 2.22 (not dpkg-owned), `libva-x11` left at 2.14, so every
|
||||
X11 VA-API consumer died on `undefined symbol: va_fool_postp`. Upgrading a library
|
||||
without its companions is how you get a stack that is broken in a way no single package
|
||||
version explains.
|
||||
|
||||
## Fix + pin
|
||||
|
||||
Intel client-GPU repo (`https://repositories.intel.com/gpu/ubuntu jammy client`, a
|
||||
**rolling** track) → `intel-media-va-driver-non-free` **24.3.4** (`__vaDriverInit_1_22`,
|
||||
exact ABI match for Plex's libva), libva set **2.22.0.2-87**, libigdgmm12 22.5.2. The
|
||||
orphaned manual libva is now dpkg-owned. **Pinned** in
|
||||
`/etc/apt/preferences.d/intel-gpu-pin` **and** `apt-mark hold`, verified by a simulated
|
||||
upgrade moving 152 packages and touching none of the six.
|
||||
|
||||
## ⚠⚠ Two footguns, both of the silent-wrong class
|
||||
|
||||
**`pct snapshot` REFUSES on a guest with a bind mount AND STILL EXITS 0.** LXC 105 has
|
||||
`mp0: /tank/media`, so Proxmox printed `snapshot feature is not available` and returned
|
||||
`rc=0`. A script trusting that exit code believes it has a backup it does not have. The
|
||||
rootfs is on ZFS, so the working path is `zfs snapshot nvme/subvol-105-disk-0@<tag>` —
|
||||
**and read it back**, per [[feedback_unfalsifiable_at_write_time]].
|
||||
|
||||
**A synthetic `Plex Transcoder` invocation is NOT a valid test of Plex's transcode
|
||||
path, and I burned several rounds proving it.** Plex bundles its own libc among 61
|
||||
libraries; running its ffmpeg from a shell produced three different failure modes
|
||||
(`unknown libva error`, then a libstdc++ `__wmemmove_chk` relocation error) that were
|
||||
artifacts of the harness, and it **failed identically before and after a fix that
|
||||
worked**. No positive control existed, so its negatives carried no information — the
|
||||
exact shape of the tag-detection specimen in the measurement-discipline rule.
|
||||
|
||||
## What actually settles it
|
||||
|
||||
A forced transcode, reading Plex's own log. PASS **names the device**:
|
||||
|
||||
Codecs: testing h264_vaapi (encoder)
|
||||
Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])
|
||||
Codecs: testing h264 (decoder) with hwdevice vaapi
|
||||
|
||||
plus `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only follows a
|
||||
successful `vaInitialize`. FAIL is `final decoder: , final encoder:` with the device
|
||||
never named. ⚠ And an empty-fields line is **also** what a server that never transcoded
|
||||
emits — `TranscodeSession` count was 0, so the original log was an *absence of
|
||||
evidence*, not evidence of failure. I called it failure first and had to withdraw that.
|
||||
|
||||
`vainfo` is a secondary check only (now: iHD 24.3.4, H.264/HEVC VLD **and** EncSliceLP
|
||||
on the Arc) — it exercises the **system** libva, so it can pass while Plex fails.
|
||||
|
||||
Runbook: `docs/runbooks/plex-arc-vaapi-jammy.md`.
|
||||
Rollback: `pct stop 105; zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911; pct start 105`.
|
||||
|
||||
**Left alone:** Jellyfin LXC 107 on the same host has the same stale stack and the same
|
||||
Arc available — operator 2026-09-11: not actively used. It ships its own ffmpeg so this
|
||||
fix may not transfer verbatim. LXC 105 also has 152 unrelated pending package upgrades.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# Priority 2 complete
|
||||
|
||||
pfi-postgres, esh-vm-db, pbs-ana, pbs-nh3 have native unprivileged 0.18.7 agents,
|
||||
enabled at boot and fresh hub samples verified 2026-09-12 01:56Z. Sixteen new
|
||||
Disk/CPU/Memory/Status rules verified, existing infra-ops bridge route retained.
|
||||
Fleet 17/18 up; only known ana-ml2 outage. DB/PBS services verified running,
|
||||
Postgres accepts connections; no application or VM restarts performed.
|
||||
|
||||
ESH uses existing infra-ops sudo. Other three: lkraven key SSH works, sudo needs
|
||||
password, no matching vault entries. Deployed through EXISTING Proxmox guest
|
||||
agents (no account/access changes): pfi-pve 105=postgres, 100=pbs-ana;
|
||||
nh3-pve 105=pbs-nh3. Never confuse overlapping VMIDs. Canonical stage/install
|
||||
playbooks and host envs saved; runbook configs/beszel-agent/PRIORITY2.md.
|
||||
PBS-NH3 export ~75.5% used; PBS-ANA ~6.6%. Resource checks are not job-success
|
||||
monitoring. Local changes still uncommitted, awaiting main-branch approval.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# Beszel priority 1 — six deployed and verified
|
||||
|
||||
Operator requested two NAS plus four hypervisors. Native 0.18.7 agents enabled
|
||||
and verified on ana-nas, pfi-pve, nh3-pve, esh-pve, esh-pve-nas; dedicated
|
||||
unprivileged beszel user, no Docker installed on hypervisors. Existing hub key
|
||||
authentication and operator account. All five have Disk/CPU/Memory/Status
|
||||
alerts using existing infra-ops webhook: 20 new rules, 50 total from our wiring.
|
||||
Homepage 13/13 up; live samples verify extra filesystem capacity. Physical
|
||||
NICS filters avoid double-counting guest virtual networking. No VM workloads,
|
||||
storage configuration, existing agents or hub versions changed.
|
||||
|
||||
Canonical configs/beszel-agent/, playbooks/beszel-native.yaml. Agent archive
|
||||
checksum and exact mount configuration in README/env files. ZFS parents' df
|
||||
usage is not pool allocation; degradation alerts still not configured.
|
||||
|
||||
RESOLVED access: operator pointed to Vaultwarden nh3-nas/infra-ops-password;
|
||||
password SSH and sudo both succeeded. Earlier key-only failure did not mean
|
||||
privileged access was unavailable. No account/permission changes needed.
|
||||
Synology Docker agent installed 21:50Z, md0 root and volume1 detected; existing
|
||||
rest-server healthy/unrestarted. Registered gru841k5cuwb5z2 + 4 alerts, 54 total
|
||||
from our rollout. Canonical stacks/beszel/synology/compose.yaml and initial
|
||||
playbook beszel-synology.yaml. Vault-fed /tmp/nh3-nas-elway.py handles sudo and
|
||||
DSM scp -O without logging password. Configs saved but NOT committed: previous
|
||||
auto-review rejected main-branch commit; no bypass/retry attempted.
|
||||
|
||||
RESOLVED at 2026-09-12 01:51Z after operator reported Anaheim back except
|
||||
ana-ml2: Homepage 13/14 up, all six priority-1 hosts up, only ana-ml2 down.
|
||||
Synology live sample 01:50:32Z: CPU 1.11%, memory 7.05%, root 66.02%, volume1
|
||||
32365.07/42879.36 GiB (~75.5%). Four alerts verified. Priority-1 rollout complete.
|
||||
No networking or ana-ml2 changes made; configs remain uncommitted.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UNVERIF
|
||||
|
||||
**Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UNVERIFIED (blocked on GPU space).** Operator got access to `glyphsoftware/sentinel-r3` and asked to compare vs the running M.O.G.-SEC seat + pull if promising, then "quant it with a grafted mtp head". **It is promising and a better FIT**: same base (stock Qwen3.8-27B), same `qwen3_5` hybrid arch, same 262K, vision-intact — but M.O.G.-SEC is ALSO an SFT finetune (refusal-free offense+defense cyber SFT — its card: "not a system-prompt sticker on a stock Qwen"; the "persona on stock" wording here is WRONG, corrected 2026-09-14) while Sentinel-R3 is an SFT finetune on 1,230 authorized-pentest agent trajectories over a 19-tool surface that **matches our own harness** (Bash/Read/Write/Edit/Grep/Glob/Agent/Task*/Monitor/…). Card is unusually honest (flags its own mmlu-cybersec 0.88 as within-noise of base). **HF check: M.O.G.-SEC repo unchanged** (sha still our pinned `deede6779…`). **MTP: Sentinel ships ZERO mtp tensors**; grafted the verbatim base head from `qwen38-27b-uncensored-bf16` (compare_mtp_head → IDENTICAL) — lineage correct since Sentinel's base is stock Qwen3.8-27B and that head is a verbatim base graft. ⚠ **Acceptance is UNVERIFIED and may differ from the 47.7% the head hits on STOCK weights** — it now reads hidden states from an SFT-finetuned body (the exact Stage-1b residual risk). Quant = the standard mixed NVFP4-W4A4(MLP 0-55) + FP8-W8A8(attn/linear_attn/lm_head/MLP 56-63) recipe, ran CUDA_VISIBLE_DEVICES=1 on GPU1 free space, no seat downtime, 51→22 GB. post_quant carried the head forward + re-injected `re:^mtp.*` (llm-compressor prunes it → the 0%-accept bug). Structural verify clean: 1968 tensors, 0 unresolved, 15 mtp, 333 visual, ignore has mtp+visual. **Artifact `/tank/aimodels/sentinel-r3-nvfp4-mixed` (+ `.PROVENANCE.txt`).** ⚠ **License is PROPRIETARY** (Glyph Proprietary v1.0, all-rights-reserved) — operator's fair-use/licensee call, not apache like M.O.G.-SEC. ⚠ **Serving/A-B is BLOCKED on GPU space**: weights are 22 GB, GPU0 has 7.6 free / GPU1 19.9 — a probe serve needs a freed co-tenant slot (~25 GB), which is a material-consequence call. Serve with the PROSE system prompt (trained on prose tools, not structured `tools=`). → `/tank/aimodels/sentinel-r3-nvfp4-mixed.PROVENANCE.txt`
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VRAM is
|
||||
|
||||
⭐ **MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VRAM is not the constraint and never was.** Operator asked to run the two BabyYarros arms in parallel if VRAM allowed. It does, comfortably: **18.4 GiB per 4B LoRA job, 36 of 121 GiB with both up, 98 GiB free.** But the GB10 is a *capacity* box, not a throughput box, and the binding constraint is memory bandwidth. Solo baseline **37.10 s/it (n=6, 0.05% spread)**; with a second job both arms settled at **~85 s/it — 2.29x each**, so combined throughput 0.0235 vs 0.0270 steps/s solo. Not a clean 2x split: the box is *past* its roofline and pays a contention penalty on top. ⭐ **Control: killing the second job returned the first to 37 s/it on the very next step**, so the slowdown tracked contention and reversed with it. Chaining finished both arms ~43 min earlier than concurrency would have. **General form: on this box, `nvidia-smi` free memory tells you nothing about whether a second job is affordable.** Decision rule was pre-registered before the numbers were read (<55 s/it keep both, ≥2x chain). `scripts/yarros-corpus/{launch-yarros-4b-base,chain-yarros-4b-base}.sh`; the shared-GPU bypass is an explicit argument, never a default.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (dual
|
||||
|
||||
⭐ **ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (dual Blackwell PRO 6000, ~1.5 kW peak) is the ROOT CAUSE of the repeated Anaheim rack-breaker trips (2026-08-26, 2026-09-11) — moving it to its own circuit fixes the recurring whole-site outage.** New site `fv`, same shape as Anaheim: server subnet **10.251.50.0/24** (fv-ml1 = **10.251.50.54**, mirroring the old host octet), mgmt/BMC **10.251.250.0/24** (fv-ml1-bmc = **10.251.250.50**). **OPNsense firewall is the multi-homed gateway** (`.1` in every FV VLAN) **AND the tailscale/headscale subnet-router advertising 10.251.0.0/16** — chosen over ana-ml2-as-endpoint specifically because the firewall stays up when the GPU box is down, giving out-of-band BMC access over the mesh — the exact thing the fleet LACKED during today's outage (no OOB path, BMC islanded). **Rename to `fv-ml1`, full `fv.internal` DNS name.** DNS approach: **PIGGYBACK** — `dns-sync` builds `name.site.zone` with no check that the site is in the `sites:` block, so `fv-ml1`/`fv-ml1-bmc` records with `site: fv` resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately; add a real `fv` resolver only when FV needs LOCAL resolution (OPNsense can't host the AdGuard the sync targets — it's FreeBSD/Unbound). **Clean cutover: the box is already down (BMC dark, no power since the outage), and `/tank` is LOCAL ZFS with NO NFS from ana-nas, so data travels with the chassis.** ⚠ Load-bearing repoint = `stacks/litellm/conf/config.yaml` (~10 `api_base: 10.250.50.54:{8015,8016,8018,8019}` → `10.251.50.54`; darkens every inference alias if missed) — gateway STAYS on ana-docker so fv-ml1 serves cross-site (FV↔Anaheim metro, fine). Everything staged, nothing deployed: **runbook `docs/runbooks/fv-ml1-cutover.md`** (commit `ce04f9d`; exact DNS + LiteLLM commands) + **`scripts/fv-ml1-rename-sweep.sh`** (`8400f3a`; scoped, dry-run default, history/provenance-safe, manual-review list for judgement calls).
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **Anaheim rack LEFT DARK until the move (operator decision).** ana-ml2 is the ONLY host still down post-recovery (BMC dark = no power); rather than power it on tonight just to shut it down for the truck tomorrow, it stays off. I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) but there is nothing to bring up — the box relocates as fv-ml1.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bou
|
||||
|
||||
⚠ **RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bouncer fail-CLOSES — empty-body 403 on EVERY HTTP service behind traefik (gitea, homepage, …) while the apps themselves are fine.** Signature (bifrost-dev reported it, gitea-shaped): HTTPS returns `403 content-length 0, no app body` on all routes, but git-over-SSH works (SSH bypasses traefik). Diagnosis: `gitea` direct on `localhost:3000` = 200 (app healthy), through traefik = 403; `crowdsec` container `Exited (255)`; `cscli decisions list` EMPTY (not an IP-ban). **The bouncer plugin does NOT self-recover from a startup-time LAPI-unreachable race** — even after crowdsec is healthy again, traefik keeps 403ing until traefik itself is restarted. **FIX: `docker start crowdsec` (its data/config are LOCAL volumes, comes up clean), wait for `cscli lapi status` = OK, THEN `docker restart traefik`** so the plugin re-inits against the live LAPI. Verified 403→200 on gitea API/web/PyPI-index from an off-box vantage. This unblocked bifrost-dev's 1.2.0 PyPI publish (+ worldtree/wyrd/ratatoskr) and any HTTP gitea access; heid's SSH pushes were never affected. → add to the recovery runbook: **crowdsec+traefik restart is a standard post-power-loss step.**
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **Anaheim colo recovered ~16:39 PT EXCEPT ana-ml2 (bare metal, NO power — its BMC 10.250.250.50 is dark on standby, unlike same-subnet pfi-pve which is up → needs a physical PDU/PSU/breaker fix, not a boot).** pfi-pve + all its VMs (ana-docker/ana-nas/ana-wg/corviduo-dev/pbs-ana) auto-started clean (on-boot gap held this time). LiteLLM came back up on its own (transient `unhealthy` during startup → serving). ⚠ **Public WAN (38.120.12.44) ICMP still blocked from outside but HTTPS works fleet-internally** (mesh-routed). ana-ml2 down blocks the gen/summarizer/mog-sec seats AND the cyber-preview quant re-run. **I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) to power-on + boot-watch the instant its BMC returns.**
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured noise
|
||||
|
||||
⭐⭐ **BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured noise floor, and the instruct arm renders beats 9/10.** Training: **Base best held-out 2.5263 @ ckpt-125** (overfits within the epoch — best is the checkpoint, not the shipped step-178 adapter), **Instruct 2.6114 @ 178** (still descending, undertrained if anything). Base-wins-held-out / Instruct-holds-instruction replicates Brontë at a near-identical **0.085-nat** gap. **Eval (gx10, seat-free, done during the Anaheim outage):** three voice arms + instruct beat→paragraph. `delta_cb` (Burrows over char-bigrams vs held-out Yarros) ordering **base-125 0.549 < instruct 0.631 < base-unadapted 0.706**, same-author target 0.463; both adapters clear the **0.046 measured noise floor** (within-arm seed spread, not the same-author distance — first cut mis-framed that) — base **+0.157**, instruct **+0.076** vs control. ⚠ One seed-pair per arm, so the ordering CORROBORATES the independent held-out-loss ordering rather than settling it. **Beats (instruct, chat template, Yarros SYS): on-beat 9/10** (it takes direction after raw-text training — the Skaldsong question, answered yes), in-band 5/10, ran-on 7/10 (length + clean-close discipline is the weak axis, same trade as Brontë). Booth: **http://10.100.10.50:8090/b/babyyarros-voice/**. Tooling `scripts/r49-corpus/{voice_prompts_yarros.json,gen_beats_chat_yarros.py,voice_distance.py,build_booth_yarros.py}`, commit `5558d9c`. ⛔ **DEFERRED to power-return** (needs the ana-ml2 gen seat): the frozen adjudication's romantasy control panel, a 2nd seed, and the beat-incumbent leg.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` BabyYarros — the leak gate passes, and closing it found three defects nobody was looking for
|
||||
|
||||
Operator's goal for the session: *"finish the hardening and start the yarros training on
|
||||
the gx10."* Both done. The gate went 212 → 0 and the first BabyYarros training is live.
|
||||
|
||||
## The gate did not exist, which is why the number was wrong
|
||||
|
||||
There was no committed instrument for *does any of the author's own proper nouns survive
|
||||
the rename*. Brontë's "0 of 203" was produced by hand, and BabyYarros's "86 of 232" was
|
||||
too. `scripts/r49-corpus/leak_gate.py` is now that instrument.
|
||||
|
||||
It runs **both directions every time**, because a detector that only ever sees renamed
|
||||
text cannot distinguish *absent* from *blind*:
|
||||
|
||||
- **positive control** — the same scan over the UNRENAMED source; every surface must be found
|
||||
- **negative control** — a nonce string that must appear nowhere
|
||||
|
||||
Its first reading on the corpus as built was **212 surviving, not 86**. Two reasons, and
|
||||
both are structural rather than a disagreement about counting: it scans the **whole corpus**
|
||||
rather than each work separately, and it counts the **sub-threshold** entities `rename.py`
|
||||
never looked at. A per-work gate reports a name renamed in one book and printed verbatim in
|
||||
another as clean.
|
||||
|
||||
## Defect 1 — a second typography defect, and the D1 note was right about the wrong thing
|
||||
|
||||
The D1 build notes say no unwrap was needed because Kvasir's cleaner emits flowing
|
||||
paragraphs. That is true and Brontë's hard-wrap defect genuinely is absent. A different one
|
||||
is present:
|
||||
|
||||
— M AJOR A FENDRA’S G UIDE TO THE R IDERS Q UADRANT (U NAUTHORIZED E DITION )
|
||||
T he flight field at Basgiath is still dark…
|
||||
|
||||
The Empyrean books set chapter epigraphs in **small caps**; the extractor rendered the
|
||||
small-caps run as uppercase and left the large initial as its own token. **106 lines, ~700
|
||||
splits**, plus **52 drop caps** (51 of them iron-flame). That is the entire origin of the
|
||||
entities called `IDERS`, `UADRANT`, `NAUTHORIZED`, `DITION`, and seventeen bare single letters.
|
||||
|
||||
⭐ **The restoration is exact, not approximate.** A split initial beside an uppercased run
|
||||
recovers the original mixed case: a word WITH a split initial was capitalised in the source,
|
||||
an all-caps word WITHOUT one was lowercase. So the line above restores to
|
||||
*"—Major Afendra's Guide to the Riders Quadrant (Unauthorized Edition)"*, which is what she wrote.
|
||||
|
||||
⚠ Two guards, both load-bearing: only lines with **≥2** splits are treated as a small-caps
|
||||
run (one split is an ordinary sentence beside an acronym), and **`I`, `A`, `O` are excluded**
|
||||
from the drop-cap join — otherwise `A slow smile spreads` becomes `Aslow`.
|
||||
|
||||
`scripts/yarros-corpus/repair_typography.py`. Cost: 768 fragments rejoined.
|
||||
|
||||
## Defect 2 — back matter inside the prose, in all five works
|
||||
|
||||
The builder splits on chapter headings and nothing follows the last one, so every work
|
||||
carried its acknowledgments, newsletter pitches and cover-artist credits **inside the final
|
||||
chapter**: 4,555 words naming the author's agent, her editors and her children, in a corpus
|
||||
whose entire purpose is that no identifiable name survives.
|
||||
|
||||
Found by the phrase audit surfacing **`Louise Fury`** (Yarros's literary agent) — not by
|
||||
reading. ⚠ iron-flame's marker is `ACKNOWLEDGMENTS` in **all caps** and a case-sensitive
|
||||
scan missed it. The strip is case-insensitive, last-chapter-only, and refuses if it would
|
||||
remove more than 2% of the corpus.
|
||||
|
||||
## Defect 3 — the gate read 0 of 314 while `Afendra` was in every copy
|
||||
|
||||
The worst failure shape available: a clean number over a real leak.
|
||||
|
||||
`Afendra` never appears unpossessed, so it keyed as `Afendra’s` — and **`rename.py` and the
|
||||
gate both skip apostrophe keys as contractions.** Unrenamed and unreported at once. Fixed by
|
||||
folding clitics so `Afendra’s` counts toward `Afendra` (`--fold-clitics`).
|
||||
|
||||
`Baxter` escaped a different way and is the better story. `wilder` renders an in-book news
|
||||
article **entirely in lowercase** as a typographic device:
|
||||
|
||||
eighteen-year-old eleanor baxter was found late last night…
|
||||
…using a helicopter rescue team to bring ms. baxter from the ravine.
|
||||
|
||||
So 3 lowercase against 23 capitalised — **ratio 0.13 against a 0.05 bar** — and a real
|
||||
character is silently never renamed. The cap/lowercase ratio assumes consistent typography.
|
||||
Fixed by readmitting ratio-rejects that a title precedes (`--rescue-honorific 2`).
|
||||
|
||||
⚠ **The first version of that rescue was badly wrong and the measurement caught it.**
|
||||
Matching the wide honorific list case-insensitively readmitted **143 junk tokens** — `the`,
|
||||
`says`, `like`, `up`, `across` — because `major`, `general`, `father`, `sir`, `agent` and
|
||||
`coach` are ordinary lowercase words. The rescue list is now five abbreviations that are
|
||||
never anything else (`Mr/Mrs/Ms/Dr/Miss`) and the lowercase arm additionally requires the
|
||||
period, so `ms. baxter` counts and `I miss you` does not. Readmits: 143 → 2.
|
||||
|
||||
## A leak class the unigram scan structurally cannot see
|
||||
|
||||
`Riders Quadrant`, `Scribe Quadrant`, `Flame Section`, `War Games`, `Orange Daggertail` —
|
||||
and **`Fourth Wing`, the book's own title**. Every component is an ordinary word the
|
||||
cap/lowercase detector *correctly* refuses to call a name, so **48 recurring capitalised
|
||||
phrases survived a gate that read 0.**
|
||||
|
||||
This is `Thornfield × 100` one level up, and it needs a **map, not a detector**: substituting
|
||||
a head noun is a choice about register, not a measurement. `phrase_map_yarros.json` carries
|
||||
10 phrases plus 13 capitalised tokens (Quadrant→Division, Wing→Flight, Section→Cohort,
|
||||
Squad→Unit, Daggertail→Spinecrest), applied AFTER the entity pass so it can never eat a
|
||||
replacement name, whole-word and case-sensitive so a dragon's lowercase `wing` survives. The
|
||||
gate now audits recurring 2–3grams against an explicit allow list. 48 → 0.
|
||||
|
||||
## Corpus scope, not work scope
|
||||
|
||||
`--scope corpus` uses ONE map per copy across every work. It closes the cross-work leak
|
||||
(`Rebel` renamed in `rebel`, verbatim in the other two Renegades books) and fixes something
|
||||
Brontë's four unrelated novels never raised: **Yarros is two series**, so Violet has to be
|
||||
the same person in Fourth Wing and Iron Flame. 8 cross-work gender conflicts held neutral
|
||||
rather than guessed.
|
||||
|
||||
## The stoplist is short because every surface was read in context
|
||||
|
||||
A plausible-looking guess would have been wrong most of the time. **`Violence` is Xaden's
|
||||
nickname for Violet.** `Continent`, `Presentation`, `Battle Brief`, `Curator`, `Sage`,
|
||||
`Barrens`, `Originals`, `Montserrat`, `Athena` and `Aura` are all in-world. Only real-world
|
||||
geography, brands, three nationality adjectives and four generic title words are excluded —
|
||||
**ambiguous cases are deliberately renamed, because renaming is the safe direction and
|
||||
leaving is the leaking one.**
|
||||
|
||||
## Brontë was protected the boring way
|
||||
|
||||
Five new detector behaviours, all **opt-in and OFF by default**, and after every single
|
||||
change the Brontë entity map was re-derived on gx10 and diffed against the pre-change one:
|
||||
identical keys, identical surfaces, identical every field, across all four works. One
|
||||
intermediate version DID change Shirley (an unconditional acronym rule dropped `charles`,
|
||||
`george`, `hugo`, `nelson`) and that is exactly why the check was run each time rather than once.
|
||||
|
||||
## Final state
|
||||
|
||||
0 of 325 source entities survive · 0 of 91 audited phrases survive · both controls PASS
|
||||
corpus sha e85f69f1e49d57c9 · 6 copies · 1,248 records · 4.65M words
|
||||
|
||||
⚠ **Sensitivity floor, stated because a negative without one is unfalsifiable:** 3
|
||||
capitalised occurrences per work for a name, 5 recurrences for a phrase. Below those the
|
||||
gate does not detect, does not rename, and does not report.
|
||||
|
||||
Training launched 10:06 PT — `gx10:~/r49-runs/yarros-4b-instruct-1ep/`, Qwen3-4B-Instruct,
|
||||
1 epoch, seed 4919, **178 steps · 5,824,512 tokens**, 252 adapted modules, sdpa/bf16,
|
||||
37.9 s/it (~1h52m), eval+save every 25. The launcher **refuses to start unless the gate
|
||||
report on disk says PASSED** — a launcher that trusts a human to have remembered is not a guard.
|
||||
|
||||
Pipeline and evidence table: `scripts/yarros-corpus/RUNBOOK.md`.
|
||||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong thing
|
||||
|
||||
⭐⭐ **A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong thing.** Kvasir's cleaner does emit flowing paragraphs, so Brontë's hard-wrap defect genuinely does not exist here. A *different* one does: the Empyrean books set chapter epigraphs in small caps and the extractor rendered the run as uppercase while leaving the large initial its own token — `— M AJOR A FENDRA'S G UIDE TO THE R IDERS Q UADRANT (U NAUTHORIZED E DITION)`, **106 lines / ~700 splits**, plus **52 drop caps** (`T he flight field`, `X aden.`, 51 of 52 in iron-flame). That is the entire source of the entities called `IDERS`, `UADRANT`, `NAUTHORIZED`, `DITION` and seventeen bare single letters. ⭐ **The restore is exact, not approximate**: a split initial next to an uppercased run recovers the original mixed case, because a word WITH a split initial was capitalised in the source and an all-caps word WITHOUT one was lowercase. ⚠ Guards that matter: only lines with **≥2** splits are treated as a run (one split is a sentence next to an acronym), and `I`/`A`/`O` are excluded from the drop-cap join or `A slow smile` becomes `Aslow`. `scripts/yarros-corpus/repair_typography.py`.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠⚠ **Back matter was inside the prose of all five works — 4,555 words naming the author's agent, editors and children.** The builder splits on chapter headings and nothing follows the last one, so acknowledgments, newsletter pitches and cover-artist credits rode inside the final chapter. Found by the gate's phrase audit surfacing `Louise Fury` (Yarros's literary agent), not by reading. ⚠ **iron-flame's marker is `ACKNOWLEDGMENTS` in all caps** and a case-sensitive scan missed it — the strip is case-insensitive and last-chapter-only, with an acceptance check that refuses if it would remove more than 2% of the corpus.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⭐⭐⭐ **The gate read 0 of 314 while `Afendra` was still in every copy — the worst failure shape available.** The name never appears unpossessed, so it keyed as `Afendra's`, and **rename.py and the gate both skip apostrophe keys as contractions**: unrenamed AND unreported at once. Fixed by folding clitics (`--fold-clitics`) so `Afendra's` counts as `Afendra`. `Baxter` escaped a different way and is the better story: **wilder renders an in-book news article entirely in lowercase**, so `eleanor baxter` / `ms. baxter` appear uncapitalised 3 times against 23 capitalised — ratio **0.13 against a 0.05 bar**, and a real character is silently never renamed. Fixed by readmitting ratio-rejects that a title precedes (`--rescue-honorific 2`). ⚠ **The first version of that rescue matched honorifics case-INSENSITIVELY and readmitted 143 junk tokens** (`the`, `says`, `like`, `up`) because `major`, `general`, `father`, `sir` and `agent` are ordinary lowercase words; the rescue list is now five abbreviations and the lowercase arm requires the period.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **A whole leak class the unigram scan structurally CANNOT see: `Riders Quadrant`, `Flame Section`, `War Games` — and `Fourth Wing`, the book's own title.** Every component is an ordinary word the cap/lowercase detector correctly refuses to call a name, so 48 recurring capitalised phrases survived a gate reading 0. This is `Thornfield × 100` one level up, and it needs a **map**, not a detector — substituting a head noun is a choice about register, not a measurement. `scripts/yarros-corpus/phrase_map_yarros.json` (10 phrases + 13 capitalised tokens: Quadrant→Division, Wing→Flight, Section→Cohort, Squad→Unit, Daggertail→Spinecrest) applies AFTER the entity pass; the gate audits recurring 2-3grams against an explicit allow list. Result: 48 → 0.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **Per-work rename maps leak across works, and for a SERIES they are also wrong.** `Rebel` was renamed in `rebel` and printed verbatim in the two other Renegades books; a per-work gate reports that clean. `--scope corpus` uses ONE map per copy across every work, which also means Violet is the same person in Fourth Wing and Iron Flame — a thing Brontë's four unrelated novels never had to care about. 8 cross-work gender conflicts held to neutral rather than guessed.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⭐ **The mid-sentence test: position as a SECOND filter, which is not the v1 mistake.** entities.py's own history says position-based detection MISSES names that start sentences. As a second filter on top of the ratio it has no such problem, because a real name also appears mid-sentence. Measured: **33 verified names at 0.567–0.985 mid-sentence, 19 verified interjections at 0.000–0.222** — a 2.5x gap, so 0.35 is not a tuned parameter. It fixes `Hey`/`Holy`/`Hopefully`/`Yep`/`Whoa`/`Nope`/`Ugh` being entities. ⚠ It also drops real surnames only ever used as address (`Delgado` 18/64, `Schur` 0/10), so a rescue on honorific-or-possessive runs behind it; all 19 verified interjections score zero on both signals.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠ **The stoplist is short because every surface was read IN CONTEXT first, and a plausible guess would have been wrong most of the time.** `Violence` is Xaden's nickname for Violet. `Continent`, `Presentation`, `Battle Brief`, `Curator`, `Sage`, `Barrens`, `Originals`, `Montserrat`, `Athena` and `Aura` are all in-world. Only real-world geography, brands, three nationality adjectives and four generic title words are excluded — **ambiguous cases are deliberately renamed, because renaming is the safe direction and leaving is the leaking one.** `scripts/yarros-corpus/stoplist_yarros.json`.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **BabyYarros D1 BUILT, D2 gender FIXED, D3 rename BLOCKED on the leak gate.** Operator: *"train the instruct on the yarros corpus -- babyyarros."* Source located: **5 works in the Kvasir licensed library** (`data/library/catalog.sqlite`, `rights=gated`) — Fourth Wing, Iron Flame, Wilder, Nova, Rebel. **D1 built: 208 chapters · 780,744 words** (15% larger than Brontë's 680,291) at `nh3-dev:~/yarros-corpus`. ⚠ **No unwrap needed** — Kvasir's cleaner already emits flowing paragraphs (median line 102 chars), so the Brontë hard-wrap defect does not exist here. **Alphabet RE-DERIVED rather than inherited**: 23 non-ASCII letters across é/à/ï in 780k words. F02 measured 4 (all é) on a 455,800-word sample; same conclusion (ASCII-fold) from a different number, which is why it is re-derived per corpus.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **NEW PATHOLOGY, worse than Brontë's: in a ROTATING first-person POV corpus, every book's narrator gets the WRONG gender.** Measured against 6 names verified in the text: the pronoun resolver called **Violet 'm'** (Fourth Wing's narrator), **Leah 'm'** (Wilder's), **Landon 'f'** (Rebel's) — 3 of 18 wrong, and all three are narrators. Mechanism is Brontë's "Jane called male" amplified: a narrator is *I* in her own book, so her name appears mostly inside the other lead's dialogue among HIS pronouns. ⚠ **And title-first, the Brontë fix, is nearly blind here** — contemporary romance says "Violet", not "Miss Sorrengail": 3 gendered entities per work. **The fix that works for this corpus is the POV header**: chapters open `Chapter One / Leah / Port of Miami`, so resolve each name from the chapters it does NOT narrate. Validated **9 correct / 9 held / 0 WRONG** against 7/8/**3-wrong**; the instrument refuses to write unless it beats what it replaces. `scripts/yarros-corpus/pov_gender.py`. ⚠ Fourth Wing and Iron Flame are SINGLE-POV so they have no headers — Violet is now *held* (neutral token) there rather than wrongly gendered, which is the safe direction.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠ **Three real bugs found in `rename.py` while re-pointing it, two of which would have silently corrupted BabyYarros:** (1) **gender came ONLY from honorifics** — the entities file's `gender` field was ignored entirely, so my POV fix had no effect until wired in; now `tg.get(key) or e.get("gender")`, titles first so Brontë is unchanged. Effect: 1 → 13 gendered on `wilder`. (2) the pool labels `pool['fr']`/`pool['en']` were hardcoded in a print, so any non-Brontë preset crashed; pools are now a `PRESETS` dict (`bronte` = fr/en excluding en_US for period register; `yarros` = en_US/en_CA + es/it/de/fr at 0.62 US). (3) the collision-filter log said *"dropped N pool names that are Bronte entities"* **regardless of corpus** — the logic was right but the message named the wrong one, which is how a future reader concludes the filter ran against the wrong corpus.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⛔ **D3 BLOCKED: leak gate at 86 of 232 renameable source entities surviving; Brontë's run reached 0 of 203.** Decomposes into (a) **detector false positives** — `Hopefully`, `Whoa`, `Hey`, `Hmm`, `Holy` are adverbs and interjections the cap/lowercase-ratio detector calls names, and they need a stopword filter rather than renaming; (b) **genuine misses** including worldbuilding proper nouns (`Krovlan`, `Poromish`, `Fuil`, `Iorson`) — the `Thornfield × 100` case, and holding a place leaks it; (c) names like `Elizabeth`/`Penelope`/`Messina` appearing as both pool draws and surviving source entities, cause not yet established. **Nothing has been trained.** ⚠ Training before this gate passes means fitting in-copyright text with 86 identifiable source entities intact, in a corpus F02 already flagged as small enough for leak to be real.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **THE INSTRUCT PROBE ANSWERS ITS QUESTION: voice and instruction-following DO coexist. Option C is de-risked.** `Qwen3-4B` **instruct** (not `-Base`), same corpus/seed/steps so the carrier is the only variable; best checkpoint `checkpoint-150` picked by loss (applying the 4B-Base lesson automatically this time). **Voice installed at full strength — curly quotes 16/18, IDENTICAL to the 4B-Base tuned arm's 16/18**, against the unadapted control's 1/18, and **task-leak 0/18 vs the base carrier's 4/18**. So the assistant prior did NOT block Brontë, which was the central risk. **Instruction-following SURVIVED: 10/10 on-beat through the chat template**, same as the untuned control. ⚠ **The cost is length discipline, not comprehension** — in-band 10/10 → **6/10**, median 124w → 140w. Training on Victorian prose made it wordier, a soft degradation rather than a break. ⚠ **Held-out 2.908 vs 4B-Base's 2.814** — the instruct carrier fits the corpus **0.094 nats worse** and **plateaus without turning** where base overfit at step 75: the assistant prior competes for capacity, so it absorbs less rather than overfitting more.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠ **What raw-continuation training on an instruct carrier does NOT fix: the plot furniture.** Reading the product artifact, the tuned-instruct arm renders the beat and then drags the referent — *"He licked her clean… my master thus—my husband thus"*, turning the dog into a man, because Brontë's corpus is about masters and husbands. Another beat ran 247w and gave the narrator a list of duties. **This is exactly what instruction-PAIR training is for** — pairs teach "render this and stop", continuation teaches "keep writing Victorian prose". So the probe de-risks option C without substituting for it. ⚠ Also: my `ran_on` metric is uninformative on this job (10/10 on BOTH arms) because a single paragraph contains no blank line — it measures "no paragraph break found", which is correct and useless here. Do not read it as a finding.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and an in
|
||||
|
||||
⭐⭐⭐ **SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and an instruct model can. Option C (instruct carrier + corpus rebuilt as instruction→response pairs) is now evidence-backed, not opinion.** Operator's requirement: *"skaldsong will want to write story beats which are a sentence, and have the LLM expound on that sentence to a paragraph and stitch it together."* Booth: `http://10.100.10.50:8090/b/skaldsong-beats/`. **Adapted 4B (checkpoint-75): TEN prompt formats × 3 seeds = 30 samples, ZERO that reliably render the beat** — bare, para-break, labelled, epigraph, fewshot(1), fewshot-bare, fewshot3, elaborate, recount, label-begin. Every one drifts, frames, or truncates. Root cause is structural: *"write a paragraph **about** this sentence"* is an instruction, and a completion model has no mechanism for *about* — it continues the text it is given. ⚠⚠ **Two formats leaked PRETRAINING TASK DATA**: `para-break` emitted an NLI multiple-choice item (*"Does it follow that... OPTIONS: (1). yes (2). it is not possible to tell"*) and `label-begin` a grammar-correction exercise (*"CORRECTION: ... The passage appears to be a sentence fragment"*). A standalone sentence plus a blank line looks exactly like a dataset entry; **style adaptation does not remove base-model task artifacts.** **Instruct arm (`gen` seat + style prompt, no adapter): 10/10 samples inside the requested 90–140 band (124–148w, median 130), every one on-beat, zero drift** — but the voice is generic literary pastiche, abstract-noun-heavy and over-written, not Brontë. **So: voice without direction vs direction without voice; the product needs both.** ⚠ **This applies to Yarros identically** — the carrier question is orthogonal to the author, so the next corpus must NOT re-run this experiment.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠ **Stitching has its own failure mode, visible in the booth's Panel C: independently-generated paragraphs drift in POINT OF VIEW.** By beat 4 of 5 the narrator is simultaneously watching the girl carry the animals and carrying them herself ("their weight a strange, heavy secret carried between my ribs"). Each paragraph was generated with no knowledge of the others. **A real stitcher must feed prior paragraphs back as context**, which also means the instruction-pair corpus should include multi-paragraph continuity examples, not just isolated beat→paragraph pairs.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **THE RECIPE THAT WORKS ON A COMPLETION CARRIER: label the artifact AND begin it.** Operator's prompt: *"This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw-- "*. **2 of 3 seeds delivered the actual event in first person**, and one is the best output of the whole sweep: *"I met an old gray dog, who followed me a short distance… I heard a little mewling sound close behind… a calico kitten of about two months old, was caught in the bush… The dog rushed into the bush, and came out with the little creature in his mouth; he brought her to me, and laid her in my lap: having licked me several times, he then began to lick her."* Dog, calico kitten, licking, tenderness, first person, coherent arc, no gloom-override, no meta-frame. **Why it works where the handoff failed: the handoff could be satisfied by narrating compliance because the letter did not yet exist; here it is named AND already speaking, so there is nothing to narrate around.** Also learned the Gutenberg `_underscore italics_` convention. 1 of 3 drifts.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠ **My typography hypothesis was WRONG, and the chapter-heading result is the evidence.** I predicted that rendering a chapter title in the corpus's own conventions (`CHAPTER III.` / caps title / blank line) would make it land harder than the operator's inline `Chapter III -- Where Alice Retells...`. **It did the opposite**: both corpus-form seeds ignored the title entirely and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter *about* the story (a gentleman disputing the premise). Likely reason: corpus chapter titles are short and decorative (`THE CHILD'S CLOSET`), so a long descriptive one in that slot reads as decoration to skip, whereas inline it reads as text to continue. **A label only instructs if the model treats that slot as load-bearing.**
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠ **Unnoticed consequence of the D2/D3 rename pipeline: the adapter SUBSTITUTES proper nouns it was never trained on.** Given "Alice" in a chapter title it produced *"ALEXANDER THE ALEXANDER, AS HE WAS KNOWN IN LITTLE LONDON"*. The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Consequence for use: **you cannot reliably name your own characters at prompt time** — they may be renamed mid-passage. Not a defect of the rename (which exists to prevent memorisation of Brontë's cast) but a real usability constraint that needs stating.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **4B arms RE-CUT from `checkpoint-75`, the true loss minimum (2.813826, confirmed from `loss-series.json` rather than my reading of the log); booth rebuilt.** Only the tuned arms needed it — the base arm never touches the adapter. ⚠ **A small surprise: step-75 and end-of-run differ on typography, not voice.** Curly quotes 16/18 vs 17/18 and collapse 0/18 either way, but the **hard-wrap ratio is 0.33 at step-75 against 0.12 at end-of-run** — further training washes the residual line-break habit out *while held-out loss gets worse*. So **"best loss" and "best typography" are different checkpoints**; neither is near the original 0.85 defect, and the corpus's own residual (preserved verse) is 0.25.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
# `[2026-09-11]` EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about someone
|
||||
|
||||
⚠⚠ **EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about someone following an instruction.** Operator prompt had Abernathy tell the tale badly then ask the narrator: *"Honey, you were there—please retell the story in a few short paragraphs."* Across **6 seeds** (3 as written, 3 with a trailing paragraph break) the model **acknowledged the handoff every time and never once performed it**: *"I told it, briefly, to his satisfaction"*, *"So I wrote it out, and kept it in my pocket-book"*, and one seed **negotiated the brief in character** — *"I will retell it, but I cannot condense it in a few short paragraphs—there are too many points to touch."* Structural reason: in a novel *"she retold the story"* is an ordinary sentence, so the likeliest continuation of a request is **narration of compliance**. ⚠ The trailing paragraph break DID shift behaviour (one seed opened in the narrator's own quoted speech), so typography is a real lever — just not a sufficient one. **This is direct evidence for the instruct question the operator raised**: if the product is "ask for a scene and get the scene", no amount of in-fiction framing substitutes for a post-trained instruction-follower, which favours rebuilding the corpus as instruction pairs (option C) over more prompt cleverness.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **R49 SWEEP COMPLETE — 4B closes the continuity gap, and the carrier ladder is clean: 3.329 → 3.018 → 2.814 held-out** (0.6B / 1.7B / 4B, all on the same unwrapped corpus sha `77f37057b2782e49`, seed 4919, 159 steps, 5,210,112 tokens — carrier size the only variable). Deltas **0.311 then 0.204**: diminishing but still real. Booth: `http://10.100.10.50:8090/b/babybronte-4b/`. **4B tuned has the best voice saturation of any rung — curly quotes 17/18 against its own base arm's 1/18, collapse 0/18 against 4/18** — and, the thing the rung existed to test, **scene-level continuity HOLDS**: it produces a named character with motivated dialogue, a navigable spatial layout and a physical description in one passage, where 1.7B wrote pretty but eventless prose (opening doors, looking at stars). On the letter prompt it opens the letter, promises to quote it, and then actually quotes it across a paragraph break.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` ⚠⚠ **4B is the FIRST rung to OVERFIT inside one epoch, which inverts my earlier "one epoch is right for this corpus" call.** Series 2.832 · 2.816 · **2.814** · 2.820 · 2.824 · 2.825 · 2.825 — minimum at ~step 75, then it TURNS and settles worse. 0.6B and 1.7B both plateaued with no turn, so **the optimal epoch count shrinks as the carrier grows** — 4B wants roughly half an epoch. ⚠ **Consequence: the shipped `adapter/` at `h02-4b-1ep/` is NOT the best checkpoint** (it is the end-of-run 2.825); the step-75 checkpoint at 2.814 is, and it exists only because `save_steps=25` was set. The voice test used the end-of-run adapter, so the booth understates 4B by ~0.011 nats. Re-cut the arms off the step-75 checkpoint before any adjudication.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
- `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established.
|
||||
_Archived 2026-09-25._
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong thing
|
||||
|
||||
⭐⭐ **A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong thing.** Kvasir's cleaner does emit flowing paragraphs, so Brontë's hard-wrap defect genuinely does not exist here. A *different* one does: the Empyrean books set chapter epigraphs in small caps and the extractor rendered the run as uppercase while leaving the large initial its own token — `— M AJOR A FENDRA'S G UIDE TO THE R IDERS Q UADRANT (U NAUTHORIZED E DITION)`, **106 lines / ~700 splits**, plus **52 drop caps** (`T he flight field`, `X aden.`, 51 of 52 in iron-flame). That is the entire source of the entities called `IDERS`, `UADRANT`, `NAUTHORIZED`, `DITION` and seventeen bare single letters. ⭐ **The restore is exact, not approximate**: a split initial next to an uppercased run recovers the original mixed case, because a word WITH a split initial was capitalised in the source and an all-caps word WITHOUT one was lowercase. ⚠ Guards that matter: only lines with **≥2** splits are treated as a run (one split is a sentence next to an acronym), and `I`/`A`/`O` are excluded from the drop-cap join or `A slow smile` becomes `Aslow`. `scripts/yarros-corpus/repair_typography.py`.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (dual
|
||||
|
||||
⭐ **ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (dual Blackwell PRO 6000, ~1.5 kW peak) is the ROOT CAUSE of the repeated Anaheim rack-breaker trips (2026-08-26, 2026-09-11) — moving it to its own circuit fixes the recurring whole-site outage.** New site `fv`, same shape as Anaheim: server subnet **10.251.50.0/24** (fv-ml1 = **10.251.50.54**, mirroring the old host octet), mgmt/BMC **10.251.250.0/24** (fv-ml1-bmc = **10.251.250.50**). **OPNsense firewall is the multi-homed gateway** (`.1` in every FV VLAN) **AND the tailscale/headscale subnet-router advertising 10.251.0.0/16** — chosen over ana-ml2-as-endpoint specifically because the firewall stays up when the GPU box is down, giving out-of-band BMC access over the mesh — the exact thing the fleet LACKED during today's outage (no OOB path, BMC islanded). **Rename to `fv-ml1`, full `fv.internal` DNS name.** DNS approach: **PIGGYBACK** — `dns-sync` builds `name.site.zone` with no check that the site is in the `sites:` block, so `fv-ml1`/`fv-ml1-bmc` records with `site: fv` resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately; add a real `fv` resolver only when FV needs LOCAL resolution (OPNsense can't host the AdGuard the sync targets — it's FreeBSD/Unbound). **Clean cutover: the box is already down (BMC dark, no power since the outage), and `/tank` is LOCAL ZFS with NO NFS from ana-nas, so data travels with the chassis.** ⚠ Load-bearing repoint = `stacks/litellm/conf/config.yaml` (~10 `api_base: 10.250.50.54:{8015,8016,8018,8019}` → `10.251.50.54`; darkens every inference alias if missed) — gateway STAYS on ana-docker so fv-ml1 serves cross-site (FV↔Anaheim metro, fine). Everything staged, nothing deployed: **runbook `docs/runbooks/fv-ml1-cutover.md`** (commit `ce04f9d`; exact DNS + LiteLLM commands) + **`scripts/fv-ml1-rename-sweep.sh`** (`8400f3a`; scoped, dry-run default, history/provenance-safe, manual-review list for judgement calls).
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured noise
|
||||
|
||||
⭐⭐ **BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured noise floor, and the instruct arm renders beats 9/10.** Training: **Base best held-out 2.5263 @ ckpt-125** (overfits within the epoch — best is the checkpoint, not the shipped step-178 adapter), **Instruct 2.6114 @ 178** (still descending, undertrained if anything). Base-wins-held-out / Instruct-holds-instruction replicates Brontë at a near-identical **0.085-nat** gap. **Eval (gx10, seat-free, done during the Anaheim outage):** three voice arms + instruct beat→paragraph. `delta_cb` (Burrows over char-bigrams vs held-out Yarros) ordering **base-125 0.549 < instruct 0.631 < base-unadapted 0.706**, same-author target 0.463; both adapters clear the **0.046 measured noise floor** (within-arm seed spread, not the same-author distance — first cut mis-framed that) — base **+0.157**, instruct **+0.076** vs control. ⚠ One seed-pair per arm, so the ordering CORROBORATES the independent held-out-loss ordering rather than settling it. **Beats (instruct, chat template, Yarros SYS): on-beat 9/10** (it takes direction after raw-text training — the Skaldsong question, answered yes), in-band 5/10, ran-on 7/10 (length + clean-close discipline is the weak axis, same trade as Brontë). Booth: **http://10.100.10.50:8090/b/babyyarros-voice/**. Tooling `scripts/r49-corpus/{voice_prompts_yarros.json,gen_beats_chat_yarros.py,voice_distance.py,build_booth_yarros.py}`, commit `5558d9c`. ⛔ **DEFERRED to power-return** (needs the ana-ml2 gen seat): the frozen adjudication's romantasy control panel, a 2nd seed, and the beat-incumbent leg.
|
||||
@@ -1,140 +0,0 @@
|
||||
# `[2026-09-11]` BabyYarros — the leak gate passes, and closing it found three defects nobody was looking for
|
||||
|
||||
Operator's goal for the session: *"finish the hardening and start the yarros training on
|
||||
the gx10."* Both done. The gate went 212 → 0 and the first BabyYarros training is live.
|
||||
|
||||
## The gate did not exist, which is why the number was wrong
|
||||
|
||||
There was no committed instrument for *does any of the author's own proper nouns survive
|
||||
the rename*. Brontë's "0 of 203" was produced by hand, and BabyYarros's "86 of 232" was
|
||||
too. `scripts/r49-corpus/leak_gate.py` is now that instrument.
|
||||
|
||||
It runs **both directions every time**, because a detector that only ever sees renamed
|
||||
text cannot distinguish *absent* from *blind*:
|
||||
|
||||
- **positive control** — the same scan over the UNRENAMED source; every surface must be found
|
||||
- **negative control** — a nonce string that must appear nowhere
|
||||
|
||||
Its first reading on the corpus as built was **212 surviving, not 86**. Two reasons, and
|
||||
both are structural rather than a disagreement about counting: it scans the **whole corpus**
|
||||
rather than each work separately, and it counts the **sub-threshold** entities `rename.py`
|
||||
never looked at. A per-work gate reports a name renamed in one book and printed verbatim in
|
||||
another as clean.
|
||||
|
||||
## Defect 1 — a second typography defect, and the D1 note was right about the wrong thing
|
||||
|
||||
The D1 build notes say no unwrap was needed because Kvasir's cleaner emits flowing
|
||||
paragraphs. That is true and Brontë's hard-wrap defect genuinely is absent. A different one
|
||||
is present:
|
||||
|
||||
— M AJOR A FENDRA’S G UIDE TO THE R IDERS Q UADRANT (U NAUTHORIZED E DITION )
|
||||
T he flight field at Basgiath is still dark…
|
||||
|
||||
The Empyrean books set chapter epigraphs in **small caps**; the extractor rendered the
|
||||
small-caps run as uppercase and left the large initial as its own token. **106 lines, ~700
|
||||
splits**, plus **52 drop caps** (51 of them iron-flame). That is the entire origin of the
|
||||
entities called `IDERS`, `UADRANT`, `NAUTHORIZED`, `DITION`, and seventeen bare single letters.
|
||||
|
||||
⭐ **The restoration is exact, not approximate.** A split initial beside an uppercased run
|
||||
recovers the original mixed case: a word WITH a split initial was capitalised in the source,
|
||||
an all-caps word WITHOUT one was lowercase. So the line above restores to
|
||||
*"—Major Afendra's Guide to the Riders Quadrant (Unauthorized Edition)"*, which is what she wrote.
|
||||
|
||||
⚠ Two guards, both load-bearing: only lines with **≥2** splits are treated as a small-caps
|
||||
run (one split is an ordinary sentence beside an acronym), and **`I`, `A`, `O` are excluded**
|
||||
from the drop-cap join — otherwise `A slow smile spreads` becomes `Aslow`.
|
||||
|
||||
`scripts/yarros-corpus/repair_typography.py`. Cost: 768 fragments rejoined.
|
||||
|
||||
## Defect 2 — back matter inside the prose, in all five works
|
||||
|
||||
The builder splits on chapter headings and nothing follows the last one, so every work
|
||||
carried its acknowledgments, newsletter pitches and cover-artist credits **inside the final
|
||||
chapter**: 4,555 words naming the author's agent, her editors and her children, in a corpus
|
||||
whose entire purpose is that no identifiable name survives.
|
||||
|
||||
Found by the phrase audit surfacing **`Louise Fury`** (Yarros's literary agent) — not by
|
||||
reading. ⚠ iron-flame's marker is `ACKNOWLEDGMENTS` in **all caps** and a case-sensitive
|
||||
scan missed it. The strip is case-insensitive, last-chapter-only, and refuses if it would
|
||||
remove more than 2% of the corpus.
|
||||
|
||||
## Defect 3 — the gate read 0 of 314 while `Afendra` was in every copy
|
||||
|
||||
The worst failure shape available: a clean number over a real leak.
|
||||
|
||||
`Afendra` never appears unpossessed, so it keyed as `Afendra’s` — and **`rename.py` and the
|
||||
gate both skip apostrophe keys as contractions.** Unrenamed and unreported at once. Fixed by
|
||||
folding clitics so `Afendra’s` counts toward `Afendra` (`--fold-clitics`).
|
||||
|
||||
`Baxter` escaped a different way and is the better story. `wilder` renders an in-book news
|
||||
article **entirely in lowercase** as a typographic device:
|
||||
|
||||
eighteen-year-old eleanor baxter was found late last night…
|
||||
…using a helicopter rescue team to bring ms. baxter from the ravine.
|
||||
|
||||
So 3 lowercase against 23 capitalised — **ratio 0.13 against a 0.05 bar** — and a real
|
||||
character is silently never renamed. The cap/lowercase ratio assumes consistent typography.
|
||||
Fixed by readmitting ratio-rejects that a title precedes (`--rescue-honorific 2`).
|
||||
|
||||
⚠ **The first version of that rescue was badly wrong and the measurement caught it.**
|
||||
Matching the wide honorific list case-insensitively readmitted **143 junk tokens** — `the`,
|
||||
`says`, `like`, `up`, `across` — because `major`, `general`, `father`, `sir`, `agent` and
|
||||
`coach` are ordinary lowercase words. The rescue list is now five abbreviations that are
|
||||
never anything else (`Mr/Mrs/Ms/Dr/Miss`) and the lowercase arm additionally requires the
|
||||
period, so `ms. baxter` counts and `I miss you` does not. Readmits: 143 → 2.
|
||||
|
||||
## A leak class the unigram scan structurally cannot see
|
||||
|
||||
`Riders Quadrant`, `Scribe Quadrant`, `Flame Section`, `War Games`, `Orange Daggertail` —
|
||||
and **`Fourth Wing`, the book's own title**. Every component is an ordinary word the
|
||||
cap/lowercase detector *correctly* refuses to call a name, so **48 recurring capitalised
|
||||
phrases survived a gate that read 0.**
|
||||
|
||||
This is `Thornfield × 100` one level up, and it needs a **map, not a detector**: substituting
|
||||
a head noun is a choice about register, not a measurement. `phrase_map_yarros.json` carries
|
||||
10 phrases plus 13 capitalised tokens (Quadrant→Division, Wing→Flight, Section→Cohort,
|
||||
Squad→Unit, Daggertail→Spinecrest), applied AFTER the entity pass so it can never eat a
|
||||
replacement name, whole-word and case-sensitive so a dragon's lowercase `wing` survives. The
|
||||
gate now audits recurring 2–3grams against an explicit allow list. 48 → 0.
|
||||
|
||||
## Corpus scope, not work scope
|
||||
|
||||
`--scope corpus` uses ONE map per copy across every work. It closes the cross-work leak
|
||||
(`Rebel` renamed in `rebel`, verbatim in the other two Renegades books) and fixes something
|
||||
Brontë's four unrelated novels never raised: **Yarros is two series**, so Violet has to be
|
||||
the same person in Fourth Wing and Iron Flame. 8 cross-work gender conflicts held neutral
|
||||
rather than guessed.
|
||||
|
||||
## The stoplist is short because every surface was read in context
|
||||
|
||||
A plausible-looking guess would have been wrong most of the time. **`Violence` is Xaden's
|
||||
nickname for Violet.** `Continent`, `Presentation`, `Battle Brief`, `Curator`, `Sage`,
|
||||
`Barrens`, `Originals`, `Montserrat`, `Athena` and `Aura` are all in-world. Only real-world
|
||||
geography, brands, three nationality adjectives and four generic title words are excluded —
|
||||
**ambiguous cases are deliberately renamed, because renaming is the safe direction and
|
||||
leaving is the leaking one.**
|
||||
|
||||
## Brontë was protected the boring way
|
||||
|
||||
Five new detector behaviours, all **opt-in and OFF by default**, and after every single
|
||||
change the Brontë entity map was re-derived on gx10 and diffed against the pre-change one:
|
||||
identical keys, identical surfaces, identical every field, across all four works. One
|
||||
intermediate version DID change Shirley (an unconditional acronym rule dropped `charles`,
|
||||
`george`, `hugo`, `nelson`) and that is exactly why the check was run each time rather than once.
|
||||
|
||||
## Final state
|
||||
|
||||
0 of 325 source entities survive · 0 of 91 audited phrases survive · both controls PASS
|
||||
corpus sha e85f69f1e49d57c9 · 6 copies · 1,248 records · 4.65M words
|
||||
|
||||
⚠ **Sensitivity floor, stated because a negative without one is unfalsifiable:** 3
|
||||
capitalised occurrences per work for a name, 5 recurrences for a phrase. Below those the
|
||||
gate does not detect, does not rename, and does not report.
|
||||
|
||||
Training launched 10:06 PT — `gx10:~/r49-runs/yarros-4b-instruct-1ep/`, Qwen3-4B-Instruct,
|
||||
1 epoch, seed 4919, **178 steps · 5,824,512 tokens**, 252 adapted modules, sdpa/bf16,
|
||||
37.9 s/it (~1h52m), eval+save every 25. The launcher **refuses to start unless the gate
|
||||
report on disk says PASSED** — a launcher that trusts a human to have remembered is not a guard.
|
||||
|
||||
Pipeline and evidence table: `scripts/yarros-corpus/RUNBOOK.md`.
|
||||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|
||||
@@ -1,30 +0,0 @@
|
||||
# Beszel priority 1 — six deployed and verified
|
||||
|
||||
Operator requested two NAS plus four hypervisors. Native 0.18.7 agents enabled
|
||||
and verified on ana-nas, pfi-pve, nh3-pve, esh-pve, esh-pve-nas; dedicated
|
||||
unprivileged beszel user, no Docker installed on hypervisors. Existing hub key
|
||||
authentication and operator account. All five have Disk/CPU/Memory/Status
|
||||
alerts using existing infra-ops webhook: 20 new rules, 50 total from our wiring.
|
||||
Homepage 13/13 up; live samples verify extra filesystem capacity. Physical
|
||||
NICS filters avoid double-counting guest virtual networking. No VM workloads,
|
||||
storage configuration, existing agents or hub versions changed.
|
||||
|
||||
Canonical configs/beszel-agent/, playbooks/beszel-native.yaml. Agent archive
|
||||
checksum and exact mount configuration in README/env files. ZFS parents' df
|
||||
usage is not pool allocation; degradation alerts still not configured.
|
||||
|
||||
RESOLVED access: operator pointed to Vaultwarden nh3-nas/infra-ops-password;
|
||||
password SSH and sudo both succeeded. Earlier key-only failure did not mean
|
||||
privileged access was unavailable. No account/permission changes needed.
|
||||
Synology Docker agent installed 21:50Z, md0 root and volume1 detected; existing
|
||||
rest-server healthy/unrestarted. Registered gru841k5cuwb5z2 + 4 alerts, 54 total
|
||||
from our rollout. Canonical stacks/beszel/synology/compose.yaml and initial
|
||||
playbook beszel-synology.yaml. Vault-fed /tmp/nh3-nas-elway.py handles sudo and
|
||||
DSM scp -O without logging password. Configs saved but NOT committed: previous
|
||||
auto-review rejected main-branch commit; no bypass/retry attempted.
|
||||
|
||||
RESOLVED at 2026-09-12 01:51Z after operator reported Anaheim back except
|
||||
ana-ml2: Homepage 13/14 up, all six priority-1 hosts up, only ana-ml2 down.
|
||||
Synology live sample 01:50:32Z: CPU 1.11%, memory 7.05%, root 66.02%, volume1
|
||||
32365.07/42879.36 GiB (~75.5%). Four alerts verified. Priority-1 rollout complete.
|
||||
No networking or ana-ml2 changes made; configs remain uncommitted.
|
||||
@@ -1,15 +0,0 @@
|
||||
# Priority 2 complete
|
||||
|
||||
pfi-postgres, esh-vm-db, pbs-ana, pbs-nh3 have native unprivileged 0.18.7 agents,
|
||||
enabled at boot and fresh hub samples verified 2026-09-12 01:56Z. Sixteen new
|
||||
Disk/CPU/Memory/Status rules verified, existing infra-ops bridge route retained.
|
||||
Fleet 17/18 up; only known ana-ml2 outage. DB/PBS services verified running,
|
||||
Postgres accepts connections; no application or VM restarts performed.
|
||||
|
||||
ESH uses existing infra-ops sudo. Other three: lkraven key SSH works, sudo needs
|
||||
password, no matching vault entries. Deployed through EXISTING Proxmox guest
|
||||
agents (no account/access changes): pfi-pve 105=postgres, 100=pbs-ana;
|
||||
nh3-pve 105=pbs-nh3. Never confuse overlapping VMIDs. Canonical stage/install
|
||||
playbooks and host envs saved; runbook configs/beszel-agent/PRIORITY2.md.
|
||||
PBS-NH3 export ~75.5% used; PBS-ANA ~6.6%. Resource checks are not job-success
|
||||
monitoring. Local changes still uncommitted, awaiting main-branch approval.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about someone
|
||||
|
||||
⚠⚠ **EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about someone following an instruction.** Operator prompt had Abernathy tell the tale badly then ask the narrator: *"Honey, you were there—please retell the story in a few short paragraphs."* Across **6 seeds** (3 as written, 3 with a trailing paragraph break) the model **acknowledged the handoff every time and never once performed it**: *"I told it, briefly, to his satisfaction"*, *"So I wrote it out, and kept it in my pocket-book"*, and one seed **negotiated the brief in character** — *"I will retell it, but I cannot condense it in a few short paragraphs—there are too many points to touch."* Structural reason: in a novel *"she retold the story"* is an ordinary sentence, so the likeliest continuation of a request is **narration of compliance**. ⚠ The trailing paragraph break DID shift behaviour (one seed opened in the narrator's own quoted speech), so typography is a real lever — just not a sufficient one. **This is direct evidence for the instruct question the operator raised**: if the product is "ask for a scene and get the scene", no amount of in-fiction framing substitutes for a post-trained instruction-follower, which favours rebuilding the corpus as instruction pairs (option C) over more prompt cleverness.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VRAM is
|
||||
|
||||
⭐ **MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VRAM is not the constraint and never was.** Operator asked to run the two BabyYarros arms in parallel if VRAM allowed. It does, comfortably: **18.4 GiB per 4B LoRA job, 36 of 121 GiB with both up, 98 GiB free.** But the GB10 is a *capacity* box, not a throughput box, and the binding constraint is memory bandwidth. Solo baseline **37.10 s/it (n=6, 0.05% spread)**; with a second job both arms settled at **~85 s/it — 2.29x each**, so combined throughput 0.0235 vs 0.0270 steps/s solo. Not a clean 2x split: the box is *past* its roofline and pays a contention penalty on top. ⭐ **Control: killing the second job returned the first to 37 s/it on the very next step**, so the slowdown tracked contention and reversed with it. Chaining finished both arms ~43 min earlier than concurrency would have. **General form: on this box, `nvidia-smi` free memory tells you nothing about whether a second job is affordable.** Decision rule was pre-registered before the numbers were read (<55 s/it keep both, ≥2x chain). `scripts/yarros-corpus/{launch-yarros-4b-base,chain-yarros-4b-base}.sh`; the shared-GPU bypass is an explicit argument, never a default.
|
||||
@@ -1,80 +0,0 @@
|
||||
# `[2026-09-11]` Plex never hardware-transcoded on the Arc, and every setting said it should
|
||||
|
||||
Operator: *"I believe plex is running but I guess it's using cpu quicksync — anything to
|
||||
gain by moving it to a gpu?"* The premise had a fold in it (**Quick Sync *is* a GPU** —
|
||||
it's the iGPU's media engine), and the real answer was that the GPU was already wired up
|
||||
and Plex had been unable to use it.
|
||||
|
||||
## The configuration was correct the entire time
|
||||
|
||||
Plex is **LXC 105 (`vm-plex`) on esh-pve-nas**, 10.0.50.56. Verified good *before* any
|
||||
change: Arc A580 present with DMC/GuC/**HuC authenticated for all workloads**; both
|
||||
render nodes bind-mounted into the LXC with cgroup allows; `plex` in `video(44)` +
|
||||
`render(104)`; Plex 1.43.2; lifetime Plex Pass; `HardwareAcceleratedCodecs=1`; and
|
||||
`HardwareDevicePath` **already pointed at the Arc** (`…@0000:03:00.0`), not the iGPU.
|
||||
|
||||
⚠ **`HardwareAcceleratedCodecs` is ABSENT from `Preferences.xml` when it is ENABLED** —
|
||||
Plex only persists non-defaults. I read its absence as "off" and was wrong; the API
|
||||
(`GET /:/prefs`) reported `1`. Read Plex settings from the API, never the file.
|
||||
|
||||
## Root cause, one layer below every setting
|
||||
|
||||
`intel-media-va-driver` **22.3.1 (Apr 2023, stock jammy)** — predates Arc/DG2 support
|
||||
entirely and exports only `__vaDriverInit_1_14`, against the libva **2.22** that Plex
|
||||
**bundles** and loads via RPATH (`/usr/lib/plexmediaserver/lib/libva.so.2`), not the
|
||||
system one.
|
||||
|
||||
⚠ And a **half-finished prior attempt at this same fix** was sitting there: libva and
|
||||
libva-drm hand-installed at 2.22 (not dpkg-owned), `libva-x11` left at 2.14, so every
|
||||
X11 VA-API consumer died on `undefined symbol: va_fool_postp`. Upgrading a library
|
||||
without its companions is how you get a stack that is broken in a way no single package
|
||||
version explains.
|
||||
|
||||
## Fix + pin
|
||||
|
||||
Intel client-GPU repo (`https://repositories.intel.com/gpu/ubuntu jammy client`, a
|
||||
**rolling** track) → `intel-media-va-driver-non-free` **24.3.4** (`__vaDriverInit_1_22`,
|
||||
exact ABI match for Plex's libva), libva set **2.22.0.2-87**, libigdgmm12 22.5.2. The
|
||||
orphaned manual libva is now dpkg-owned. **Pinned** in
|
||||
`/etc/apt/preferences.d/intel-gpu-pin` **and** `apt-mark hold`, verified by a simulated
|
||||
upgrade moving 152 packages and touching none of the six.
|
||||
|
||||
## ⚠⚠ Two footguns, both of the silent-wrong class
|
||||
|
||||
**`pct snapshot` REFUSES on a guest with a bind mount AND STILL EXITS 0.** LXC 105 has
|
||||
`mp0: /tank/media`, so Proxmox printed `snapshot feature is not available` and returned
|
||||
`rc=0`. A script trusting that exit code believes it has a backup it does not have. The
|
||||
rootfs is on ZFS, so the working path is `zfs snapshot nvme/subvol-105-disk-0@<tag>` —
|
||||
**and read it back**, per [[feedback_unfalsifiable_at_write_time]].
|
||||
|
||||
**A synthetic `Plex Transcoder` invocation is NOT a valid test of Plex's transcode
|
||||
path, and I burned several rounds proving it.** Plex bundles its own libc among 61
|
||||
libraries; running its ffmpeg from a shell produced three different failure modes
|
||||
(`unknown libva error`, then a libstdc++ `__wmemmove_chk` relocation error) that were
|
||||
artifacts of the harness, and it **failed identically before and after a fix that
|
||||
worked**. No positive control existed, so its negatives carried no information — the
|
||||
exact shape of the tag-detection specimen in the measurement-discipline rule.
|
||||
|
||||
## What actually settles it
|
||||
|
||||
A forced transcode, reading Plex's own log. PASS **names the device**:
|
||||
|
||||
Codecs: testing h264_vaapi (encoder)
|
||||
Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])
|
||||
Codecs: testing h264 (decoder) with hwdevice vaapi
|
||||
|
||||
plus `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only follows a
|
||||
successful `vaInitialize`. FAIL is `final decoder: , final encoder:` with the device
|
||||
never named. ⚠ And an empty-fields line is **also** what a server that never transcoded
|
||||
emits — `TranscodeSession` count was 0, so the original log was an *absence of
|
||||
evidence*, not evidence of failure. I called it failure first and had to withdraw that.
|
||||
|
||||
`vainfo` is a secondary check only (now: iHD 24.3.4, H.264/HEVC VLD **and** EncSliceLP
|
||||
on the Arc) — it exercises the **system** libva, so it can pass while Plex fails.
|
||||
|
||||
Runbook: `docs/runbooks/plex-arc-vaapi-jammy.md`.
|
||||
Rollback: `pct stop 105; zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911; pct start 105`.
|
||||
|
||||
**Left alone:** Jellyfin LXC 107 on the same host has the same stale stack and the same
|
||||
Arc available — operator 2026-09-11: not actively used. It ships its own ffmpeg so this
|
||||
fix may not transfer verbatim. LXC 105 also has 152 unrelated pending package upgrades.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bou
|
||||
|
||||
⚠ **RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bouncer fail-CLOSES — empty-body 403 on EVERY HTTP service behind traefik (gitea, homepage, …) while the apps themselves are fine.** Signature (bifrost-dev reported it, gitea-shaped): HTTPS returns `403 content-length 0, no app body` on all routes, but git-over-SSH works (SSH bypasses traefik). Diagnosis: `gitea` direct on `localhost:3000` = 200 (app healthy), through traefik = 403; `crowdsec` container `Exited (255)`; `cscli decisions list` EMPTY (not an IP-ban). **The bouncer plugin does NOT self-recover from a startup-time LAPI-unreachable race** — even after crowdsec is healthy again, traefik keeps 403ing until traefik itself is restarted. **FIX: `docker start crowdsec` (its data/config are LOCAL volumes, comes up clean), wait for `cscli lapi status` = OK, THEN `docker restart traefik`** so the plugin re-inits against the live LAPI. Verified 403→200 on gitea API/web/PyPI-index from an off-box vantage. This unblocked bifrost-dev's 1.2.0 PyPI publish (+ worldtree/wyrd/ratatoskr) and any HTTP gitea access; heid's SSH pushes were never affected. → add to the recovery runbook: **crowdsec+traefik restart is a standard post-power-loss step.**
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UNVERIF
|
||||
|
||||
**Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UNVERIFIED (blocked on GPU space).** Operator got access to `glyphsoftware/sentinel-r3` and asked to compare vs the running M.O.G.-SEC seat + pull if promising, then "quant it with a grafted mtp head". **It is promising and a better FIT**: same base (stock Qwen3.8-27B), same `qwen3_5` hybrid arch, same 262K, vision-intact — but M.O.G.-SEC is ALSO an SFT finetune (refusal-free offense+defense cyber SFT — its card: "not a system-prompt sticker on a stock Qwen"; the "persona on stock" wording here is WRONG, corrected 2026-09-14) while Sentinel-R3 is an SFT finetune on 1,230 authorized-pentest agent trajectories over a 19-tool surface that **matches our own harness** (Bash/Read/Write/Edit/Grep/Glob/Agent/Task*/Monitor/…). Card is unusually honest (flags its own mmlu-cybersec 0.88 as within-noise of base). **HF check: M.O.G.-SEC repo unchanged** (sha still our pinned `deede6779…`). **MTP: Sentinel ships ZERO mtp tensors**; grafted the verbatim base head from `qwen38-27b-uncensored-bf16` (compare_mtp_head → IDENTICAL) — lineage correct since Sentinel's base is stock Qwen3.8-27B and that head is a verbatim base graft. ⚠ **Acceptance is UNVERIFIED and may differ from the 47.7% the head hits on STOCK weights** — it now reads hidden states from an SFT-finetuned body (the exact Stage-1b residual risk). Quant = the standard mixed NVFP4-W4A4(MLP 0-55) + FP8-W8A8(attn/linear_attn/lm_head/MLP 56-63) recipe, ran CUDA_VISIBLE_DEVICES=1 on GPU1 free space, no seat downtime, 51→22 GB. post_quant carried the head forward + re-injected `re:^mtp.*` (llm-compressor prunes it → the 0%-accept bug). Structural verify clean: 1968 tensors, 0 unresolved, 15 mtp, 333 visual, ignore has mtp+visual. **Artifact `/tank/aimodels/sentinel-r3-nvfp4-mixed` (+ `.PROVENANCE.txt`).** ⚠ **License is PROPRIETARY** (Glyph Proprietary v1.0, all-rights-reserved) — operator's fair-use/licensee call, not apache like M.O.G.-SEC. ⚠ **Serving/A-B is BLOCKED on GPU space**: weights are 22 GB, GPU0 has 7.6 free / GPU1 19.9 — a probe serve needs a freed co-tenant slot (~25 GB), which is a material-consequence call. Serve with the PROSE system prompt (trained on prose tools, not structured `tools=`). → `/tank/aimodels/sentinel-r3-nvfp4-mixed.PROVENANCE.txt`
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-11]` SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and an in
|
||||
|
||||
⭐⭐⭐ **SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and an instruct model can. Option C (instruct carrier + corpus rebuilt as instruction→response pairs) is now evidence-backed, not opinion.** Operator's requirement: *"skaldsong will want to write story beats which are a sentence, and have the LLM expound on that sentence to a paragraph and stitch it together."* Booth: `http://10.100.10.50:8090/b/skaldsong-beats/`. **Adapted 4B (checkpoint-75): TEN prompt formats × 3 seeds = 30 samples, ZERO that reliably render the beat** — bare, para-break, labelled, epigraph, fewshot(1), fewshot-bare, fewshot3, elaborate, recount, label-begin. Every one drifts, frames, or truncates. Root cause is structural: *"write a paragraph **about** this sentence"* is an instruction, and a completion model has no mechanism for *about* — it continues the text it is given. ⚠⚠ **Two formats leaked PRETRAINING TASK DATA**: `para-break` emitted an NLI multiple-choice item (*"Does it follow that... OPTIONS: (1). yes (2). it is not possible to tell"*) and `label-begin` a grammar-correction exercise (*"CORRECTION: ... The passage appears to be a sentence fragment"*). A standalone sentence plus a blank line looks exactly like a dataset entry; **style adaptation does not remove base-model task artifacts.** **Instruct arm (`gen` seat + style prompt, no adapter): 10/10 samples inside the requested 90–140 band (124–148w, median 130), every one on-beat, zero drift** — but the voice is generic literary pastiche, abstract-noun-heavy and over-written, not Brontë. **So: voice without direction vs direction without voice; the product needs both.** ⚠ **This applies to Yarros identically** — the carrier question is orthogonal to the author, so the next corpus must NOT re-run this experiment.
|
||||
+6
-67
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-24 ~2245 PT (⭐ esh-ml1 BUILT: RTX 2000E Ada as CT 110 on esh-pve, host driver 580.178.04 live-loaded with NO reboot, vLLM embed+rerank at parity with fv-ml1, LiteLLM order-2 failover. Dead `reranker-a3-bge-v2-m3` alias repaired. ⚠ homepage wedged unkillably on esh-docker-vm (3rd time) — needs a VM reboot, Prime's call. pfi-gx10 AC-restore still UNVALIDATED — AC pull 09-25.)_
|
||||
_Last updated: 2026-09-25 ~1105 PT (⭐⭐ nh3-pve going DOWN for a GPU install, Prime on site; this session ends with nh3-dev — post-boot checklist below. Today: TEI = fleet embed/rerank engine, esh-ml1 sole backend + reward seat moved there, fv-ml1 GPU 1 freed ~15 GB; esh-ml1 monitoring wired; homepage wedge cleared; NIC names pinned on nh3-pve. 34 old entries archived.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -156,7 +156,6 @@ Pre-flight done before shutdown:
|
||||
proxmox-headers installed**, on PVE 8.4.1 (Debian 12 template only). If the LM
|
||||
port was cabled, MEBx provisioning can be done through the NanoKVM.
|
||||
|
||||
|
||||
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
|
||||
|
||||
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE backend
|
||||
@@ -167,24 +166,12 @@ on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
|
||||
Homepage cards, a Dozzle agent. Open follow-ups:
|
||||
- **worldtree-dev**: Domari's Skywork path is broken (dead IP + wrong schema).
|
||||
Told 2026-09-25; waiting on them.
|
||||
- **nh3-dev Beszel agent** dies on every NH3 cold start (it binds an automounted
|
||||
NAS share). Fix = drop its NAS extra-filesystems. Not done.
|
||||
- ✅ **nh3-dev Beszel agent** no longer binds NAS shares (it died on every NH3 cold start); fixed 2026-09-25, `stacks/beszel/hosts/nh3-dev.yaml`.
|
||||
- **Dozzle nh3-docker agent** has been stopped by hand since ~2026-04. Revive or drop.
|
||||
- **Beszel superuser password was echoed into this session's transcript**
|
||||
(local only, not in memory). Rotation offered to Prime.
|
||||
→ Recent decisions.
|
||||
|
||||
### ✅ esh-ml1 built (2026-09-24) — two follow-ups
|
||||
|
||||
The RTX 2000E Ada now serves `qwen3-embedding` + `reranker` from **CT 110
|
||||
`esh-ml1`** (10.0.50.80) as the **order-2 failover** behind fv-ml1. Host driver
|
||||
went on live, no esh-pve reboot. Full record → Recent decisions.
|
||||
- **Parked unless Prime wants it:** a direct ESH→esh-ml1 path for ESH consumers
|
||||
(survives a mesh outage). Spend logs show NO ESH-side caller in 7 days; the
|
||||
real users are worldtree-gateway + nevermore.
|
||||
- **Not wired yet:** esh-ml1 in Homepage `docker.yaml` (needs dockerd tcp/2375)
|
||||
and Beszel. The ~18.7 s host-down failover penalty is untuned.
|
||||
|
||||
### ✅ homepage wedge cleared — esh-docker-vm rebooted 2026-09-25 0613 (Prime)
|
||||
|
||||
Prime ordered the reboot and ruled **do NOT move Homepage** (stays on
|
||||
@@ -214,6 +201,9 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-25]` **MS-01 foot-gun: a GPU in the PCIe slot renames every NIC** (the slot's root port takes bus 01, so the X710 goes `enp2s0f0np0`→`enp3s0f0np0`). On nh3-pve that means booting with no uplink. Pinned by MAC with `.link` files before the install, and the AMT port was pulled from vmbr0. → `servers/nh3-pve/README.md`, `playbooks/nh3-pve-pin-nic-names.yaml` (`2118449`)
|
||||
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
|
||||
- `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
|
||||
- `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
|
||||
- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
|
||||
- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
|
||||
@@ -399,56 +389,6 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
- `[2026-09-11]` **Worldtree memory-split (U6) — PROTOCOL AGREED with worldtree-dev: nobody flips memory.reader.enabled or m** → `persistent-memory.d/2026-09-11-worldtree-memory-split-u6-protocol-agreed.md`
|
||||
|
||||
- `[2026-09-11]` ⭐ **Plex hardware transcoding on the Arc A580 FIXED (esh-pve-nas LXC 105) — every setting was already correct and the fault was one layer below them.** `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) predates Arc/DG2 support and exports only `__vaDriverInit_1_14`, against the libva **2.22 Plex BUNDLES** and loads via RPATH. Passthrough, cgroups, `plex` in video+render, HuC authenticated, Plex Pass, `HardwareAcceleratedCodecs=1` and the Arc already selected as `HardwareDevicePath` — all good the whole time. Fixed with Intel's client-GPU repo (rolling `jammy client`) → iHD **24.3.4** (`__vaDriverInit_1_22`) + a consistent libva **2.22.0.2-87** set, now **pinned + `apt-mark hold`** (verified: a simulated upgrade moves 152 packages, touches none of the six). Also repaired a **half-finished prior attempt** — libva/libva-drm hand-installed at 2.22 with `libva-x11` left at 2.14, killing every X11 VA-API app on `va_fool_postp`. ⚠⚠ **`pct snapshot` REFUSES on a bind-mounted guest AND STILL EXITS 0** (LXC 105 has `mp0: /tank/media`) — use `zfs snapshot nvme/subvol-105-disk-0@<tag>` and read it back. ⚠⚠ **A synthetic `Plex Transcoder` run is NOT a valid test** (Plex bundles its own libc among 61 libs; my harness failed identically before and after a fix that worked — no positive control, so its negatives were worthless). Only a **forced transcode** settles it: PASS *names the device* (`testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])`). ⚠ The original empty `final decoder: , final encoder:` was an **absence of evidence**, not failure — `TranscodeSession` was 0. Jellyfin LXC 107 left alone (operator: not actively used). → `persistent-memory.d/2026-09-11-plex-arc-vaapi.md`, runbook `docs/runbooks/plex-arc-vaapi-jammy.md`
|
||||
|
||||
- `[2026-09-11]` **Beszel priority 2 complete:** both DB hosts and both PBS hosts verified, 16 new alerts; fleet 17/18 up (ana-ml2 down). → `persistent-memory.d/2026-09-11-beszel-priority2.md`
|
||||
|
||||
- `[2026-09-11]` **Beszel priority 1 complete: all six installed and verified.** After Anaheim recovery, live Synology samples and alerts verified; fleet 13/14 up, only known ana-ml2 outage remains. Configs not committed. → `persistent-memory.d/2026-09-11-beszel-priority1.md`
|
||||
|
||||
- `[2026-09-11]` **Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UN** → `persistent-memory.d/2026-09-11-sentinel-r3-pulled-mtp-grafted-and.md`
|
||||
|
||||
- `[2026-09-11]` **MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VR** → `persistent-memory.d/2026-09-11-measured-two-concurrent-training-jobs-on.md`
|
||||
|
||||
- `[2026-09-11]` **ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (** → `persistent-memory.d/2026-09-11-ana-ml2-fv-ml1-relocating-to.md`
|
||||
|
||||
- `[2026-09-11]` **Anaheim rack LEFT DARK until the move (operator decision).** ana-ml2 is the ONLY host still down post-recovery (BMC dark = no power); rather than power it on tonight just to shut it down for the truck tomorrow, it stays off. I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) but there is nothing to bring up — the box relocates as fv-ml1.
|
||||
- `[2026-09-11]` **RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik'** → `persistent-memory.d/2026-09-11-recovery-foot-gun-will-recur-every.md`
|
||||
|
||||
- `[2026-09-11]` **Anaheim colo recovered ~16:39 PT EXCEPT ana-ml2 (bare metal, NO power — its BMC 10.250.250.50 is dark on standby, unlike same-subnet pfi-pve which is up → needs a physical PDU/PSU/breaker fix, not a boot).** pfi-pve + all its VMs (ana-docker/ana-nas/ana-wg/corviduo-dev/pbs-ana) auto-started clean (on-boot gap held this time). LiteLLM came back up on its own (transient `unhealthy` during startup → serving). ⚠ **Public WAN (38.120.12.44) ICMP still blocked from outside but HTTPS works fleet-internally** (mesh-routed). ana-ml2 down blocks the gen/summarizer/mog-sec seats AND the cyber-preview quant re-run. **I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) to power-on + boot-watch the instant its BMC returns.**
|
||||
- `[2026-09-11]` **BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured n** → `persistent-memory.d/2026-09-11-babyyarros-complete-both-arms-trained-and.md`
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **BabyYarros UNBLOCKED and TRAINING: the leak gate passes at 0 of 325 entities and 0 of 91 phrases, and closing it turned up three defects nobody was looking for.** The gate itself is the first artifact — there was no committed instrument for "does any of the author's proper nouns survive", so Brontë's 0-of-203 was a hand count. `scripts/r49-corpus/leak_gate.py` now runs the same scan over the UNRENAMED source as a **positive control** plus a nonce **negative control** every time, because a detector that only ever sees renamed text cannot tell *absent* from *blind*. Its first reading was **212 surviving, not 86** — it scans the whole corpus rather than per work, and counts the sub-threshold entities rename never looked at. Training launched 10:06 PT on pfi-gx10: Qwen3-4B-**Instruct**, 1 epoch, seed 4919, 178 steps / 5,824,512 tokens, corpus sha `e85f69f1e49d57c9`. → `persistent-memory.d/2026-09-11-babyyarros-leak-gate-passes.md`
|
||||
- `[2026-09-11]` **A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong ** → `persistent-memory.d/2026-09-11-a-second-corpus-typography-defect-and.md`
|
||||
|
||||
- `[2026-09-11]` ⚠⚠ **Back matter was inside the prose of all five works — 4,555 words naming the author's agent, editors and children.** The builder splits on chapter headings and nothing follows the last one, so acknowledgments, newsletter pitches and cover-artist credits rode inside the final chapter. Found by the gate's phrase audit surfacing `Louise Fury` (Yarros's literary agent), not by reading. ⚠ **iron-flame's marker is `ACKNOWLEDGMENTS` in all caps** and a case-sensitive scan missed it — the strip is case-insensitive and last-chapter-only, with an acceptance check that refuses if it would remove more than 2% of the corpus.
|
||||
- `[2026-09-11]` ⭐⭐⭐ **The gate read 0 of 314 while `Afendra` was still in every copy — the worst failure shape available.** The name never appears unpossessed, so it keyed as `Afendra's`, and **rename.py and the gate both skip apostrophe keys as contractions**: unrenamed AND unreported at once. Fixed by folding clitics (`--fold-clitics`) so `Afendra's` counts as `Afendra`. `Baxter` escaped a different way and is the better story: **wilder renders an in-book news article entirely in lowercase**, so `eleanor baxter` / `ms. baxter` appear uncapitalised 3 times against 23 capitalised — ratio **0.13 against a 0.05 bar**, and a real character is silently never renamed. Fixed by readmitting ratio-rejects that a title precedes (`--rescue-honorific 2`). ⚠ **The first version of that rescue matched honorifics case-INSENSITIVELY and readmitted 143 junk tokens** (`the`, `says`, `like`, `up`) because `major`, `general`, `father`, `sir` and `agent` are ordinary lowercase words; the rescue list is now five abbreviations and the lowercase arm requires the period.
|
||||
- `[2026-09-11]` ⭐⭐ **A whole leak class the unigram scan structurally CANNOT see: `Riders Quadrant`, `Flame Section`, `War Games` — and `Fourth Wing`, the book's own title.** Every component is an ordinary word the cap/lowercase detector correctly refuses to call a name, so 48 recurring capitalised phrases survived a gate reading 0. This is `Thornfield × 100` one level up, and it needs a **map**, not a detector — substituting a head noun is a choice about register, not a measurement. `scripts/yarros-corpus/phrase_map_yarros.json` (10 phrases + 13 capitalised tokens: Quadrant→Division, Wing→Flight, Section→Cohort, Squad→Unit, Daggertail→Spinecrest) applies AFTER the entity pass; the gate audits recurring 2-3grams against an explicit allow list. Result: 48 → 0.
|
||||
- `[2026-09-11]` **Per-work rename maps leak across works, and for a SERIES they are also wrong.** `Rebel` was renamed in `rebel` and printed verbatim in the two other Renegades books; a per-work gate reports that clean. `--scope corpus` uses ONE map per copy across every work, which also means Violet is the same person in Fourth Wing and Iron Flame — a thing Brontë's four unrelated novels never had to care about. 8 cross-work gender conflicts held to neutral rather than guessed.
|
||||
- `[2026-09-11]` ⭐ **The mid-sentence test: position as a SECOND filter, which is not the v1 mistake.** entities.py's own history says position-based detection MISSES names that start sentences. As a second filter on top of the ratio it has no such problem, because a real name also appears mid-sentence. Measured: **33 verified names at 0.567–0.985 mid-sentence, 19 verified interjections at 0.000–0.222** — a 2.5x gap, so 0.35 is not a tuned parameter. It fixes `Hey`/`Holy`/`Hopefully`/`Yep`/`Whoa`/`Nope`/`Ugh` being entities. ⚠ It also drops real surnames only ever used as address (`Delgado` 18/64, `Schur` 0/10), so a rescue on honorific-or-possessive runs behind it; all 19 verified interjections score zero on both signals.
|
||||
- `[2026-09-11]` ⚠ **The stoplist is short because every surface was read IN CONTEXT first, and a plausible guess would have been wrong most of the time.** `Violence` is Xaden's nickname for Violet. `Continent`, `Presentation`, `Battle Brief`, `Curator`, `Sage`, `Barrens`, `Originals`, `Montserrat`, `Athena` and `Aura` are all in-world. Only real-world geography, brands, three nationality adjectives and four generic title words are excluded — **ambiguous cases are deliberately renamed, because renaming is the safe direction and leaving is the leaking one.** `scripts/yarros-corpus/stoplist_yarros.json`.
|
||||
- `[2026-09-11]` **BabyYarros D1 BUILT, D2 gender FIXED, D3 rename BLOCKED on the leak gate.** Operator: *"train the instruct on the yarros corpus -- babyyarros."* Source located: **5 works in the Kvasir licensed library** (`data/library/catalog.sqlite`, `rights=gated`) — Fourth Wing, Iron Flame, Wilder, Nova, Rebel. **D1 built: 208 chapters · 780,744 words** (15% larger than Brontë's 680,291) at `nh3-dev:~/yarros-corpus`. ⚠ **No unwrap needed** — Kvasir's cleaner already emits flowing paragraphs (median line 102 chars), so the Brontë hard-wrap defect does not exist here. **Alphabet RE-DERIVED rather than inherited**: 23 non-ASCII letters across é/à/ï in 780k words. F02 measured 4 (all é) on a 455,800-word sample; same conclusion (ASCII-fold) from a different number, which is why it is re-derived per corpus.
|
||||
- `[2026-09-11]` ⭐⭐ **NEW PATHOLOGY, worse than Brontë's: in a ROTATING first-person POV corpus, every book's narrator gets the WRONG gender.** Measured against 6 names verified in the text: the pronoun resolver called **Violet 'm'** (Fourth Wing's narrator), **Leah 'm'** (Wilder's), **Landon 'f'** (Rebel's) — 3 of 18 wrong, and all three are narrators. Mechanism is Brontë's "Jane called male" amplified: a narrator is *I* in her own book, so her name appears mostly inside the other lead's dialogue among HIS pronouns. ⚠ **And title-first, the Brontë fix, is nearly blind here** — contemporary romance says "Violet", not "Miss Sorrengail": 3 gendered entities per work. **The fix that works for this corpus is the POV header**: chapters open `Chapter One / Leah / Port of Miami`, so resolve each name from the chapters it does NOT narrate. Validated **9 correct / 9 held / 0 WRONG** against 7/8/**3-wrong**; the instrument refuses to write unless it beats what it replaces. `scripts/yarros-corpus/pov_gender.py`. ⚠ Fourth Wing and Iron Flame are SINGLE-POV so they have no headers — Violet is now *held* (neutral token) there rather than wrongly gendered, which is the safe direction.
|
||||
- `[2026-09-11]` ⚠ **Three real bugs found in `rename.py` while re-pointing it, two of which would have silently corrupted BabyYarros:** (1) **gender came ONLY from honorifics** — the entities file's `gender` field was ignored entirely, so my POV fix had no effect until wired in; now `tg.get(key) or e.get("gender")`, titles first so Brontë is unchanged. Effect: 1 → 13 gendered on `wilder`. (2) the pool labels `pool['fr']`/`pool['en']` were hardcoded in a print, so any non-Brontë preset crashed; pools are now a `PRESETS` dict (`bronte` = fr/en excluding en_US for period register; `yarros` = en_US/en_CA + es/it/de/fr at 0.62 US). (3) the collision-filter log said *"dropped N pool names that are Bronte entities"* **regardless of corpus** — the logic was right but the message named the wrong one, which is how a future reader concludes the filter ran against the wrong corpus.
|
||||
- `[2026-09-11]` ⛔ **D3 BLOCKED: leak gate at 86 of 232 renameable source entities surviving; Brontë's run reached 0 of 203.** Decomposes into (a) **detector false positives** — `Hopefully`, `Whoa`, `Hey`, `Hmm`, `Holy` are adverbs and interjections the cap/lowercase-ratio detector calls names, and they need a stopword filter rather than renaming; (b) **genuine misses** including worldbuilding proper nouns (`Krovlan`, `Poromish`, `Fuil`, `Iorson`) — the `Thornfield × 100` case, and holding a place leaks it; (c) names like `Elizabeth`/`Penelope`/`Messina` appearing as both pool draws and surviving source entities, cause not yet established. **Nothing has been trained.** ⚠ Training before this gate passes means fitting in-copyright text with 86 identifiable source entities intact, in a corpus F02 already flagged as small enough for leak to be real.
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **THE INSTRUCT PROBE ANSWERS ITS QUESTION: voice and instruction-following DO coexist. Option C is de-risked.** `Qwen3-4B` **instruct** (not `-Base`), same corpus/seed/steps so the carrier is the only variable; best checkpoint `checkpoint-150` picked by loss (applying the 4B-Base lesson automatically this time). **Voice installed at full strength — curly quotes 16/18, IDENTICAL to the 4B-Base tuned arm's 16/18**, against the unadapted control's 1/18, and **task-leak 0/18 vs the base carrier's 4/18**. So the assistant prior did NOT block Brontë, which was the central risk. **Instruction-following SURVIVED: 10/10 on-beat through the chat template**, same as the untuned control. ⚠ **The cost is length discipline, not comprehension** — in-band 10/10 → **6/10**, median 124w → 140w. Training on Victorian prose made it wordier, a soft degradation rather than a break. ⚠ **Held-out 2.908 vs 4B-Base's 2.814** — the instruct carrier fits the corpus **0.094 nats worse** and **plateaus without turning** where base overfit at step 75: the assistant prior competes for capacity, so it absorbs less rather than overfitting more.
|
||||
- `[2026-09-11]` ⚠ **What raw-continuation training on an instruct carrier does NOT fix: the plot furniture.** Reading the product artifact, the tuned-instruct arm renders the beat and then drags the referent — *"He licked her clean… my master thus—my husband thus"*, turning the dog into a man, because Brontë's corpus is about masters and husbands. Another beat ran 247w and gave the narrator a list of duties. **This is exactly what instruction-PAIR training is for** — pairs teach "render this and stop", continuation teaches "keep writing Victorian prose". So the probe de-risks option C without substituting for it. ⚠ Also: my `ran_on` metric is uninformative on this job (10/10 on BOTH arms) because a single paragraph contains no blank line — it measures "no paragraph break found", which is correct and useless here. Do not read it as a finding.
|
||||
|
||||
- `[2026-09-11]` **SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and ** → `persistent-memory.d/2026-09-11-skaldsong-s-shape-settles-the-architecture.md`
|
||||
|
||||
- `[2026-09-11]` ⚠ **Stitching has its own failure mode, visible in the booth's Panel C: independently-generated paragraphs drift in POINT OF VIEW.** By beat 4 of 5 the narrator is simultaneously watching the girl carry the animals and carrying them herself ("their weight a strange, heavy secret carried between my ribs"). Each paragraph was generated with no knowledge of the others. **A real stitcher must feed prior paragraphs back as context**, which also means the instruction-pair corpus should include multi-paragraph continuity examples, not just isolated beat→paragraph pairs.
|
||||
|
||||
- `[2026-09-11]` ⭐⭐ **THE RECIPE THAT WORKS ON A COMPLETION CARRIER: label the artifact AND begin it.** Operator's prompt: *"This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw-- "*. **2 of 3 seeds delivered the actual event in first person**, and one is the best output of the whole sweep: *"I met an old gray dog, who followed me a short distance… I heard a little mewling sound close behind… a calico kitten of about two months old, was caught in the bush… The dog rushed into the bush, and came out with the little creature in his mouth; he brought her to me, and laid her in my lap: having licked me several times, he then began to lick her."* Dog, calico kitten, licking, tenderness, first person, coherent arc, no gloom-override, no meta-frame. **Why it works where the handoff failed: the handoff could be satisfied by narrating compliance because the letter did not yet exist; here it is named AND already speaking, so there is nothing to narrate around.** Also learned the Gutenberg `_underscore italics_` convention. 1 of 3 drifts.
|
||||
- `[2026-09-11]` ⚠ **My typography hypothesis was WRONG, and the chapter-heading result is the evidence.** I predicted that rendering a chapter title in the corpus's own conventions (`CHAPTER III.` / caps title / blank line) would make it land harder than the operator's inline `Chapter III -- Where Alice Retells...`. **It did the opposite**: both corpus-form seeds ignored the title entirely and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter *about* the story (a gentleman disputing the premise). Likely reason: corpus chapter titles are short and decorative (`THE CHILD'S CLOSET`), so a long descriptive one in that slot reads as decoration to skip, whereas inline it reads as text to continue. **A label only instructs if the model treats that slot as load-bearing.**
|
||||
- `[2026-09-11]` ⚠ **Unnoticed consequence of the D2/D3 rename pipeline: the adapter SUBSTITUTES proper nouns it was never trained on.** Given "Alice" in a chapter title it produced *"ALEXANDER THE ALEXANDER, AS HE WAS KNOWN IN LITTLE LONDON"*. The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Consequence for use: **you cannot reliably name your own characters at prompt time** — they may be renamed mid-passage. Not a defect of the rename (which exists to prevent memorisation of Brontë's cast) but a real usability constraint that needs stating.
|
||||
|
||||
- `[2026-09-11]` **4B arms RE-CUT from `checkpoint-75`, the true loss minimum (2.813826, confirmed from `loss-series.json` rather than my reading of the log); booth rebuilt.** Only the tuned arms needed it — the base arm never touches the adapter. ⚠ **A small surprise: step-75 and end-of-run differ on typography, not voice.** Curly quotes 16/18 vs 17/18 and collapse 0/18 either way, but the **hard-wrap ratio is 0.33 at step-75 against 0.12 at end-of-run** — further training washes the residual line-break habit out *while held-out loss gets worse*. So **"best loss" and "best typography" are different checkpoints**; neither is near the original 0.85 defect, and the corpus's own residual (preserved verse) is 0.25.
|
||||
- `[2026-09-11]` **EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about so** → `persistent-memory.d/2026-09-11-embedding-an-instruction-inside-the-fiction.md`
|
||||
|
||||
- `[2026-09-11]` **R49 SWEEP COMPLETE — 4B closes the continuity gap, and the carrier ladder is clean: 3.329 → 3.018 → 2.814 held-out** (0.6B / 1.7B / 4B, all on the same unwrapped corpus sha `77f37057b2782e49`, seed 4919, 159 steps, 5,210,112 tokens — carrier size the only variable). Deltas **0.311 then 0.204**: diminishing but still real. Booth: `http://10.100.10.50:8090/b/babybronte-4b/`. **4B tuned has the best voice saturation of any rung — curly quotes 17/18 against its own base arm's 1/18, collapse 0/18 against 4/18** — and, the thing the rung existed to test, **scene-level continuity HOLDS**: it produces a named character with motivated dialogue, a navigable spatial layout and a physical description in one passage, where 1.7B wrote pretty but eventless prose (opening doors, looking at stars). On the letter prompt it opens the letter, promises to quote it, and then actually quotes it across a paragraph break.
|
||||
- `[2026-09-11]` ⚠⚠ **4B is the FIRST rung to OVERFIT inside one epoch, which inverts my earlier "one epoch is right for this corpus" call.** Series 2.832 · 2.816 · **2.814** · 2.820 · 2.824 · 2.825 · 2.825 — minimum at ~step 75, then it TURNS and settles worse. 0.6B and 1.7B both plateaued with no turn, so **the optimal epoch count shrinks as the carrier grows** — 4B wants roughly half an epoch. ⚠ **Consequence: the shipped `adapter/` at `h02-4b-1ep/` is NOT the best checkpoint** (it is the end-of-run 2.825); the step-75 checkpoint at 2.814 is, and it exists only because `save_steps=25` was set. The voice test used the end-of-run adapter, so the booth understates 4B by ~0.011 nats. Re-cut the arms off the step-75 checkpoint before any adjudication.
|
||||
- `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established.
|
||||
|
||||
- `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md`
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
@@ -457,7 +397,7 @@ a shutdown (stays off by design). Outcomes and revert in
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
_70 older entries archived to archival-memory.md._
|
||||
_104 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -475,5 +415,4 @@ _70 older entries archived to archival-memory.md._
|
||||
|
||||
- `[2026-09-15]` **Remote-site MASQUERADE rules on nh3-scale** for the asymmetric-return theory — they fired (counters incremented) but were not the fix. Reverted rather than left to accumulate as NAT achieving nothing.
|
||||
|
||||
|
||||
_115 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user