memory: snapshot — run-5 RESCUED + seat live, base settled STOCK, cc-channel route, Miranda relay authority; session fleet-fixes + open items
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
# `[2026-09-08]` R47 base provenance SETTLED from bytes — it is STOCK, not heretic
|
||||
|
||||
Operator-directed question (relayed via brokkr): which weights are at
|
||||
`/home/infra-ops/models/gemma4-26b-a4b-it-bf16` — stock instruct or the heretic
|
||||
abliteration? Two labels disagreed (recipes say `-heretic-bf16`; run-04
|
||||
provenance says stock) on ONE path with no model card. Settle from bytes.
|
||||
|
||||
## Answer: STOCK `google/gemma-4-26B-A4B-it`, byte-for-byte.
|
||||
|
||||
Three-way match, zero ambiguity:
|
||||
|
||||
1. **Local shards** (sha256 on gx10): `model-00001` =
|
||||
`1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3`,
|
||||
`model-00002` = `aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7`.
|
||||
Both **equal the HF download etags** in `.cache/huggingface/download/*.metadata`
|
||||
→ the local copy is uncorrupted, exactly what was pulled.
|
||||
2. **Download commit** in that same metadata: `4d7ae4984b7db7de8f8457170b3f1a419ee76d52`.
|
||||
3. **HF `google/gemma-4-26B-A4B-it`** (gated:false, queried with the local
|
||||
`hf_HBlJ…` token in `~/.config/secrets/env.sh`): **HEAD sha ==
|
||||
`4d7ae498…`**, and its two safetensors LFS oids are `1127684971…` / `aab47033…`
|
||||
— identical to (1).
|
||||
|
||||
Commit + both shard hashes match at the stock repo's current HEAD. Behavioral
|
||||
corroboration: the base hard-refuses ~77.7% (brokkr's k=25) = stock, not a
|
||||
3/100-refusal abliteration.
|
||||
|
||||
## What it means
|
||||
|
||||
- The recipes' `-heretic-bf16` label is a **naming error**; run-04's "stock"
|
||||
provenance was right. Every run 3/3c/4/5 trained from a refusing STOCK base.
|
||||
- Why plausible: the 2026-08-24 trainee-selection SELECTED llmfan46's Gemma-4-26B-A4B
|
||||
Heretic v1.2.0 ARA (3/100 refusals, bf16 51.6 GB) — but llmfan46 ships that
|
||||
26B-A4B abliteration **GGUF-only**, no bf16 safetensors, so the bf16 that got
|
||||
pulled was stock google and the `-heretic` name rode along from intent.
|
||||
- Brokkr took the corrections his side (drop `-heretic` from the run-5 emitter +
|
||||
docs, annotate run-4's recipe, fix the 2026-08-24 note). The accept-vs-swap
|
||||
choice is the operator's, now evidenced — see [[2026-09-08-run5-rescued]].
|
||||
|
||||
Commit: 1282698. Thread: althing `01M207X7SKY2ZDKYREBAQ17CE1`.
|
||||
@@ -0,0 +1,49 @@
|
||||
# `[2026-09-08]` ERP-seat SFT run 5 — gate = RESCUED (landmark R49.5)
|
||||
|
||||
The FIRST arm of the R47 ERP-seat line where the capability gate did NOT fail.
|
||||
Operator launched it directly to infra-ops ("yes go, launch run 5", the relayed
|
||||
grant `operator-2026-09-07-rnd-run5` closed in-session per the no-relayed-auth
|
||||
rule). Brokkr built the corpus; infra-ops staged + launched + served the gate.
|
||||
|
||||
## The result
|
||||
|
||||
- **RESCUED.** Structure of the loss was the lever, not its mass: the
|
||||
dependency-forcing slot (GovReport 496 + QMSum 97, only **3.46% of loss** at
|
||||
20% ctx) broke the diversity↔coherence coupling that run 4 (STILL-COUPLED,
|
||||
20.6% instruct slot) and 3c (20× LR cut) could not. **INERT did not fire.**
|
||||
- Mechanism: **T4 long-context 8/8** (run 4: 5/8; base 8/8); t4_dissect noise@31
|
||||
tuned **0.9062** vs run-3 tuned 0.5625; diversity held (rp density 3.37→0.00,
|
||||
story 2.86→1.58).
|
||||
- Reported-beside (not in the cell, de-gated + stated): T3 constraint 8/8→**6/8**
|
||||
(a NEW loss, ship-path list); RP length 68w vs the 250-floor = **PARTIAL fail**
|
||||
(short-QA slot + style shift); refusal erosion rides with the style shift (k=25
|
||||
both arms, CSAM clean, benign 99/100); free-check base LEVELS 5–6pp below run 4
|
||||
on identical bytes/stack (a generations shift, vLLM 0.28.0 unchanged — infra
|
||||
confirmed; taxes every cross-run number).
|
||||
- Write-up: brokkr-smithy `research/R47-premium-corpus-gate/run05-gate/RESULT-run05-gate.md`;
|
||||
`FLOOR-LOCKED 0f3e4e2` cites infra-ops' base index-sha 907826a6.
|
||||
|
||||
## The run (infra side)
|
||||
|
||||
- pfi-gx10 (GB10, sm_121, aarch64), 524 steps / 1 epoch / ~11.8 h / 81 s/it,
|
||||
train_loss 3.235; adapter verified (410 tensors). Launch gates all passed:
|
||||
7/7 root shas + shard hashes, survivor join 8,212 = recipe, holdout disjoint,
|
||||
**window_count belt-and-suspenders** (govreport 496/496 + qmsum 97/97 fit_whole),
|
||||
realized `[mix]` slot loss 3.46% (pred 3.4%). `provenance.json` carried the
|
||||
override id + both blockers + all substitute controls.
|
||||
- Gate served on gx10:8098, base→tuned, name-keyed swap, hands-off honoured; base
|
||||
sha + vLLM-stack answers on the record before Brokkr needed them.
|
||||
- Canonical: eshpfi `scripts/erp-tune-gx10/` (run-05-gx10.json, launch-run-05.sh,
|
||||
build_r5_survivors.py) + `docs/runbooks/gx10-run-05.md`. survivors-r5 =
|
||||
survivors-r4 − airoboros + govreport+qmsum whole (kvasir held byte-identical),
|
||||
sha a25169a6.
|
||||
|
||||
## Seat + open decision
|
||||
|
||||
`erp-tune-v5` served on gx10:8098 (merged-run05); LiteLLM `trial` alias repointed
|
||||
3c→v5 (operator 2026-09-08) — verified end-to-end. Seat is hand-launched
|
||||
(`vllm-run05.pid`, no systemd) — dies on a gx10 reboot, yields to next training
|
||||
(~6 min re-serve). **OPEN (operator + brokkr): accept RESCUED-on-stock, or source
|
||||
a bf16 abliteration + re-run** — now evidenced, see [[2026-09-08-base-provenance-stock]].
|
||||
|
||||
Commits: 265357e (launch) · 37f2caa (RESCUED memory) · f8a7985 (trial→v5).
|
||||
@@ -144,6 +144,31 @@ COMPLETE and gated RESCUED (02:13 PDT). Live open items:_
|
||||
pulled was stock google, and the `-heretic` name rode along from intent. **Operator/Brokkr decision
|
||||
now evidenced (not a label guess): accept RESCUED-on-stock, or swap to a real abliteration (needs a
|
||||
bf16 source, not the GGUF) + re-run. Recipes should drop `-heretic` from the base name.**
|
||||
- **⏳ OPEN (operator + brokkr, not urgent): accept RESCUED-on-stock as the ERP seat, or source a bf16
|
||||
abliteration + re-run the line.** Now an evidenced choice (base is stock). No deadline.
|
||||
- **📮 althing reachability on THIS bg seat = the cc-channel route, NOT the waiter.** `althing-listen`
|
||||
reaps within ~seconds-to-minutes on an idle background seat (harness kills detached bg tasks; hit it
|
||||
repeatedly this session). Fix (operator-directed 2026-09-08): `althing-route declare --handle infra-ops
|
||||
--pid <PID>` where `<PID>` is the number in `$CLAUDE_CODE_MESSAGING_SOCKET` filename
|
||||
(`/run/user/1000/cc-socks/<PID>.sock`) — `--discover-pid` REFUSES on a forked child session (ancestry
|
||||
≠ zellij launcher). Flips `postbox status` to `mode: push, reachable: True`, no process to reap. A fresh
|
||||
bg session should re-declare it (the pid changes per launch). See the `[2026-09-07]` cc-channel entry in
|
||||
Recent decisions for the durable why.
|
||||
- **✅ Fleet fixes shipped this session (2026-09-08), all committed:** WhereTF Homepage card + DNS alias
|
||||
(`wherethef.nh3.internal`, e0d1c44/4506ef6); ext-tts LiteLLM alias repointed to `irv-ml1.nh3.internal`
|
||||
via DB `/model/update` + `extra_hosts` (957c8f1); the 09-06 irv-ml1 move's stale-IP trail repointed
|
||||
across 25 stack composes + services.yaml + ssh-target → DNS name (e0d1c44); Homepage `/api/services`
|
||||
outage fixed — irv-ml1 docker.yaml repointed + **ana-ml2 discovery via a socat proxy on ana-docker**
|
||||
(`stacks/ana-ml2-proxy`, 913d2d2, reversible when ana-ml2 gets an ESH return route).
|
||||
- **⏳ Minor cleanup leftovers (offered, operator hasn't taken — not urgent):** (#2) the ~10 RUNNING
|
||||
irv-ml1 service cards still show dead `10.100.79.3` hrefs — canonical fixed, but each running container
|
||||
needs a recreate to apply the label (bounces the service); (#3) deployed `.env` for asset-engine /
|
||||
open-webui / skaldsong may still hold the dead default (canonical defaults fixed; a read-only check would
|
||||
confirm which deployments are broken).
|
||||
- **⚠ persistent-memory.md is 2.4× the soft cap (~710 lines).** Bulk is the non-archivable structural
|
||||
sections (sister-repos table + backup architecture under Tools and conventions) — archival only touches
|
||||
dated logs, so it can't reach 250. The real fix is a redundancy-trim of Tools-and-conventions rows that
|
||||
CLAUDE.md already covers (deliberate pass, not archival). Flagged, not done.
|
||||
- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116
|
||||
backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed:
|
||||
`zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead
|
||||
@@ -548,6 +573,10 @@ below is a live commitment or a known-open risk._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-08]` **ERP run 5 = RESCUED (landmark R49.5)** — first capability-gate pass in the ERP-seat line; the 3.46%-loss dependency-forcing slot (GovReport+QMSum) broke the coupling runs 3c/4 couldn't. Seat `erp-tune-v5` served on gx10:8098, `trial` alias repointed 3c→v5. → `persistent-memory.d/2026-09-08-run5-rescued.md`
|
||||
- `[2026-09-08]` **R47 base settled from bytes = STOCK `google/gemma-4-26B-A4B-it`** — three-way sha match (local == HF etag == stock LFS oid; commit `4d7ae498` == stock HEAD); the `-heretic` label is a naming error, all runs trained from stock. Accept-vs-swap now evidenced. → `persistent-memory.d/2026-09-08-base-provenance-stock.md`
|
||||
- `[2026-09-08]` **Miranda = operator's chief of staff, may relay his directives** — added to user-level `~/.claude/CLAUDE.md` (dotfiles `7134a22`) as the named exception to the no-relayed-auth rule (unidentified peer relays still excluded); material-consequence calls she relays stay the operator's own.
|
||||
- `[2026-09-08]` **Fleet fixes shipped** — WhereTF Homepage card + DNS (`4506ef6`); ext-tts LiteLLM alias → `irv-ml1.nh3.internal` (DB `/model/update` + `extra_hosts`, `957c8f1`); the 09-06 irv-ml1 stale-IP trail repointed across 25 composes + services.yaml + ssh-target → DNS name (`e0d1c44`); Homepage `/api/services` outage fixed — ana-ml2 discovery via a socat proxy on ana-docker (`stacks/ana-ml2-proxy`, `913d2d2`, reversible).
|
||||
- `[2026-09-07]` **Fleet internal TLS pattern shipped** — caddy (cloudflare-plugin build, `~/.local/bin/caddy-cf`, `fleet-tls-caddy.service`) on nh3-dev is the wildcard cert authority: publicly-trusted LE `*.nh3.phasefinal.com` via Cloudflare DNS-01, resolved internally by an AdGuard split-horizon rewrite (all 3 resolvers). `talk` self-terminates on :8092 with the trusted cert (operator's in-container-TLS ruling), renewal auto-synced by `fleet-tls-cert-check.timer`. Interstitial gone; secure-context+AudioWorklet verified via headless Chromium. Pattern + foot-guns (restart-disrupts-inflight → clients need retry; wildcard = name-only, never IP) → auto-memory `reference_fleet_internal_tls_pattern`.
|
||||
- `[2026-09-07]` **cc-channel registered for this infra-ops session's wake** — `althing-route` cc route → the CC session's `$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`; herald pokes the socket directly at a turn boundary. Replaces the FIFO/poll waiter that Claude Code 2.1.257 kept killing while the seat sat in `shell`. Session-local — re-declare per session.
|
||||
- `[2026-09-07]` **irv-ml1 /mnt/smithy remount fixed post-cutover** — export allowed `10.0.0.0/8` (old wg0) but not the mesh `100.64.0.0/10` irv-ml1 now sources from → all-uid "permission denied"; added the mesh range to the nh3-nas smithy export + remounted (clientaddr now 100.64.0.6). nh3-nas admin = `infra-ops` PASSWORD auth (vault `nh3-nas/infra-ops-password`), sudo ALL, SFTP subsystem OFF. → auto-memory `reference_irv_ml1_gpu_r14` (corrected).
|
||||
|
||||
Reference in New Issue
Block a user