diff --git a/docs/runbooks/gx10-run-07.md b/docs/runbooks/gx10-run-07.md index bca422e..b750117 100644 --- a/docs/runbooks/gx10-run-07.md +++ b/docs/runbooks/gx10-run-07.md @@ -55,3 +55,72 @@ tokenizer + template all unchanged from run 6. Gate choreography after the adapter lands: brokkr-smithy `SFT-RECIPE-run7-SCOPE.md §5` (cells LENGTH-FIXED / LENGTH-MOVED / BOUGHT-T4 / FLAT); serve names `erp-seat-base-ara` then `erp-tune-v7`. + +## Outcome — COMPLETE 2026-09-09 13:23 PT + +Ran to the end with no interventions, no resumes, no errors: **542/542 optimizer +steps, one epoch, 14h17m wall clock** (23:06 PT 09-08 → 13:23 PT 09-09). + +| | | +|---|---| +| `train_loss` (run mean) | **3.205** | +| last logged points | 2.9366 @ step 530 (lr 3.5e-07), 2.9389 @ step 540 (lr 1.86e-08) | +| lowest logged | **2.7992 @ step 420** (from 7.1692 @ step 10) | +| step seconds | n=541, p50 **82.1**, mean 94.1, min 17.4, max 334.3, p99 268.3 | +| adapter | 410 tensors, 297 MB, `adapter_artifact` verified | + +Provenance (`run-07/adapter/provenance.json`), the fields worth re-reading before +trusting any number: recipe `recipe-erp-seat-sft-r7.json` sha `2972559e4c2940dc`; +harness `0a6bd2e0`, `harness_dirty_at_launch False`; **`flex_attention` requested +AND resolved** (§4 of the training-throughput playbook — the backend delta that +voided two earlier conclusions); `quantized_base False`; torch 2.14.0+cu130, +transformers 5.16.1; chat template `ae53464b`; 8,661 encoded records from 8,505 +source samples; 6,106 masked turns; `unfittable_disposition drop` (cwm-2602, +cwm-1826 — their loss moved into their opening-split rows, as designed). + +Effective mix as the optimiser saw it, identical to the free check, so the single +variable landed: kvasir .3348 / fireball .3023 / c2-logs .1143 / bluemoon .0964 / +cwm .0942 / govreport .0289 / **opening-split .0235** / qmsum .0055. + +**Grad-norm spikes, recorded rather than smoothed.** Four isolated points — +step 80 = 52.1, step 350 = 9.54, step 160 = 6.27, step 60 = 7.66 — against a body +at 0.34–1.9 from step 100 on. None moved the loss where it occurred, none +clustered, and a mid-run "every ~80 steps" hypothesis died at step 240 (0.398). +Read as occasional hard batches, not instability. Series shipped to brokkr and +written to `run-07/loss-series-r7.json` (sha `8322920f05ea6712`, 54 points, the +harness logs every 10 steps). + +⚠ **Sampler padding 17.1%** (run 6: 0.0%) — the short opening-split rows pair +badly under the length-paired sampler. Throughput only; it does not touch what +the optimiser saw, and it is why the clock read 14h rather than 12h. + +## Serving state after the run + + merge ~/ml/.venv/bin/python merge_lora.py \ + --base /home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a \ + --adapter run-07/adapter --out serve/merged-run07 \ + --chat-template run-07/adapter/chat_template.jinja + +⚠ **Merge under `~/ml/.venv`, not `~/vllm-env`** — the serving venv has no `peft` +and the merge dies in the import. ⚠ **`--chat-template` must be passed +explicitly**: the script defaults to `/tank/aimodels/…`, which does not exist on +the GX10 (no NFS). The adapter's own `chat_template.jinja` is the right file and +its sha is `ae53464b`, the stock 390-line upstream template — verify before +trusting it. + +Merged artifact `serve/merged-run07`: 48.1 GiB, 1013 tensors, +`Gemma4ForConditionalGeneration` bf16, template `ae53464b`, tokenizer `cc8d3a0c` +(stock), `processor_config.json` byte-identical to base and stock (`32bdf45d`). +The merge verified a sampled target actually changed — the check that catches a +silent no-op adapter. + + serve ./gpu-clear.sh && vllm serve --served-model-name erp-seat-base-ara \ + --host 0.0.0.0 --port 8098 --dtype bfloat16 --max-model-len 8192 \ + --max-num-seqs 8 --gpu-memory-utilization 0.60 \ + --enable-auto-tool-choice --tool-call-parser gemma4 + log vllm-base-run07gate.log pid file vllm-base-run07gate.pid + +`erp-seat-base-ara` came up 13:58 PT, health 200, `/v1/models` correct, round trip +verified at temperature 0. **`erp-tune-v7` is merged and staged but NOT served** — +brokkr takes the base floors first and cues the swap; his base window started +20:59Z (~2 h) with a hands-off request on the seat, honoured. diff --git a/persistent-memory.md b/persistent-memory.md index 9e82248..edbf5bb 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 14:05 PT (fleet-ops: ERP run 7 COMPLETE on pfi-gx10 — 542/542 steps, adapter 13:23 PT, merged to serve/merged-run07; `erp-seat-base-ara` SERVING on 10.100.50.60:8098 for brokkr's floors, `erp-tune-v7` staged awaiting his swap cue; Miranda notified. Booth gained the ASKS primitive [radio+notes -> answer sidecar, multi-question, verbatim-booth fix]. ana-ml2 pool actions landed. sox on nh3-dev for yt-voice-clipper-dev)_ +_Last updated: 2026-09-09 15:10 PT (fleet-ops: ERP run 7 COMPLETE — `erp-seat-base-ara` serving for brokkr's base window, `erp-tune-v7` staged awaiting his swap cue; Booth ASKS primitive shipped and then made INLINE per operator ruling; ana-ml2 pool actions landed; sox on nh3-dev; docs refreshed across booth/ana-ml2/nh3-dev/run-07 runbook)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -148,6 +148,7 @@ clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved ## Recent decisions - `[2026-09-09]` **ERP run 7 COMPLETE and the base arm is serving.** 542/542 steps in 14h17m on pfi-gx10, adapter 13:23 PT, `train_loss` 3.205 / low 2.799, merge verified a sampled target actually changed (the silent-no-op check). `erp-seat-base-ara` up on `10.100.50.60:8098` for brokkr's floors, `erp-tune-v7` merged and staged pending his cue; Miranda notified for the operator. Runbook `docs/runbooks/gx10-run-07.md`. +- `[2026-09-09]` **Booth asks render INLINE in a custom report, placed by the author** (v0.1.14) — operator ruling: *"the asks should be inline with the artifacts, not on a separate page."* Placeholders `data-booth-ask=""` / `":"` / `data-booth-ask-submit`, plus ``; per-question fragments bind to ONE form via the HTML5 `form=` attribute so a four-voice audition submits every pick in a single POST. ⚠ The placeholder must sit OUTSIDE any grid/flex parent or it becomes a cell (measured on `redo-anchors`: a 224 px sixth grid cell). Unplaced questions + a missing submit block are appended, so a partially marked-up page can never yield an unsubmittable 400 — a test caught that as a real drop. `redo-anchors/index.html` was hand-marked-up on the LIVE copy; tts-dev told to move it into the generator or a regeneration loses it. - `[2026-09-09]` **The Booth gained an ASKS primitive** (v0.1.12): a session drops `.ask.json` in a booth, the operator answers a radio form + notes in the browser, the pick lands as `.answer.json` the session reads (`booth ask|asks|answer --wait`). Multi-question form via a `questions` list. ⚠ Two defects found and fixed the same day: a booth serving its OWN `index.html` never rendered the panel (verbatim path returns early) → amber chip + standalone `/b//asks` page; and single-ask `title` was silently dropped. The `booth` CLI was ALSO not on PATH anywhere despite the global link-board convention telling every session to run it → symlinked to `~/.local/bin`. Global `CLAUDE.md` now teaches the primitive. - `[2026-09-09]` **ana-ml2 pool actions LANDED (scrub 0 errors in 1h33 → `zpool clear`; nvme-cli + full-drive SMART table; zroot 91→73% via dangling-image + builder prune, tagged rollback seats kept) — and the missed-scrub mystery SOLVED: nvme7 (slot 0-5, `S47VNY0K600221`) was absent from every boot 04-23→09-05, tank was raidz2-DEGRADED for 4½ months, Debian's scrub/trim cron only touches `ONLINE` pools, and ZED's alert went to a root mailbox with no MTA.** nvme7's 2084 media errors did not move across the scrub → historical, keep + watch. Playbook `playbooks/ana-ml2-pool-health.yaml`; inventory in `servers/ana-ml2/README.md`. → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md` - `[2026-09-09]` **ana-ml2 `tank`: 2 CKSUM errors on nvme7n1 after a boot-time resilver, NO scrub since 04-12, zroot 91% — three actions DEFERRED to a clean-context session** (scrub → nvme-cli SMART → docker prune), operator ruling "we'll do all 3 on clean context"; tracked at commit `3e18a04` + the post-clear handoff. ESH 10G links measured clean (fiber run live on UDM SFP+2 ↔ USW-Pro-XG Media). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md` diff --git a/servers/ana-ml2/README.md b/servers/ana-ml2/README.md index 6f673b1..394b669 100644 --- a/servers/ana-ml2/README.md +++ b/servers/ana-ml2/README.md @@ -44,8 +44,19 @@ at import. While it was missing `tank` was DEGRADED, and Debian's `zfsutils-linu (`/usr/lib/zfs-linux/{scrub,trim}`) only touches pools whose health is `ONLINE`, so tank got **no scrub and no trim from 04-12 to 09-06**. ZED's `ZED_EMAIL_ADDR=root` has no MTA behind it, so the 4½-month degradation alerted nobody. `media_errors=2084` on -nvme7 is a lifetime counter; the 2026-09-09 scrub is the first fresh measurement -(baseline 2084 at 00:32 PT — compare after any future event, growth = replace). +nvme7 is a lifetime counter. + +**Settled by the 2026-09-09 scrub** (00:29–02:02 PT, `scrub repaired 0B in 01:32:44 +with 0 errors`, then `zpool clear tank` → CKSUM 2 → 0): `media_errors` read **2084 +before and 2084 after** a full 6.84 TiB verify, so the counter is prior-life +history, not an active fault, and the 2 CKSUM were the stale-block artefact of the +09-05 late resilver. **nvme7 stays in service; watch the counter at every visit and +replace on growth** (`zpool replace tank nvme7n1 `; any PM1725b 1.6 TB or +larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on +visit — a bay that dropped a drive for 4½ months is the likelier fault than the +drive. Playbook: `playbooks/ana-ml2-pool-health.yaml` (idempotent; rerunning is a +no-op). ⚠ **Nothing alerts on this** — see the open follow-up in +`persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`. ## Key paths diff --git a/servers/nh3-dev/README.md b/servers/nh3-dev/README.md index 3b06d0d..ab9d5cf 100644 --- a/servers/nh3-dev/README.md +++ b/servers/nh3-dev/README.md @@ -25,6 +25,15 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo - **The Booth** — ephemeral media drop board (`:8090`, `booth.service`), from eshpfi `services/booth/`. Lets CC sessions surface A/B renders + smoke results (and browser uploads for pickup) to the operator; 24h TTL, Homepage-linked. + Since 2026-09-09 it also carries **asks** — a session poses a multiple-choice + question in a booth, the operator answers a radio form + notes in the browser, + and the pick lands as an answer sidecar the session reads (`booth ask` / + `booth answer --wait`). ⚠ The **`booth` CLI is on PATH via + `~/.local/bin/booth` → `services/booth/scripts/booth`**, symlinked 2026-09-09; + before that it was on no PATH at all, so every session following the global + link-board convention was hitting `command not found` unless it used the full + path. `~/.zshenv` puts `~/.local/bin` in PATH for non-interactive `ssh nh3-dev + ''` too. - **jackdaw-compose** — JackDAW AI Composer `/compose` backend (`:8787`, `jackdaw-compose.service`), a thin stateless `bun server/index.ts` from `~/development/jackdaw` → LiteLLM `gen`. Origin-gated (INV-BK04/BK05), reached diff --git a/services/booth/README.md b/services/booth/README.md index 3eb693e..dfade93 100644 --- a/services/booth/README.md +++ b/services/booth/README.md @@ -161,37 +161,55 @@ Rules of the primitive: - **A broken ask is shown as broken**, not hidden: if the JSON does not validate, the page says why, so a session never thinks it posted a question the operator cannot see. -- **A booth with its own `index.html` gets the ask INLINE, where you put it.** - That page is served verbatim, so the Booth substitutes placeholders in your - markup rather than rendering the panel above a gallery it does not have: - - ```html -
-
-
- - ``` - - Per-question fragments bind to **one** form with the HTML5 `form=` attribute, - so a four-voice audition can put each radio group under that voice's audio and - still submit all four picks in a single POST — which is what a multi-question - ask requires. The fragments ship their own scoped styles and inherit nothing - from your page. No JavaScript. - - ⚠ Put the placeholder **outside** any CSS grid or flex container, or it - becomes a cell in it. A sibling of the block it belongs to is right. - - Placement is optional: a page with no placeholders gets the whole ask appended - at the end, so an ask is never invisible — markup only moves it somewhere - better. Mark up some questions and not others and the rest are appended too, - because a multi-question form that is missing a question is a 400 the operator - would only meet after filling it in. The amber chip stays as a jump link to the - first open ask, and `/b//asks` still renders every ask on its own page. - Ask/answer files are not gallery items and do not count toward the booth's item count; they render as the panel above the gallery. Answering bumps the booth's mtime, so it lives another TTL — the session has 24h to read it. - Works with JavaScript off (plain form POST). No auth, same as everything here. +### Where the form renders + +Two booth shapes, two placements. Either way the ask is never invisible — that +is the guarantee; markup only moves it somewhere better. + +**Auto-gallery booth** (no `index.html` of its own): the asks panel renders +above the gallery, styled like the rest of the Booth. Nothing to do. + +**A booth serving its own `index.html`**: that page is returned verbatim, so the +Booth substitutes **placeholders in your markup** rather than rendering a panel +above a gallery that does not exist. The question then sits with the artifact it +is about, which is the whole point (operator ruling 2026-09-09: *"the asks should +be inline with the artifacts, not on a separate page"*). + +```html +
+
+
+ +``` + +Per-question fragments bind to **one** form via the HTML5 `form=` attribute, so a +four-voice audition puts each radio group under that voice's audio and still +submits every pick in a single POST — which is what a multi-question ask +requires. Fragments ship their own scoped styles, inherit nothing from your page, +and use no JavaScript. + +⚠ **Put the placeholder outside any CSS grid or flex container**, or it becomes a +cell in it — measured on `redo-anchors`, where the first attempt rendered as a +224 px sixth grid cell wedged between two audio players. A sibling of the block +it belongs to is right. + +The fallbacks, so a page can never strand a question: + +| you marked up | what happens | +|---|---| +| nothing | the whole ask is appended at the end of the page | +| some questions, no submit | the rest of the questions **and** a submit block are appended | +| a stem this booth does not have | your markup is left alone, untouched; the real ask is still appended | + +An amber `? N open asks` chip floats top-right as a jump link to the first open +ask, and `GET /b//asks` still renders every ask on a plain page of its own +— useful when you want to hand someone only the question. + ## Upload for pickup The reverse direction — put files in through the web, pick them up by id: