docs: refresh what today's work made stale — booth asks (inline placement promoted to its own section), ana-ml2 nvme7 settled by the scrub result, nh3-dev booth entry + the CLI-on-PATH fix, run-07 runbook outcome + serving state

2026-09-09 14:18:34 -07:00
parent 78c3a7c170
commit 6e0b85ba27
5 changed files with 137 additions and 29 deletions
@@ -55,3 +55,72 @@ tokenizer + template all unchanged from run 6.

 Gate choreography after the adapter lands: brokkr-smithy `SFT-RECIPE-run7-SCOPE.md §5` (cells
 LENGTH-FIXED / LENGTH-MOVED / BOUGHT-T4 / FLAT); serve names `erp-seat-base-ara` then `erp-tune-v7`.
+
+## Outcome — COMPLETE 2026-09-09 13:23 PT
+
+Ran to the end with no interventions, no resumes, no errors: **542/542 optimizer
+steps, one epoch, 14h17m wall clock** (23:06 PT 09-08 → 13:23 PT 09-09).
+
+| | |
+|---|---|
+| `train_loss` (run mean) | **3.205** |
+| last logged points | 2.9366 @ step 530 (lr 3.5e-07), 2.9389 @ step 540 (lr 1.86e-08) |
+| lowest logged | **2.7992 @ step 420** (from 7.1692 @ step 10) |
+| step seconds | n=541, p50 **82.1**, mean 94.1, min 17.4, max 334.3, p99 268.3 |
+| adapter | 410 tensors, 297 MB, `adapter_artifact` verified |
+
+Provenance (`run-07/adapter/provenance.json`), the fields worth re-reading before
+trusting any number: recipe `recipe-erp-seat-sft-r7.json` sha `2972559e4c2940dc`;
+harness `0a6bd2e0`, `harness_dirty_at_launch False`; **`flex_attention` requested
+AND resolved** (§4 of the training-throughput playbook — the backend delta that
+voided two earlier conclusions); `quantized_base False`; torch 2.14.0+cu130,
+transformers 5.16.1; chat template `ae53464b`; 8,661 encoded records from 8,505
+source samples; 6,106 masked turns; `unfittable_disposition drop` (cwm-2602,
+cwm-1826 — their loss moved into their opening-split rows, as designed).
+
+Effective mix as the optimiser saw it, identical to the free check, so the single
+variable landed: kvasir .3348 / fireball .3023 / c2-logs .1143 / bluemoon .0964 /
+cwm .0942 / govreport .0289 / **opening-split .0235** / qmsum .0055.
+
+**Grad-norm spikes, recorded rather than smoothed.** Four isolated points —
+step 80 = 52.1, step 350 = 9.54, step 160 = 6.27, step 60 = 7.66 — against a body
+at 0.34–1.9 from step 100 on. None moved the loss where it occurred, none
+clustered, and a mid-run "every ~80 steps" hypothesis died at step 240 (0.398).
+Read as occasional hard batches, not instability. Series shipped to brokkr and
+written to `run-07/loss-series-r7.json` (sha `8322920f05ea6712`, 54 points, the
+harness logs every 10 steps).
+
+⚠ **Sampler padding 17.1%** (run 6: 0.0%) — the short opening-split rows pair
+badly under the length-paired sampler. Throughput only; it does not touch what
+the optimiser saw, and it is why the clock read 14h rather than 12h.
+
+## Serving state after the run
+
+    merge   ~/ml/.venv/bin/python merge_lora.py \
+              --base /home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a \
+              --adapter run-07/adapter --out serve/merged-run07 \
+              --chat-template run-07/adapter/chat_template.jinja
+
+⚠ **Merge under `~/ml/.venv`, not `~/vllm-env`** — the serving venv has no `peft`
+and the merge dies in the import. ⚠ **`--chat-template` must be passed
+explicitly**: the script defaults to `/tank/aimodels/…`, which does not exist on
+the GX10 (no NFS). The adapter's own `chat_template.jinja` is the right file and
+its sha is `ae53464b`, the stock 390-line upstream template — verify before
+trusting it.
+
+Merged artifact `serve/merged-run07`: 48.1 GiB, 1013 tensors,
+`Gemma4ForConditionalGeneration` bf16, template `ae53464b`, tokenizer `cc8d3a0c`
+(stock), `processor_config.json` byte-identical to base and stock (`32bdf45d`).
+The merge verified a sampled target actually changed — the check that catches a
+silent no-op adapter.
+
+    serve   ./gpu-clear.sh && vllm serve <ARA base> --served-model-name erp-seat-base-ara \
+              --host 0.0.0.0 --port 8098 --dtype bfloat16 --max-model-len 8192 \
+              --max-num-seqs 8 --gpu-memory-utilization 0.60 \
+              --enable-auto-tool-choice --tool-call-parser gemma4
+    log     vllm-base-run07gate.log   pid file vllm-base-run07gate.pid
+
+`erp-seat-base-ara` came up 13:58 PT, health 200, `/v1/models` correct, round trip
+verified at temperature 0. **`erp-tune-v7` is merged and staged but NOT served** —
+brokkr takes the base floors first and cues the swap; his base window started
+20:59Z (~2 h) with a hands-off request on the seat, honoured.