diff --git a/persistent-memory.md b/persistent-memory.md index edbf5bb..ab6a01e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 15:10 PT (fleet-ops: ERP run 7 COMPLETE — `erp-seat-base-ara` serving for brokkr's base window, `erp-tune-v7` staged awaiting his swap cue; Booth ASKS primitive shipped and then made INLINE per operator ruling; ana-ml2 pool actions landed; sox on nh3-dev; docs refreshed across booth/ana-ml2/nh3-dev/run-07 runbook)_ +_Last updated: 2026-09-09 15:35 PT (fleet-ops: ERP run 7 COMPLETE → quantized NVFP4A16 and SERVING as the `trial` seat on ana-ml2 :8021; `erp-seat-base-ara` still up on gx10 for brokkr's base window, `erp-tune-v7` bf16 staged there awaiting his swap cue; Booth asks now inline + partial answers legal; ana-ml2 pool actions landed)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -147,6 +147,8 @@ clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved ## Recent decisions +- `[2026-09-09]` **run 7 quantized NVFP4A16 and serving as `trial`** — 49 GiB bf16 relayed gx10→ana-ml2 (16 min, 53 MB/s), quant 49→16 GiB via `services/erp-seat-quant/run_quant_erp_v7.sh` (dry-run gate passed: 11,725 targets / 11,520 experts, routers+vision BF16), seat on `:8021` under its TRUE name `erp-tune-v7-nvfp4a16`, LiteLLM `trial` repointed (config-file alias — `/model/update` REFUSES a config model, must edit `stacks/litellm/conf/config.yaml` + restart). Rollback: v6 artifact on disk + `/tmp/erp-seat-env.v6.bak`. ⚠ **`no direct path` was WRONG** — gx10↔ana-ml2 ROUTING is fine both ways; neither box holds a private key (only `authorized_keys`), so neither can *initiate*. `ssh -A` agent forwarding from nh3-dev gives a genuine direct path, verified. The relay costs nothing here anyway: both gx10 and nh3-dev are at NH3, so the WAN hop happens once either way. +- `[2026-09-09]` **Booth: partial ask answers are legal** (v0.1.15) — operator: the form failed when a question was left blank. `required` dropped from the radios; answered questions recorded, blanks land in `unanswered`, `complete` says whether the set is finished; refused only when there is no pick anywhere AND no notes. Reading sessions must check `complete`. - `[2026-09-09]` **ERP run 7 COMPLETE and the base arm is serving.** 542/542 steps in 14h17m on pfi-gx10, adapter 13:23 PT, `train_loss` 3.205 / low 2.799, merge verified a sampled target actually changed (the silent-no-op check). `erp-seat-base-ara` up on `10.100.50.60:8098` for brokkr's floors, `erp-tune-v7` merged and staged pending his cue; Miranda notified for the operator. Runbook `docs/runbooks/gx10-run-07.md`. - `[2026-09-09]` **Booth asks render INLINE in a custom report, placed by the author** (v0.1.14) — operator ruling: *"the asks should be inline with the artifacts, not on a separate page."* Placeholders `data-booth-ask=""` / `":"` / `data-booth-ask-submit`, plus ``; per-question fragments bind to ONE form via the HTML5 `form=` attribute so a four-voice audition submits every pick in a single POST. ⚠ The placeholder must sit OUTSIDE any grid/flex parent or it becomes a cell (measured on `redo-anchors`: a 224 px sixth grid cell). Unplaced questions + a missing submit block are appended, so a partially marked-up page can never yield an unsubmittable 400 — a test caught that as a real drop. `redo-anchors/index.html` was hand-marked-up on the LIVE copy; tts-dev told to move it into the generator or a regeneration loses it. - `[2026-09-09]` **The Booth gained an ASKS primitive** (v0.1.12): a session drops `.ask.json` in a booth, the operator answers a radio form + notes in the browser, the pick lands as `.answer.json` the session reads (`booth ask|asks|answer --wait`). Multi-question form via a `questions` list. ⚠ Two defects found and fixed the same day: a booth serving its OWN `index.html` never rendered the panel (verbatim path returns early) → amber chip + standalone `/b//asks` page; and single-ask `title` was silently dropped. The `booth` CLI was ALSO not on PATH anywhere despite the global link-board convention telling every session to run it → symlinked to `~/.local/bin`. Global `CLAUDE.md` now teaches the primitive. diff --git a/services/erp-seat-quant/run_quant_erp_v7.sh b/services/erp-seat-quant/run_quant_erp_v7.sh new file mode 100755 index 0000000..4bf3bab --- /dev/null +++ b/services/erp-seat-quant/run_quant_erp_v7.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# NVFP4A16 quant of the ERP run-7 merged model on ana-ml2 GPU1 (co-resident with the GPU1 seats; +# CPU-resident load, per-layer onload). Detached container; watch with `docker logs -f erp-v7-quant`. +# NOTE: no PYTORCH_CUDA_ALLOC_CONF=expandable_segments (playbook §3.10). +set -euo pipefail +WORK=/tank/aimodels/erp-tune-v7-quant-work +SRC="${1:-/tank/aimodels/erp-tune-v7-bf16}" +OUT="${2:-/tank/aimodels/erp-tune-v7-nvfp4a16}" +MODE="${3:-full}" # full | dry-run +EXTRA=""; [ "$MODE" = "dry-run" ] && EXTRA="--dry-run" +NAME=erp-v7-quant; [ "$MODE" = "dry-run" ] && NAME=erp-v7-quant-dry +docker rm -f "$NAME" 2>/dev/null || true +docker run -d --name "$NAME" --gpus '"device=1"' --ipc host \ + -v /tank/aimodels:/tank/aimodels \ + --entrypoint python3 vllm-llmcompressor:latest \ + "$WORK/quant_nvfp4a16_gemma4_moe.py" --model "$SRC" --out "$OUT" \ + --num-samples "${NUM_SAMPLES:-256}" --seqlen "${SEQLEN:-8192}" $EXTRA +echo "launched $NAME: $(docker ps --filter name=$NAME --format '{{.Status}}')" diff --git a/stacks/erp-seat/README.md b/stacks/erp-seat/README.md index 469f2f7..a1d94fb 100644 --- a/stacks/erp-seat/README.md +++ b/stacks/erp-seat/README.md @@ -1,11 +1,18 @@ # erp-seat — ERP-tune seat on ana-ml2 (GPU1, `:8021`) Serves the latest gated ERP LoRA merge as an **NVFP4A16** (weight-only) compressed-tensors -checkpoint so the GX10 is free to train the next run. First occupant: **run 6** — -`erp-tune-v6-nvfp4a16` = merged-run06 (jenerallee78 ARA-abliterated Gemma-4-26B-A4B-it, index -`33c59654…`, + R47 SFT r6) quantized by `services/erp-seat-quant/`. +checkpoint so the GX10 is free to train the next run. -- **True name only.** `--served-model-name erp-tune-v6-nvfp4a16`. Gateway aliases (`trial`) are +**Current occupant: run 7** — `erp-tune-v7-nvfp4a16` = merged-run07 (jenerallee78 +ARA-abliterated Gemma-4-26B-A4B-it, index `33c59654…`, + R47 SFT r7 = r6 plus the opening-split +slot and its companion loss mask), quantized by `services/erp-seat-quant/` on 2026-09-09. +49 GiB bf16 → 16 GiB NVFP4A16. Runbook `docs/runbooks/gx10-run-07.md`. + +*Previous: run 6 (`erp-tune-v6-nvfp4a16`, 2026-09-08). Its artifact is still on `/tank/aimodels/` +and the pre-swap host env is at `/tmp/erp-seat-env.v6.bak` on ana-ml2, so a rollback is an `.env` +flip plus `docker compose up -d`.* + +- **True name only.** `--served-model-name erp-tune-v7-nvfp4a16`. Gateway aliases (`trial`) are set in LiteLLM on the operator's word, never here (no silent substitution — the bf16 arm on the GX10 and this NVFP4 arm are different artifacts). - **Recipe** = `stacks/gemma4-charrp` (same arch + format, proven on this box): `gemma4` tool diff --git a/stacks/erp-seat/compose.yaml b/stacks/erp-seat/compose.yaml index 3cb3b67..bf41c67 100644 --- a/stacks/erp-seat/compose.yaml +++ b/stacks/erp-seat/compose.yaml @@ -1,5 +1,5 @@ # erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of the latest gated ERP LoRA merge -# (run 6 = jenerallee78 ARA-abliterated Gemma-4-26B-A4B + R47 SFT), served under its TRUE name. +# (run 7 = jenerallee78 ARA-abliterated Gemma-4-26B-A4B + R47 SFT + the opening-split slot), served under its TRUE name. # Routing aliases (e.g. LiteLLM `trial`) are the operator's call and live in the gateway, not here. # # Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this @@ -23,11 +23,11 @@ services: environment: - VLLM_API_KEY=${API_KEY:-} command: - - ${ERP_MODEL:-/tank/aimodels/erp-tune-v6-nvfp4a16} + - ${ERP_MODEL:-/tank/aimodels/erp-tune-v7-nvfp4a16} - --quantization - compressed-tensors - --served-model-name - - ${ERP_SERVED_NAME:-erp-tune-v6-nvfp4a16} + - ${ERP_SERVED_NAME:-erp-tune-v7-nvfp4a16} - --tool-call-parser - gemma4 - --enable-auto-tool-choice @@ -47,7 +47,7 @@ services: # an empty turn. The flag drops the tools from the prompt so the model answers in prose. - --exclude-tools-when-tool-choice-none - --chat-template - - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja} + - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v7-nvfp4a16/chat_template.jinja} - --max-model-len - "${ERP_MAX_MODEL_LEN:-32768}" - --max-num-seqs @@ -76,9 +76,9 @@ services: - tnet labels: - homepage.group=AI - Inference - - homepage.name=erp-tune-v6 (Gemma-4 26B-A4B ARA, NVFP4A16) + - homepage.name=erp-tune-v7 (Gemma-4 26B-A4B ARA, NVFP4A16) - homepage.icon=mdi-fire - - homepage.description=ERP-seat run-6 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1) + - homepage.description=ERP-seat run-7 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1) - homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs networks: diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index f6e0803..39b7963 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -851,23 +851,30 @@ model_list: # api_base: http://10.250.50.54:8099/v1 # api_key: os.environ/VLLM_API_KEY - # trial -> ERP/RP SFT RUN 6, NVFP4A16 (weight-only) quant of the merged LoRA, served on - # ana-ml2 GPU1 (:8021, vLLM v0.26.0, stacks/erp-seat -- a REAL compose stack with a restart - # policy, unlike the run-3c/run-5 hand-launched gx10 seats). Backing artifact - # /tank/aimodels/erp-tune-v6-nvfp4a16 (quant pipeline services/erp-seat-quant/). + # trial -> ERP/RP SFT RUN 7, NVFP4A16 (weight-only) quant of the merged LoRA, served on + # ana-ml2 GPU1 (:8021, vLLM nightly 311b3513, stacks/erp-seat -- a REAL compose stack with a + # restart policy, unlike the run-3c/run-5 hand-launched gx10 seats). Backing artifact + # /tank/aimodels/erp-tune-v7-nvfp4a16 (quant pipeline services/erp-seat-quant/). # - # NAME: deliberately `trial`, NOT `erp-tune-v6-nvfp4a16`. Repointed 2026-09-08 (evening) - # from run 5 (bf16 on gx10) to run 6 (NVFP4 on ana-ml2) at the operator's request -- "alias - # erp-tune-v6-nvfp4 to trial, please. no gate." -- to hand-test prosody/feel in Open WebUI. + # NAME: deliberately `trial`, NOT `erp-tune-v7-nvfp4a16`. Repointed 2026-09-09 (afternoon) + # from run 6 to run 7 at the operator's request -- "quant the latest train to nvfp4 and serve + # on ana-ml2 as the trial seat". Same standing purpose: hand-testing prosody/feel in Open + # WebUI. (Repointed 2026-09-08 from run 5 to run 6 on the same standing instruction.) # - # BASE: run 6 is the line's first GENUINELY abliterated base (jenerallee78 ARA @ 0631379a, - # index 33c59654); runs 3-5 were settled from bytes as stock. Run-5 recipe byte-held. + # RUN 7 = run 6 + ONE variable: an opening-split slot (293 rows) and its companion loss mask + # (224 entries, union with lossmask-r3). Base HELD from run 6 (jenerallee78 ARA @ 0631379a, + # index 33c59654). Trained 542/542 steps, train_loss 3.205, adapter 2026-09-09 13:23 PT. + # Runbook docs/runbooks/gx10-run-07.md. # - # GATE: NONE on this artifact, by operator ruling. brokkr's run-6 gate runs against the bf16 - # arm on gx10:8098 (erp-tune-v6), not this NVFP4 build; this seat has a smoke test and a - # 3-run decode probe (~200 tok/s single-stream) only. Refusal behaviour is expected to be - # LOWER than run 5's (abliterated base + k=5 base profile ~0% on 30/35 axes per brokkr) -- - # this seat is for the operator's ear; treat it as unrated on every safety axis. + # GATE: NONE on this artifact, by operator ruling -- unchanged from run 6. brokkr's run-7 + # gate runs against the BF16 arm on gx10:8098, not this NVFP4 build; this seat has a smoke + # test only. Refusal behaviour is expected to be LOWER than run 5's (abliterated base + k=5 + # base profile ~0% on 30/35 axes per brokkr) -- this seat is for the operator's ear; treat + # it as unrated on every safety axis. + # + # ROLLBACK: /tank/aimodels/erp-tune-v6-nvfp4a16 is still on disk, and the previous host env + # is at /tmp/erp-seat-env.v6.bak on ana-ml2 -- flip ERP_MODEL/ERP_SERVED_NAME/ERP_CHAT_TEMPLATE + # in /opt/docker/compose/erp-seat/.env back to v6 and `docker compose up -d`. # # THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here, # in every session and project. Removing this alias does not remove the operator's @@ -877,7 +884,7 @@ model_list: - model_name: trial litellm_params: - model: hosted_vllm/erp-tune-v6-nvfp4a16 + model: hosted_vllm/erp-tune-v7-nvfp4a16 api_base: http://10.250.50.54:8021/v1 api_key: os.environ/VLLM_API_KEY