From dae77ee118be627cda6561c07868c0a8d0da3f40 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 3 Sep 2026 22:46:28 -0700 Subject: [PATCH] =?UTF-8?q?feat(pfi-gx10):=20stage=20ERP-seat=20SFT=20run?= =?UTF-8?q?=203c=20=E2=80=94=20verified,=20not=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base, recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h) and correct — an Anaheim breaker trip costs a 40-minute drive each way and 13 hosts down, three of them SureFire client machines, while the GX10 is a ~240 W appliance at NH3 that can take nothing else down. Verified rather than assumed, because ana-ml2 ran transformers 5.15.1 on x86-64 and this box runs 5.16.1 on aarch64 — the silent backend-delta class that has already voided conclusions here: - both 49 GB base shards sha256-match ana-ml2's (size equality is a weaker claim and was already true) - a full encode was run into a throwaway dir and the encoded corpus compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3, identical. Every aggregate matched too. That verified artifact is what the run will train on — it is seeded into run-03c/encode-cache - the harness's own suite: 122 passed on aarch64 - the config generator asserts key-by-key that no non-path value differs from run-03c.json The encode-cache filename differs by design (base_model_path is part of the key) — an input hash, not an output hash. Documented so it is not misread as drift, or "fixed" by faking /tank on this box. Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on the same subnet, which makes mounting it tempting and still wrong under a 13 h unattended run. The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f erp_sft_harness` invoked over ssh matches the invoking shell's own argv. That self-match cost a shell during staging. Not launched. 13.3 h is the operator's call. --- docs/runbooks/gx10-run-03c.md | 115 ++++++++++++++++++ .../2026-09-03-gx10-run3c-staged.md | 91 ++++++++++++++ persistent-memory.md | 10 +- scripts/erp-tune-gx10/README.md | 20 +++ scripts/erp-tune-gx10/launch-run-03c.sh | 73 +++++++++++ scripts/erp-tune-gx10/run-03c-gx10.json | 47 +++++++ servers/pfi-gx10/README.md | 57 +++++---- 7 files changed, 389 insertions(+), 24 deletions(-) create mode 100644 docs/runbooks/gx10-run-03c.md create mode 100644 persistent-memory.d/2026-09-03-gx10-run3c-staged.md create mode 100644 scripts/erp-tune-gx10/README.md create mode 100755 scripts/erp-tune-gx10/launch-run-03c.sh create mode 100644 scripts/erp-tune-gx10/run-03c-gx10.json diff --git a/docs/runbooks/gx10-run-03c.md b/docs/runbooks/gx10-run-03c.md new file mode 100644 index 0000000..1057ec4 --- /dev/null +++ b/docs/runbooks/gx10-run-03c.md @@ -0,0 +1,115 @@ +# pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch + +Staged 2026-09-03. **Not launched** — the launch is the operator's call. + +Run 3c is the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim +breaker tripped. It is rehomed to pfi-gx10 unchanged: same corpus, same base, +same hyperparameters, different host. + +## Why this box and not the fast one + +ana-ml2 finishes this run in ~2.5 h and pfi-gx10 needs ~13.3 h, and the slow box +is still correct. An Anaheim power event is not priced in lost training steps — +it is a 40-minute drive each way on the operator's time, with 13 hosts down +including `pbs-ana` (the fleet's primary backup server) and **three SureFire +client machines** under a hosting agreement. The GX10 is a ~240 W appliance at +NH3 that can take nothing else down, and nothing is waiting on this run. + +Power-capping ana-ml2 is not a third option: both RTX PRO 6000s floor at 250 W +against a 300 W default, so the whole lever is 100 W on a ~1 kW box. + +## Layout on the box + + /home/infra-ops/models/gemma4-26b-a4b-it-bf16 49 GB base (sha-verified, see below) + /home/infra-ops/erp-tune/ + eitri-smithy/ harness, git 0a6bd2e, tracked tree clean + recipe-r3/ recipe, survivors, loss-mask + datasets/derived/ 2.4 GB corpus -- COPIED, see the NFS note + datasets/holdout/ holdout, for the disjointness assertion + run-03c/encode-cache/ pre-seeded with the VERIFIED encode (below) + run-03c-gx10.json the config + launch-run-03c.sh the launcher + /home/infra-ops/ml/.venv torch 2.14.0+cu130, transformers 5.16.1, peft 0.20.0 + +Canonical copies of the config and launcher live in this repo under +`scripts/erp-tune-gx10/`. + +## ⚠ The corpus is COPIED, and the box mounts no NFS at all + +`/mnt/smithy` is deliberately **not** mounted here. A 13-hour unattended run is +the worst possible place for a hard NFS dependency — the fleet has already lost +a host to a hard mount wedging into unkillable D-state. The corpus is 2.4 GB and +copied in 50 s from nh3-dev's mount; there is nothing to gain by mounting it. + +Verify before every launch: `mount | grep -E 'nfs|cifs'` returns nothing. + +## What was verified, rather than assumed + +**The base model is byte-identical to ana-ml2's.** Both shards sha256-matched +across 51.6 GB, not just size-checked: + + model-00001-of-00002.safetensors 1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3 + model-00002-of-00002.safetensors aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7 + +**The tokenizer did not drift across the library delta.** ana-ml2 ran +transformers 5.15.1 / torch 2.13.0 on x86-64; this box runs 5.16.1 / 2.14.0+cu130 +on aarch64. That is exactly the silent backend-delta class that has already +voided conclusions on this project, so it was measured, not reasoned about: a +full encode was run here into a throwaway output dir and the encoded corpus +compared byte-for-byte against ana-ml2's. + + ana-ml2 encoded-c16316f1c1bb21da.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3… + pfi-gx10 encoded-fd8fe1944fb316b2.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3… + +Identical. Every aggregate matched too — 9,504 training vs 8,404 holdout ids with +0 overlap, 15 unfittable dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930 +tokens, and the five mix shares to four decimals. + +⚠ **The cache-key filenames differ and that is correct, not a mismatch.** +`base_model_path` is part of the encode-cache key by design, so rehoming the base +changes the key while leaving the content identical. The key is an input hash; +the sha above is the output. Do not read the differing filenames as drift. + +That verified artifact was then copied into `run-03c/encode-cache/`, so the run +trains on the exact bytes that were compared — it will report `[encode] cache +hit` rather than re-encoding. + +**The harness runs on aarch64.** Its own suite: 122 passed. + +**Throughput.** ~79.4 s/it measured on this exact geometry (median of 10 timed +steps, 0.19% spread), peak 75.1 of 121.6 GiB. 604 steps ≈ 13.3 h. Against +ana-ml2's 10.8–15.8 s/it that is ~6× slower where raw compute predicts ~2.7×; +the unconfirmed hypothesis is unified-LPDDR5X bandwidth, not a misconfiguration. + +## Launch + + ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh' + +The script refuses rather than proceeds if the GPU is not clear, if +`run-03c.pid` names a live process, if the log already exists, if any input is +missing, or if there is under 40 GB free. It detaches with `setsid nohup` and +logs on the box, because a long remote job piped through a foreground SSH has +already been reaped once on this hardware — the work survived, the output did not. + +Watching, and stopping: + + tail -f ~/erp-tune/run-03c.log | tr '\r' '\n' + kill $(cat ~/erp-tune/run-03c.pid) + +⚠ **Never `pkill -f erp_sft_harness` over SSH.** The pattern appears in your own +ssh argv, so it kills your shell as well as the run. This bit again during +staging. Kill by PID. The launcher's own guard uses the pidfile for the same +reason. + +## What to expect in the log + +Order is preflight → encode (cache hit) → mix report → weight load (~2 min) → +sampler → train. Checkpoints land every 50 steps at ~852 MB each, so ~10 GB over +the run against 764 GB free. First checkpoint ≈ 66 minutes in. + +## Not verified + +The harness's train loop has not been exercised end to end on this silicon — the +throughput baseline used a synthetic replica of the geometry, and the staging +encode was killed before the weight load. If something is going to break, it +breaks in the first two minutes after the sampler line. diff --git a/persistent-memory.d/2026-09-03-gx10-run3c-staged.md b/persistent-memory.d/2026-09-03-gx10-run3c-staged.md new file mode 100644 index 0000000..487b56d --- /dev/null +++ b/persistent-memory.d/2026-09-03-gx10-run3c-staged.md @@ -0,0 +1,91 @@ +# `[2026-09-03]` Run 3c STAGED on pfi-gx10 — verified end to end, deliberately NOT launched + +The ERP-seat SFT LoRA that died on ana-ml2 at step 24 of 604 to an Anaheim breaker trip is +now staged on pfi-gx10, unchanged. **The launch is the operator's call and was not taken** — +he stood this port down once before, so a 13.3 h commitment is not an agent default. + +Runbook `docs/runbooks/gx10-run-03c.md`; canonical config + launcher +`scripts/erp-tune-gx10/`; on the box `/home/infra-ops/erp-tune/`. + + ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh' + +## What is on the box + + ~/models/gemma4-26b-a4b-it-bf16 49 GB base, ALREADY THERE from the 09-01 probe + ~/erp-tune/eitri-smithy harness, git 0a6bd2e, tracked tree clean + ~/erp-tune/recipe-r3 recipe / survivors / loss-mask + ~/erp-tune/datasets/{derived,holdout} 2.4 GB, COPIED (50 s at 49 MB/s from nh3-dev) + ~/erp-tune/run-03c/encode-cache PRE-SEEDED with the verified encode + ~/ml/.venv + protobuf, pytest (the only two gaps vs ana-ml2) + +⚠ **The corpus is copied and the box mounts NO NFS.** `/mnt/smithy` lives on nh3-nas, now on +the *same subnet* as the racked GX10 — which makes mounting it tempting and still wrong. A +13 h unattended run is the worst place for a hard NFS dependency +([[incident_esh_docker_nfs_boot_race]]). 2.4 GB copies in under a minute; there is nothing to +buy. + +## The verification that actually mattered — and it was NOT free reasoning + +ana-ml2 ran transformers 5.15.1 / torch 2.13.0 on x86-64. The GX10 runs 5.16.1 / 2.14.0+cu130 +on aarch64. That is precisely the silent backend-delta class CLAUDE.md records as having voided +two frontier-panel conclusions. So it was **measured**: a full encode was run into a throwaway +output dir and the encoded corpus compared byte-for-byte. + + ana-ml2 encoded-c16316f1c1bb21da.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3... + pfi-gx10 encoded-fd8fe1944fb316b2.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3... + +**Byte-identical.** Every aggregate matched too: 9,504 vs 8,404 ids / 0 overlap, 15 unfittable +dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930 tok, five mix shares to 4 dp. + +⚠ **The cache-key FILENAMES differ and that is correct, not drift.** `base_model_path` is in +the encode-cache key *by design* (so a different base cannot silently reuse an encode), and +rehoming the base changes the key while leaving content identical. **The key is an input hash; +the sha is the output.** Do not read the differing filenames as a mismatch — and do not +"fix" it by symlinking `/tank/aimodels` onto this box to force a key match. That verified +artifact was then copied into `run-03c/encode-cache/`, so the run trains on the exact bytes +compared and will report `[encode] cache hit`. + +Also verified rather than assumed: **both 49 GB base shards sha256-match ana-ml2's** (size +equality was already true and is not the same claim), the harness's own suite is **122 passed** +on aarch64, and every one of the config's 8 path keys resolves to an existing local file. + +## The config is provably the same run + +`run-03c-gx10.json` = ana-ml2's `run-03c.json` with 8 path keys rehomed and 2 +`substitute_controls` entries appended (host move; library delta). A generator asserted +**key-by-key that no non-path value differs** rather than eyeballing a diff — lr 1e-05, rank 64, +alpha 128, seq 16384, batch 2 x accum 8, save_steps 50, seed 20260824 all intact, and the +existing 10 substitute_controls are a byte-identical prefix. + +## ⚠ I TRIPPED THE pkill SELF-MATCH AGAIN, ~20 MINUTES AFTER READING THE MEMORY ABOUT IT + +`ssh gx10 'pkill -f "erp_sft_harness --config .../encode-check.json"'` — the pattern is in the +remote shell's OWN argv, so it killed my shell alongside the target and the command returned +nothing. [[feedback_pkill_ssh_self_match]] describes this exactly. Reading the memory did not +prevent it; **the guard has to be in the artifact, not in recall.** + +So the launcher's already-running guard is a **pidfile**, not a pgrep — `pgrep -f +erp_sft_harness` in a script invoked over ssh matches the invoking shell and would refuse every +launch. Same root cause, and it would have presented as a mysterious always-refusing launcher. + +## The launcher's other guards, each bought with a past failure + + GPU-clear assertion a stuck orphan held 80 GB while PyTorch reported 0 allocated; + every relaunch was doomed and blamed the NEW run + setsid nohup + on-box log a foreground ssh reaped the 09-01 probe: work survived, output did not + log-exists refusal two runs must not share a log + >=40 GB free 12 checkpoints x 852 MB (measured off run-03, not estimated) + +## Why the slow box is still the right box (unchanged, restated because it is the whole case) + +~79.4 s/it here vs 10.8-15.8 on ana-ml2 -> 13.3 h vs ~2.5 h. An Anaheim breaker trip is not +priced in lost steps: it is a 40-minute drive **each way** on the operator's time, 13 hosts +down including `pbs-ana` and **three SureFire client machines**. Nothing is waiting on this run, +so the slowness is close to free. + +## NOT verified — the honest gap + +The harness's **train loop** has not run end to end on sm_121. The 79.4 s/it baseline used a +synthetic replica of the geometry, and the staging encode was killed before the weight load. +If it breaks, it breaks in the first two minutes after the `[sampler]` line — roughly three +minutes after launch, well before the first checkpoint at ~66 min. diff --git a/persistent-memory.md b/persistent-memory.md index db3ac18..30a4580 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -112,8 +112,13 @@ _As of 2026-09-03 — **nothing is mid-action; the session closed clean.** Every item below is a live commitment or a known-open risk, not work in progress._ - **Nothing is running.** No training, no deploys pending, no background jobs. - 35 commits sit unpushed on `main` — all docs, runbooks, memory and two - scripts; push is the operator's call. + Commits sit unpushed on `main` — all docs, runbooks, memory and scripts; + push is the operator's call. + +- **Run 3c is staged on pfi-gx10 and awaiting the operator's go.** Everything is + verified and one command away (`ssh infra-ops@10.100.50.60 + '~/erp-tune/launch-run-03c.sh'`); ~13.3 h once started. Its train loop is the + one piece never exercised on sm_121 — watch the first three minutes. - **`web_search` needs a session restart to appear.** The SearXNG MCP server is registered at user scope and `claude mcp list` reports it Connected, but MCP @@ -145,6 +150,7 @@ item below is a live commitment or a known-open risk, not work in progress._ ## Recent decisions +- `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md` - `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md` - `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I ` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md` - `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md` diff --git a/scripts/erp-tune-gx10/README.md b/scripts/erp-tune-gx10/README.md new file mode 100644 index 0000000..2b35223 --- /dev/null +++ b/scripts/erp-tune-gx10/README.md @@ -0,0 +1,20 @@ +# erp-tune on pfi-gx10 + +Canonical copies of what is staged on pfi-gx10 for ERP-seat SFT **run 3c** — +the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim breaker tripped. + +| file | deployed to | +|---|---| +| `run-03c-gx10.json` | `/home/infra-ops/erp-tune/run-03c-gx10.json` | +| `launch-run-03c.sh` | `/home/infra-ops/erp-tune/launch-run-03c.sh` | + +`run-03c-gx10.json` is `ana-ml2:/tank/erp-tune/run-03c.json` with its eight path +keys rehomed to local NVMe and two `substitute_controls` entries appended +recording the host move and the library delta. **No hyperparameter, corpus, +base, mask or recipe value differs** — verified key-by-key at generation. + +Full context, what was verified and how to launch: +[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md). + +The harness itself (`eitri-smithy`) is not vendored here; it lives on the box at +`/home/infra-ops/erp-tune/eitri-smithy`, git `0a6bd2e`. diff --git a/scripts/erp-tune-gx10/launch-run-03c.sh b/scripts/erp-tune-gx10/launch-run-03c.sh new file mode 100755 index 0000000..703b4c3 --- /dev/null +++ b/scripts/erp-tune-gx10/launch-run-03c.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# Launch ERP-seat SFT run 3c on pfi-gx10 (NVIDIA GB10, aarch64, sm_121). +# +# Run this ON pfi-gx10 as infra-ops. It detaches the job from the invoking +# shell and logs to the box, so a reaped SSH session cannot take the run with +# it -- the failure mode that lost the first probe launch on 2026-09-01. +# +# Expected: 604 optimizer steps at ~79.4 s/it => ~13.3 h. +# Checkpoints every 50 steps, ~852 MB each (~10 GB total). +set -euo pipefail + +ROOT=/home/infra-ops/erp-tune +HARNESS=$ROOT/eitri-smithy +VENV=/home/infra-ops/ml/.venv/bin/python +CONFIG=$ROOT/run-03c-gx10.json +LOG=$ROOT/run-03c.log + +# --- Preconditions, asserted rather than assumed ----------------------------- + +# A stuck orphan holding unified memory while PyTorch reports zero allocated +# already doomed three relaunches on this box and got blamed on the new run +# each time. Assert the GPU is clear. +apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]') +if [ -n "$apps" ]; then + echo "REFUSING: GPU is not clear -- compute apps still resident:" >&2 + nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2 + exit 1 +fi + +# Deliberately NOT `pgrep -f erp_sft_harness`: run this over ssh and the +# pattern appears in the invoking shell's own argv, so the guard matches +# itself and refuses every launch. Same self-match that makes `pkill -f` +# unsafe over ssh. The pidfile is exact and cannot self-match; the GPU +# assertion above catches an orphan under any name. +if [ -f "$ROOT/run-03c.pid" ] && kill -0 "$(cat "$ROOT/run-03c.pid")" 2>/dev/null; then + echo "REFUSING: run-03c.pid names a live process $(cat "$ROOT/run-03c.pid"):" >&2 + ps -p "$(cat "$ROOT/run-03c.pid")" -o pid,etime,cmd >&2 + exit 1 +fi + +if [ -e "$LOG" ]; then + echo "REFUSING: $LOG exists. Move it aside first so two runs cannot share a log." >&2 + exit 1 +fi + +for p in "$HARNESS/erp_sft_harness/__main__.py" "$VENV" "$CONFIG"; do + [ -e "$p" ] || { echo "REFUSING: missing $p" >&2; exit 1; } +done + +# Free space for checkpoints: 12 x 852 MB + final adapter, with headroom. +avail=$(df --output=avail -BG "$ROOT" | tail -1 | tr -dc '0-9') +if [ "$avail" -lt 40 ]; then + echo "REFUSING: only ${avail}G free under $ROOT; want >=40G for checkpoints." >&2 + exit 1 +fi + +# --- Launch ------------------------------------------------------------------ + +cd "$HARNESS" +{ + echo "# launched $(date -Is) on $(hostname) by ${USER}" + echo "# harness $(git rev-parse --short HEAD) config $CONFIG" +} > "$LOG" + +setsid nohup "$VENV" -m erp_sft_harness --config "$CONFIG" >> "$LOG" 2>&1 < /dev/null & +pid=$! +echo "$pid" > "$ROOT/run-03c.pid" + +echo "launched pid $pid -> $LOG" +echo +echo "watch: tail -f $LOG | tr '\\r' '\\n'" +echo "steps: grep -ao '[0-9]*/604 \[[^]]*\]' $LOG | tail -1" +echo "stop: kill \$(cat $ROOT/run-03c.pid) # by PID -- never pkill -f over ssh" diff --git a/scripts/erp-tune-gx10/run-03c-gx10.json b/scripts/erp-tune-gx10/run-03c-gx10.json new file mode 100644 index 0000000..9303ac8 --- /dev/null +++ b/scripts/erp-tune-gx10/run-03c-gx10.json @@ -0,0 +1,47 @@ +{ + "output_dir": "/home/infra-ops/erp-tune/run-03c", + "roots_dir": "/home/infra-ops/erp-tune/datasets/derived", + "base_model_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16", + "base_model_revision": "google/gemma-4-26B-A4B-it (stock instruct, upstream) -- UNCHANGED from runs 2 and 3; run 3c varies the LEARNING RATE alone", + "recipe": "/home/infra-ops/erp-tune/recipe-r3/recipe-erp-seat-sft-r3.json", + "survivors": "/home/infra-ops/erp-tune/recipe-r3/survivors-r3.jsonl", + "chat_template_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16/chat_template.jinja", + "impersonation_mask_path": "/home/infra-ops/erp-tune/recipe-r3/lossmask-r3.jsonl", + "lora_rank": 64, + "lora_alpha": 128, + "lora_dropout": 0.0, + "max_seq_len": 16384, + "epochs": 1, + "seed": 20260824, + "per_device_batch_size": 2, + "gradient_accumulation_steps": 8, + "learning_rate": 1e-05, + "warmup_ratio": 0.1, + "lr_scheduler_type": "cosine", + "weight_decay": 0.01, + "load_in_4bit": false, + "gradient_checkpointing": true, + "loss_chunk_tokens": 1024, + "training_eligibility_override": "operator-2026-08-26-run3-go", + "overridden_blockers": [ + "contamination-scan-not-implemented", + "stage-2-csam-detector-inert" + ], + "substitute_controls": [ + "pre-training holdout (8,002 samples, work/card/session split)", + "stage-A lexical quarantine, RP only (829 records held)", + "F1 pippa root excision (target-list change, recipe erp-seat-sft-r3)", + "F2/F3/F4 low-quality loss-mask, 5,867 turns (length floor 250w, register cap 20 asterisks/1kw, placeholder leak) -- MASKED not deleted", + "impersonation loss-mask unioned into the same file, 16 turns not covered by F2/F3/F4; 5,883 masked turns total, sha-asserted", + "kvasir subset to the recipe's stated 38.0% context share (3,498 of 12,003, seeded shuffle at run seed) -- the recipe names the share but stages no selection; see recipe-r3/RUN-03-BUILD-NOTE.md", + "DEVIATION RECORDED: recipe token accounting is not the harness tokenizer; bluemoon is NOT the largest loss contributor as trained (32.9% vs c2-logs 35.6%) -- see RUN-03-BUILD-NOTE.md section 4", + "SINGLE VARIABLE vs run 3: learning_rate 0.0002 -> 1e-05 (20x cut). Corpus BYTE-IDENTICAL (survivors-r3.jsonl, lossmask-r3.jsonl, same shas); base, max_seq_len 16384, rank 64, alpha 128, dropout 0.0, cosine, warmup 0.1, wd 0.01, batch 2, 1 epoch and target_modules ALL UNCHANGED. Rationale: the zerofata MeroMero v1 card, on this exact base, warns \"Google left the instruct model at the razor's edge of overfitting; finetune it at all and it feels like it'll rapidly lose intelligence\" -- and MeroMero v1 trains at 1e-5. Run 3 used 20x that.", + "OPERATIONAL DEVIATION (not scientific): save_steps 100 -> 50. Checkpoint cadence only; it changes nothing about the optimisation, the corpus or the single-variable lr claim above. Recorded as a separate entry so the two deviations cannot be confused. Rationale: an Anaheim site power event on 2026-08-27 killed this run's first launch at step 80 of 604 with save_steps=100, so no checkpoint existed and it restarted from zero.", + "relaunch_reason: Anaheim site power event 2026-08-27T01:2x UTC, confirmed by a 1-minute uptime on the host's return. First launch reached step 80 of 604 with save_steps=100, so NO checkpoint existed and this run starts from zero. Configured identically apart from save_steps; NOT a resumption.", + "HOST MOVE (operational, not scientific): this run executes on pfi-gx10 (NVIDIA GB10, aarch64, sm_121, 121 GB unified) instead of ana-ml2. Paths are rehomed to local NVMe; NO hyperparameter, corpus, base, mask or recipe value differs from run-03c.json. Base model verified sha256-identical to ana-ml2's copy (both shards, 2026-09-03). Rationale: an Anaheim breaker trip already killed this run once and costs a 40-minute drive plus 13 hosts down, three of them SureFire client machines; the GX10 is a ~240 W appliance at NH3 that can take nothing else down. Measured cost: ~79.4 s/it vs ana-ml2's 10.8-15.8, so 604 steps is ~13.3 h instead of ~2.5 h.", + "LIBRARY DELTA (recorded, not assumed harmless): torch 2.13.0 -> 2.14.0+cu130, transformers 5.15.1 -> 5.16.1, triton 3.7.1 -> 3.8.0; peft 0.20.0 and accelerate 1.14.0 unchanged. Attention backend flex_attention on both sides. Tokenisation equivalence across the transformers delta is asserted by comparing this host's encoded corpus against ana-ml2's byte-for-byte before launch, not inferred." + ], + "unfittable": "drop", + "holdout_dir": "/home/infra-ops/erp-tune/datasets/holdout", + "save_steps": 50 +} diff --git a/servers/pfi-gx10/README.md b/servers/pfi-gx10/README.md index 4157793..7e52725 100644 --- a/servers/pfi-gx10/README.md +++ b/servers/pfi-gx10/README.md @@ -11,33 +11,25 @@ Grace-Blackwell desktop supercomputer. Registered 2026-09-01. | Kernel | 6.17.0-1031-nvidia | | Hostname | `pfi-gx10` (shipped with static `gx10-a745`, corrected) | -## ⚠ The address in `ssh-target` is TEMPORARY +## Network — racked, and single-path -As of 2026-09-01 this box is **on a desk, on Wi-Fi**, holding a DHCP lease at -`10.100.10.226` on `nh3-userland` (VLAN 10). Ethernet `enP7s7` has no carrier. +Racked 2026-09-03. `pfi-gx10.nh3.internal` → **10.100.50.60**, wired only on +`enP7s7`, VLAN 50 (`nh3-servers`), UniFi switch port 22. -It is going into the rack later. **Target settled (operator, 2026-09-01): -`nh3-servers`, VLAN 50, static `10.100.50.60`** — clear of `.40` nh3-docker, -`.42` nh3-extdev, `.50` nh3-nas, `.90` pbs-nh3, and below the `.150` DHCP pool -where fleet statics live. +**The address lives on the switch side, not the host** — a DHCP *reservation* +against the wired MAC `30:c5:99:3d:a7:45`, with the host left on DHCP. Operator +ruling: a reservation moves with the box, a netplan static goes stale the moment +it does. -Nothing was configured on the desk — an address that is about to be wrong is -worse than DHCP. The move is `playbooks/gx10-rack-network.yaml`. +⚠ **Wi-Fi is deliberately off and there is now exactly ONE path in.** If the +switch port or the reservation breaks, this is a rack visit. Correct end state +for a racked server, but it is a posture change from the desk setup. -**Nothing is needed from the operator beyond racking it.** The wired NIC has its -own MAC (`30:c5:99:3d:a7:45`, distinct from the Wi-Fi `50:bb:b5:a2:00:a8`), so -its post-move address is discoverable from the UDM without being told: +Full detail, including the order that made the move safe: +[`docs/runbooks/gx10-rack-network.md`](../../docs/runbooks/gx10-rack-network.md). -```bash -curl -sk "https://10.100.0.1/proxy/network/api/s/default/stat/sta" \ - -H "X-API-KEY: $(secret get unifi/pfi-udmse-api-key)" \ - | python3 -c "import json,sys;[print(c['ip'],c.get('sw_port')) for c in json.load(sys.stdin)['data'] if c['mac']=='30:c5:99:3d:a7:45']" -``` - -That also returns the switch port, which must be set to the `nh3-servers` -network or the box lands back on VLAN 10. - -Expect ~65–80 ms RTT until it is wired. That is Wi-Fi power-save, not a fault. +⚠ **The box mounts no NFS, on purpose.** Working data is copied to local NVMe — +see the training section below. ## Access @@ -62,6 +54,27 @@ trivially while the desktop is still running. The playbook will not stop GDM while someone holds a seat session. Override with `--var force_dm_stop=true`, or just let the rack-install reboot handle it. +## Training — run 3c is staged and ready + +The ERP-seat SFT LoRA (run 3c) that died on ana-ml2 at step 24 of 604 to an +Anaheim breaker trip is staged here, unchanged, and **not launched** — that call +is the operator's. + + ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh' + +~79.4 s/it measured on this geometry → 604 steps ≈ 13.3 h, peak 75.1 of +121.6 GiB. Slower than ana-ml2's ~2.5 h and still the right host: this is a +~240 W appliance at NH3 that cannot take a client's machines dark. + +Base model and the encoded corpus were both sha256-verified identical to +ana-ml2's, so the library delta (transformers 5.15.1 → 5.16.1, x86-64 → aarch64) +is measured to be inert rather than assumed harmless. Runbook: +[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md); canonical +config + launcher in [`scripts/erp-tune-gx10/`](../../scripts/erp-tune-gx10/). + +⚠ **Never `pkill -f erp_sft_harness` over SSH** — the pattern is in your own ssh +argv and you kill your shell with it. Kill by PID from `~/erp-tune/run-03c.pid`. + ## Relevance to Flash-Next `sm_121`, not `sm_120`. The SGLang fork evaluated for ana-ml2 (henge item 49)