Files
esh-pfi-infrastructure/docs/runbooks/gx10-run-06.md
T
vh 3fec668bf2 feat(erp-tune): run 6 on pfi-gx10 — jenerallee78 ARA-abliterated base (index 33c59654) pulled + byte-verified, run-5 recipe byte-held, launched under operator-2026-09-08-rnd-run6
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
  revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
  set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
  a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
  base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
  reproduces run 5 exactly), hf download --include gotcha, gate naming
  (erp-seat-base-ara / erp-tune-v6)
2026-09-08 04:24:38 -07:00

4.5 KiB
Raw Blame History

pfi-gx10 — ERP-seat SFT run 6 (abliterated base)

Launched 2026-09-08 04:17 PDT (11:17:43Z) on pfi-gx10, pid 4100375. Grant: the operator's direct in-session directive to infra-ops — "unload the gx10 and commence training on the gx10. window is open now." — recorded on both sides as operator-2026-09-08-rnd-run6 (brokkr-smithy TRAINING-ELIGIBILITY-OVERRIDE-run6.md).

What run 6 is

Run 5's recipe byte-held on a different base. The single variable is the base: jenerallee78/gemma-4-26B-A4B-it-ara-abliterated @ 0631379a3d859e0059bc8d9b21ab5b654dfc272c (ARA 2-pass abliteration of stock google/gemma-4-26B-A4B-it, layers 1324, o_proj + down_proj). Runs 3/3c/4/5 were settled from bytes on 2026-09-08 as having trained on stock (index sha 907826a6…) despite the -heretic name; this is the line's first genuinely abliterated base. Pick and pins: brokkr-smithy research/R47-premium-corpus-gate/ABLITERATED-BASE-HUNT-2026-09-08.md + base-pin-jenerallee78.json; recipe recipe-erp-seat-sft-r6.json (sha 64995554…, brokkr-smithy 4dd7590).

Base pull + verify (what pull-verify-jenerallee78.sh did)

Landed at /home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a — named for the bytes, never for the intent (the lesson of -heretic-bf16).

  • Root shards + small files only, revision-pinned; the two root GGUFs, mmproj and mlx-4bit/ were not pulled. ~143 MB/s, 32 shards in ~7 min.
  • Registry cross-check from nh3-dev first: HF tree API at the pinned revision, all 32 LFS oids + sizes == pins.
  • After landing: every shard's sha256 AND size == pin (32/32); index weight_map set-equal to stock's 1013 names; total_size 51,611,872,412 == stock; config.json Gemma4ForConditionalGeneration / bfloat16.
  • Base identity (index sha256): 33c59654e658a30fa29cdc87ccd6a752bfa0bb3e32cd56f95ff1eb82075e593a.
  • Tokenizer hazard (brokkr, measured): the repo's tokenizer.json ships with "truncation": {"max_length": 256} baked in — vocab identical to stock, but loaded as shipped it silently cuts every text past 256 tokens and the window_count guard would not notice. The STOCK three were copied over it (repo originals kept as *.repo), re-hashed in the landed dir: tokenizer.json cc8d3a0c… / tokenizer_config.json 9f4fec4b… / chat_template.jinja ae53464b… (the July stock template runs 35 used; the repo's is the older April one, 2dfbfc7d…).
  • hf download gotcha: multiple patterns after one --include are parsed as explicit FILENAMES and the include is silently ignored ("Fetching 8 files"). Use one --include per pattern. Attempt 1 landed 62 MB and failed verify 32/32; attempt 2 is the recorded one.

Config

run-06-gx10.json = run-05-gx10.json with base_model_path → the landed dir, reciperecipe-r6/, survivorsrecipe-r5/survivors-r5.jsonl verbatim (r6 ships no survivor list; same bytes, sha a25169a6…), chat_template_path → the stock file (same path as run 5), output_dirrun-06, override → operator-2026-09-08-rnd-run6. Hyperparameters, mask (lossmask-r3), seed all unchanged.

Free check — passed exactly

Same corpus + same tokenizer + same template ⇒ the encode must reproduce run 5: [encode] 8,197 samples -> 8,370 records; ctx 18,598,779 tok, loss 9,935,076 tok, [mix] shares identical to four places, govreport 496/496 and qmsum 97/97 fit_whole, 0 chunked / 0 truncated. Any difference = wrong tokenizer/template → kill before [train]. Encode-cache filename differs by design (base_model_path is in the key).

Launch / watch / stop

ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-06.sh'
ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-06.log | tail"
ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-06.pid)'   # by PID — never pkill -f over ssh

The erp-tune-v5 seat (vllm-run05.pid) was stopped to clear the GPU; the LiteLLM trial alias is dark until the next serve.

After the adapter lands — gate choreography (brokkr-smithy-dev, cc channel)

Preregistered before any data: cells TRANSFERRED / COUPLED-HERE / FLAT on this base's own floors, never stock's. Naming is load-bearing for Brokkr's pipelines: serve the abliterated base as erp-seat-base-ara (erp-seat-base means stock), the merged arm as erp-tune-v6. Same stack/flags as run 5 (bf16, max-model-len 8192, max-num-seqs 8, gpu-util 0.60, gemma4 tool parser, template ae53464b). Base floors → lock → swap cue → tuned arm. Hands-off through both.