feat(erp-tune): run 7 launched on pfi-gx10 (opening-split slot + mask union) — config, launcher, builder, runbook; erp-seat: pin vLLM nightly 311b3513 (forced tool_choice 1/9 -> 6/9, round trip clean), README on the gemma4-parser design limit
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
# pfi-gx10 — ERP-seat SFT run 7 (opening-split slot)
|
||||
|
||||
Launched 2026-09-08 23:06:11 PT on pfi-gx10, pid 599489, under the operator's direct grant
|
||||
`operator-2026-09-08-rnd-run7` (in-session `/goal: launch training run 7 on gx10`, 23:03 PT).
|
||||
Run 6's CSAM soft trip was adjudicated GO by the operator at 22:45 PT (verbatim in the config's
|
||||
`substitute_controls`); run06-gate closed as TRANSFERRED.
|
||||
|
||||
## What run 7 is
|
||||
|
||||
Run 6 + **one variable**: an opening-split slot and its companion loss mask (brokkr-smithy
|
||||
`recipe-erp-seat-sft-r7.json`, `SFT-RECIPE-run7.md`). 293 rows (bluemoon 109 / cwm 179 / c2-logs
|
||||
5): the corpus's own first exchanges re-seated into the serving shape — scene in the card, one
|
||||
user line, the ≥250-word first bot turn as the only loss turn. The mask (224 entries) removes loss
|
||||
from those conversations' ORIGINAL first bot turns in the held dialogue roots: loss moved, not
|
||||
doubled. Chosen by brokkr's cue-length probe on the run-6 seat (5-word cue → 54/62 words median;
|
||||
221-word → 292). Base, seven held roots, kvasir cut, longdep slot, hyperparameters, stock
|
||||
tokenizer + template all unchanged from run 6.
|
||||
|
||||
## Build (`build_r7.py` on the box)
|
||||
|
||||
- `recipe-r7/survivors-r7.jsonl` = survivors-r5 (8,212) + 293 opening-split ids = 8,505.
|
||||
- `recipe-r7/lossmask-r7.jsonl` = lossmask-r3 (5,883) ∪ mask-original-first-turn (224), 1 duplicate
|
||||
→ 6,106 entries, sha `ad77f452…`.
|
||||
- `datasets/derived/opening-split/{clean-v1,holdout-v1}` relayed NAS → nh3-dev → box (no NFS on
|
||||
the GX10), sha-verified: root jsonl `85a75718…`, mask `dda151fa…`, holdout `7224e049…`.
|
||||
- `run-07-gx10.json` = run-06 config with recipe/survivors/mask/output/override swapped and the
|
||||
substitute_controls rewritten (single variable, holdout opening-split/holdout-v1 16 rows, the
|
||||
CSAM adjudication verbatim).
|
||||
- ⚠ First launch died in preflight: the recipe JSON had not landed (a zsh quoting error aborted
|
||||
the staging chain silently). `run-07.log.attempt1-missing-recipe` kept.
|
||||
|
||||
## Free check — passed with two explained deltas
|
||||
|
||||
[encode] 8,488 samples -> 8,661 records (173 chunked, 30 single+truncated, 59 lost turns);
|
||||
ctx 19,114,072 tok, loss 9,998,300 tok
|
||||
[mix] kvasir .3348 fireball .3023 c2-logs .1143 bluemoon .0964 cwm .0942
|
||||
govreport .0289 opening-split .0235 qmsum .0055
|
||||
|
||||
- Held roots encode identically to run 6 (same chunk/truncate/lost-turn counts).
|
||||
- Slot: 293/293 `fit_whole`, 0 chunked, 0 truncated (window_count 1 everywhere) — ctx 522,951 /
|
||||
loss 235,347 vs the recipe's 525,002 / 233,589 (different tokenizer instrument; the harness
|
||||
`[mix]` supersedes).
|
||||
- Records 8,661 = 8,370 + 293 − 2: **cwm-2602 and cwm-1826 dropped as unfittable** — lossmask-r3
|
||||
already masked every bot turn but the first, the companion mask masked the first, so no loss
|
||||
turn remained. Their loss now lives in their opening-split rows. Expected under "loss moved".
|
||||
- Mask match: 6,106 turns across 567 samples (run 6: 5,883 / 426) → all 224 companion entries
|
||||
landed (+1 duplicate with r3).
|
||||
- Sampler padding 17.1% (run 6: 0.0%) — the short slot rows pair badly; a throughput cost, not
|
||||
a correctness one. 542 optimizer steps; first step 155 s (JIT warm-up), expect ~80 s/it after.
|
||||
|
||||
## Launch / watch / stop
|
||||
|
||||
ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-07.log | tail"
|
||||
ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-07.pid)' # by PID — never pkill -f over ssh
|
||||
|
||||
Gate choreography after the adapter lands: brokkr-smithy `SFT-RECIPE-run7-SCOPE.md §5` (cells
|
||||
LENGTH-FIXED / LENGTH-MOVED / BOUGHT-T4 / FLAT); serve names `erp-seat-base-ara` then `erp-tune-v7`.
|
||||
@@ -14,6 +14,9 @@ The harness itself (`eitri-smithy`) is not vendored here; it lives on the box at
|
||||
| `run-06-gx10.json` | `/home/infra-ops/erp-tune/run-06-gx10.json` | 6 |
|
||||
| `launch-run-06.sh` | `/home/infra-ops/erp-tune/launch-run-06.sh` | 6 |
|
||||
| `pull-verify-jenerallee78.sh` | `/home/infra-ops/erp-tune/pull-verify-jenerallee78.sh` | 6 (base pull + byte verify) |
|
||||
| `run-07-gx10.json` | `/home/infra-ops/erp-tune/run-07-gx10.json` | 7 |
|
||||
| `launch-run-07.sh` | `/home/infra-ops/erp-tune/launch-run-07.sh` | 7 |
|
||||
| `build_r7.py` | `/home/infra-ops/erp-tune/build_r7.py` | 7 (survivors-r7 + lossmask-r7 union + config) |
|
||||
| `base-pin-jenerallee78-shards.txt` | `/home/infra-ops/erp-tune/base-pin-jenerallee78-shards.txt` | 6 (32 shard pins, from brokkr-smithy `base-pin-jenerallee78.json`) |
|
||||
|
||||
Runbooks: [`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md),
|
||||
@@ -37,6 +40,14 @@ survivors (`survivors-r5.jsonl`), mask, template and hyperparameters unchanged.
|
||||
The landed dir carries the STOCK tokenizer set (the repo's `tokenizer.json`
|
||||
bakes in a 256-token truncation); repo originals kept beside as `*.repo`.
|
||||
|
||||
**Run 7** — run 6 + an OPENING-SPLIT slot (293 rows, `opening-split/clean-v1`) with its companion
|
||||
loss mask (224 entries, UNION with lossmask-r3 → `lossmask-r7`, 1 duplicate). `survivors-r7` =
|
||||
survivors-r5 + the 293 slot ids (8,505). Launched 2026-09-08 23:06 PT under
|
||||
`operator-2026-09-08-rnd-run7`. Free check: held roots encode identical to run 6 (same 173
|
||||
chunked / 30 truncated / 59 lost turns), slot 293/293 fit whole, two cwm conversations
|
||||
(cwm-2602, cwm-1826) dropped as unfittable because the mask left them no loss turn — their
|
||||
loss now lives in their opening-split rows. 542 steps, 17.1% padding (run 6: 0%).
|
||||
|
||||
> **Run 4 is not vendored here.** It ran on the box (config `run-04-gx10.json`,
|
||||
> gated STILL-COUPLED) but its canonical copies were never committed; run 5's
|
||||
> `build_r5_survivors.py` derives from `survivors-r4.jsonl` on the box, so run 4
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build run-7 inputs on pfi-gx10: survivors-r7 (r5 + opening-split), lossmask-r7 (r3 UNION companion mask), run-07-gx10.json."""
|
||||
import json, hashlib, os
|
||||
os.chdir(os.path.expanduser("~/erp-tune"))
|
||||
surv=[l for l in open("recipe-r5/survivors-r5.jsonl") if l.strip()]
|
||||
os_rows=[json.loads(l) for l in open("datasets/derived/opening-split/clean-v1/opening-split-clean-v1.jsonl") if l.strip()]
|
||||
ids=[r["id"] for r in os_rows]; assert len(ids)==293 and len(set(ids))==293, len(ids)
|
||||
with open("recipe-r7/survivors-r7.jsonl","w") as f:
|
||||
f.writelines(surv); f.writelines(json.dumps({"dataset_id":"opening-split","id":i},separators=(",",":"))+"\n" for i in ids)
|
||||
key=lambda d:(d["dataset_id"],d["id"],d["turn_index"],d["content_sha_prefix"])
|
||||
m3=[json.loads(l) for l in open("recipe-r3/lossmask-r3.jsonl") if l.strip()]
|
||||
mo=[json.loads(l) for l in open("datasets/derived/opening-split/clean-v1/mask-original-first-turn.jsonl") if l.strip()]
|
||||
seen=set(); out=[]
|
||||
for d in m3+mo:
|
||||
k=key(d)
|
||||
if k in seen: continue
|
||||
seen.add(k); out.append(d)
|
||||
dup=len(m3)+len(mo)-len(out)
|
||||
with open("recipe-r7/lossmask-r7.jsonl","w") as f:
|
||||
f.writelines(json.dumps(d,sort_keys=True)+"\n" for d in out)
|
||||
def sha(p): return hashlib.sha256(open(p,"rb").read()).hexdigest()[:16]
|
||||
print(f"survivors-r7: {len(surv)}+{len(ids)} = {len(surv)+len(ids)} rows, sha {sha('recipe-r7/survivors-r7.jsonl')}")
|
||||
print(f"lossmask-r7: {len(m3)} + {len(mo)} - {dup} dup = {len(out)} entries, sha {sha('recipe-r7/lossmask-r7.jsonl')}")
|
||||
c=json.load(open("run-06-gx10.json"))
|
||||
c["output_dir"]="/home/infra-ops/erp-tune/run-07"
|
||||
c["recipe"]="/home/infra-ops/erp-tune/recipe-r7/recipe-erp-seat-sft-r7.json"
|
||||
c["survivors"]="/home/infra-ops/erp-tune/recipe-r7/survivors-r7.jsonl"
|
||||
c["impersonation_mask_path"]="/home/infra-ops/erp-tune/recipe-r7/lossmask-r7.jsonl"
|
||||
c["training_eligibility_override"]="operator-2026-09-08-rnd-run7"
|
||||
c["base_model_revision"]=c["base_model_revision"].replace("THE SINGLE VARIABLE vs run 5: base only. Run-5 recipe, survivors, mask, template, hyperparameters all UNCHANGED.","BASE HELD from run 6 (index 33c59654). RUN 7 SINGLE VARIABLE: an OPENING-SPLIT slot ADDED (293 rows, opening-split/clean-v1, root 3ab41c08) + its companion loss mask (224 entries, UNION with lossmask-r3 -> lossmask-r7). Everything else run 6.")
|
||||
sc=[s for s in c["substitute_controls"] if not s.startswith(("SINGLE VARIABLE","FREE CHECK","HOST:","SURVIVORS:"))]
|
||||
sc+=[
|
||||
"SINGLE VARIABLE vs run 6: an OPENING-SPLIT slot ADDED with its companion loss mask (recipe-erp-seat-sft-r7, brokkr-smithy). 293 rows (bluemoon 109 / cwm 179 / c2-logs 5): the corpus's own first exchanges re-seated into the serving shape -- scene in the card, one line from the user, the >=250-word first bot turn as the only loss turn. Mask = lossmask-r3 UNION mask-original-first-turn (224 entries: the split conversations' ORIGINAL first bot turns -- loss MOVED to the turn-1 position, not doubled). Chosen by the cue-length probe on the run-6 tuned seat (5-word cue -> 54/62 words; 221-word -> 292). Base, seven held roots + shas, kvasir cut, longdep slot, lr 2e-04, max_seq_len 16384, rank/alpha/dropout, cosine/warmup/wd, batch 2 x accum 8, seed, stock tokenizer + template ae53464b ALL UNCHANGED from run 6.",
|
||||
"SURVIVORS: survivors-r7.jsonl = survivors-r5.jsonl (8,212, held byte-identical) + the 293 opening-split clean-v1 row ids = 8,505.",
|
||||
"FREE CHECK (brokkr, SFT-RECIPE-run7.md section 5): held targets must reproduce run 6's 8,370 records exactly (minus loss on the masked turns); the slot adds 293 records / ctx 525,002 / loss 233,589; window_count MUST be 1 on every slot row (max row 4,539 tokens) -- any chunked/truncated slot row is a BUILD DEFECT, kill before [train]. Report the mask-union match count and the realized [mix] block.",
|
||||
"pre-training holdout, opening-split/holdout-v1 (16 rows: bluemoon 6 / cwm 9 / c2-logs 1, sha256-ranked 5%, never_trained_on; staged at datasets/derived/opening-split/holdout-v1/)",
|
||||
"CSAM LINE: run 6's level=review SOFT trip (one soft term, minor_hard=0) was adjudicated GO by the operator in-session 2026-09-08 22:45 PT -- verbatim: in the vernacular, baby is about the most common pet name you can get, ESPECIALLY during sex. I am going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is not one of them. -- relayed to brokkr-smithy-dev; run06-gate CLOSED as TRANSFERRED. Run 7 = this recipe on that base under that adjudication.",
|
||||
"HOST: pfi-gx10 (GB10, aarch64, sm_121, 121 GB unified). Base unchanged from run 6 (sha256-verified); opening-split root + mask + holdout sha256-verified after the NAS -> box relay (85a75718 / dda151fa / 7224e049); harness eitri-smithy 0a6bd2e; corpus COPIED, box mounts no NFS. Grant: operator directive to infra-ops in-session 2026-09-08 23:03 PT (/goal: launch training run 7 on gx10)."
|
||||
]
|
||||
c["substitute_controls"]=sc
|
||||
json.dump(c,open("run-07-gx10.json","w"),indent=2); open("run-07-gx10.json","a").write("\n")
|
||||
print("config written:",c["recipe"].split("/")[-1],c["survivors"].split("/")[-1],c["impersonation_mask_path"].split("/")[-1],c["training_eligibility_override"])
|
||||
Executable
+72
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch ERP-seat SFT run 7 on pfi-gx10 (NVIDIA GB10, aarch64, sm_121).
|
||||
#
|
||||
# Run this ON pfi-gx10 as infra-ops. It detaches the job from the invoking
|
||||
# shell and logs to the box, so a reaped SSH session cannot take the run with
|
||||
# it -- the failure mode that lost the first probe launch on 2026-09-01.
|
||||
#
|
||||
# Run 6 = run 5 recipe UNCHANGED on the jenerallee78 ARA-abliterated base (the single variable).
|
||||
# 8,505 survivors -> ~543 optimizer steps; held targets must match run 6, slot adds 293 records.
|
||||
# Checkpoints every 50 steps.
|
||||
set -euo pipefail
|
||||
|
||||
ROOT=/home/infra-ops/erp-tune
|
||||
HARNESS=$ROOT/eitri-smithy
|
||||
VENV=/home/infra-ops/ml/.venv/bin/python
|
||||
CONFIG=$ROOT/run-07-gx10.json
|
||||
LOG=$ROOT/run-07.log
|
||||
|
||||
# --- Preconditions, asserted rather than assumed -----------------------------
|
||||
|
||||
# A stuck orphan holding unified memory while PyTorch reports zero allocated
|
||||
# already doomed three relaunches on this box and got blamed on the new run
|
||||
# each time. Assert the GPU is clear.
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "REFUSING: GPU is not clear -- compute apps still resident:" >&2
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Deliberately NOT `pgrep -f erp_sft_harness`: run this over ssh and the
|
||||
# pattern appears in the invoking shell's own argv, so the guard matches
|
||||
# itself and refuses every launch. The pidfile is exact and cannot self-match;
|
||||
# the GPU assertion above catches an orphan under any name.
|
||||
if [ -f "$ROOT/run-07.pid" ] && kill -0 "$(cat "$ROOT/run-07.pid")" 2>/dev/null; then
|
||||
echo "REFUSING: run-07.pid names a live process $(cat "$ROOT/run-07.pid"):" >&2
|
||||
ps -p "$(cat "$ROOT/run-07.pid")" -o pid,etime,cmd >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -e "$LOG" ]; then
|
||||
echo "REFUSING: $LOG exists. Move it aside first so two runs cannot share a log." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for p in "$HARNESS/erp_sft_harness/__main__.py" "$VENV" "$CONFIG"; do
|
||||
[ -e "$p" ] || { echo "REFUSING: missing $p" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# Free space for checkpoints, with headroom.
|
||||
avail=$(df --output=avail -BG "$ROOT" | tail -1 | tr -dc '0-9')
|
||||
if [ "$avail" -lt 40 ]; then
|
||||
echo "REFUSING: only ${avail}G free under $ROOT; want >=40G for checkpoints." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Launch ------------------------------------------------------------------
|
||||
|
||||
cd "$HARNESS"
|
||||
{
|
||||
echo "# launched $(date -Is) on $(hostname) by ${USER}"
|
||||
echo "# harness $(git rev-parse --short HEAD) config $CONFIG"
|
||||
} > "$LOG"
|
||||
|
||||
setsid nohup "$VENV" -m erp_sft_harness --config "$CONFIG" >> "$LOG" 2>&1 < /dev/null &
|
||||
pid=$!
|
||||
echo "$pid" > "$ROOT/run-07.pid"
|
||||
|
||||
echo "launched pid $pid -> $LOG"
|
||||
echo
|
||||
echo "watch: tail -f $LOG | tr '\\r' '\\n'"
|
||||
echo "stop: kill \$(cat $ROOT/run-07.pid) # by PID -- never pkill -f over ssh"
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"output_dir": "/home/infra-ops/erp-tune/run-07",
|
||||
"roots_dir": "/home/infra-ops/erp-tune/datasets/derived",
|
||||
"base_model_path": "/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a",
|
||||
"base_model_revision": "jenerallee78/gemma-4-26B-A4B-it-ara-abliterated @ 0631379a3d859e0059bc8d9b21ab5b654dfc272c (ARA abliteration of stock google/gemma-4-26B-A4B-it; 32 bf16 root shards sha256-verified against brokkr-smithy base-pin-jenerallee78.json; index sha256 33c59654e658a30fa29cdc87ccd6a752bfa0bb3e32cd56f95ff1eb82075e593a). BASE HELD from run 6 (index 33c59654). RUN 7 SINGLE VARIABLE: an OPENING-SPLIT slot ADDED (293 rows, opening-split/clean-v1, root 3ab41c08) + its companion loss mask (224 entries, UNION with lossmask-r3 -> lossmask-r7). Everything else run 6. Tokenizer set = STOCK (tokenizer.json cc8d3a0c / tokenizer_config.json 9f4fec4b / chat_template.jinja ae53464b) copied over the repo's, whose shipped tokenizer.json carries a baked-in max_length=256 truncation; repo originals kept beside as *.repo. Runs 3/3c/4/5 were settled 2026-09-08 as STOCK base (index 907826a6), so this is the line's first abliterated base.",
|
||||
"recipe": "/home/infra-ops/erp-tune/recipe-r7/recipe-erp-seat-sft-r7.json",
|
||||
"survivors": "/home/infra-ops/erp-tune/recipe-r7/survivors-r7.jsonl",
|
||||
"chat_template_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16/chat_template.jinja",
|
||||
"impersonation_mask_path": "/home/infra-ops/erp-tune/recipe-r7/lossmask-r7.jsonl",
|
||||
"lora_rank": 64,
|
||||
"lora_alpha": 128,
|
||||
"lora_dropout": 0.0,
|
||||
"max_seq_len": 16384,
|
||||
"epochs": 1,
|
||||
"seed": 20260824,
|
||||
"per_device_batch_size": 2,
|
||||
"gradient_accumulation_steps": 8,
|
||||
"learning_rate": 0.0002,
|
||||
"warmup_ratio": 0.1,
|
||||
"lr_scheduler_type": "cosine",
|
||||
"weight_decay": 0.01,
|
||||
"load_in_4bit": false,
|
||||
"gradient_checkpointing": true,
|
||||
"loss_chunk_tokens": 1024,
|
||||
"training_eligibility_override": "operator-2026-09-08-rnd-run7",
|
||||
"overridden_blockers": [
|
||||
"contamination-scan-not-implemented",
|
||||
"stage-2-csam-detector-inert"
|
||||
],
|
||||
"substitute_controls": [
|
||||
"pre-training holdout, run-1 (8,404 samples, work/card/session split)",
|
||||
"pre-training holdout, govreport/holdout-v1 (416 reports, sha256-ranked, never_trained_on)",
|
||||
"pre-training holdout, qmsum/holdout-v1 (5 transcripts, sha256-ranked, never_trained_on)",
|
||||
"stage-A lexical quarantine, RP (829 records held unread)",
|
||||
"stage-A lexical quarantine, run-5 slot (133 records held unread, /mnt/smithy/datasets/quarantine/r47-run5-longdep-screen/)",
|
||||
"SCROLLS-membership disclosure on both slot sources (avoidance, NOT a scan): govreport + qmsum are SCROLLS/ZeroSCROLLS members, in no hoard/default-benchmarks.yaml entry and used by no R47 instrument",
|
||||
"kvasir is HELD, not re-cut: the 1,613 kvasir survivors are reused verbatim from survivors-r4.jsonl (which cut run-3's seed-20260824 prefix at 3,347,622 ctx). survivors-r5.jsonl = survivors-r4 minus airoboros plus the govreport + qmsum roots whole; sha256 a25169a6258cd4abb0cb494a176a921c0e98eb73d65c53d033b6ee18293a43ae.",
|
||||
"window_count belt-and-suspenders (SFT-RECIPE-run5-SCOPE.md 7.1): every govreport + qmsum row renders <= 14,000 tokens (max 9,385 / 13,700) and the harness never packs across samples, so window_count MUST be 1 on every slot row; a chunked_into_2 or single_window_truncated on either new root in truncation-report.json is a BUILD DEFECT and the run is killed before training.",
|
||||
"SINGLE VARIABLE vs run 6: an OPENING-SPLIT slot ADDED with its companion loss mask (recipe-erp-seat-sft-r7, brokkr-smithy). 293 rows (bluemoon 109 / cwm 179 / c2-logs 5): the corpus's own first exchanges re-seated into the serving shape -- scene in the card, one line from the user, the >=250-word first bot turn as the only loss turn. Mask = lossmask-r3 UNION mask-original-first-turn (224 entries: the split conversations' ORIGINAL first bot turns -- loss MOVED to the turn-1 position, not doubled). Chosen by the cue-length probe on the run-6 tuned seat (5-word cue -> 54/62 words; 221-word -> 292). Base, seven held roots + shas, kvasir cut, longdep slot, lr 2e-04, max_seq_len 16384, rank/alpha/dropout, cosine/warmup/wd, batch 2 x accum 8, seed, stock tokenizer + template ae53464b ALL UNCHANGED from run 6.",
|
||||
"SURVIVORS: survivors-r7.jsonl = survivors-r5.jsonl (8,212, held byte-identical) + the 293 opening-split clean-v1 row ids = 8,505.",
|
||||
"FREE CHECK (brokkr, SFT-RECIPE-run7.md section 5): held targets must reproduce run 6's 8,370 records exactly (minus loss on the masked turns); the slot adds 293 records / ctx 525,002 / loss 233,589; window_count MUST be 1 on every slot row (max row 4,539 tokens) -- any chunked/truncated slot row is a BUILD DEFECT, kill before [train]. Report the mask-union match count and the realized [mix] block.",
|
||||
"pre-training holdout, opening-split/holdout-v1 (16 rows: bluemoon 6 / cwm 9 / c2-logs 1, sha256-ranked 5%, never_trained_on; staged at datasets/derived/opening-split/holdout-v1/)",
|
||||
"CSAM LINE: run 6's level=review SOFT trip (one soft term, minor_hard=0) was adjudicated GO by the operator in-session 2026-09-08 22:45 PT -- verbatim: in the vernacular, baby is about the most common pet name you can get, ESPECIALLY during sex. I am going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is not one of them. -- relayed to brokkr-smithy-dev; run06-gate CLOSED as TRANSFERRED. Run 7 = this recipe on that base under that adjudication.",
|
||||
"HOST: pfi-gx10 (GB10, aarch64, sm_121, 121 GB unified). Base unchanged from run 6 (sha256-verified); opening-split root + mask + holdout sha256-verified after the NAS -> box relay (85a75718 / dda151fa / 7224e049); harness eitri-smithy 0a6bd2e; corpus COPIED, box mounts no NFS. Grant: operator directive to infra-ops in-session 2026-09-08 23:03 PT (/goal: launch training run 7 on gx10)."
|
||||
],
|
||||
"unfittable": "drop",
|
||||
"holdout_dir": "/home/infra-ops/erp-tune/datasets/holdout",
|
||||
"save_steps": 50
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
# erp-seat — ana-ml2 GPU1. Real .env lives on the host at /opt/docker/compose/erp-seat/.env.
|
||||
ERP_IMAGE=vllm/vllm-openai:v0.26.0
|
||||
ERP_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0
|
||||
ERP_MODEL=/tank/aimodels/erp-tune-v6-nvfp4a16
|
||||
ERP_SERVED_NAME=erp-tune-v6-nvfp4a16
|
||||
ERP_CHAT_TEMPLATE=/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja
|
||||
|
||||
@@ -19,6 +19,18 @@ checkpoint so the GX10 is free to train the next run. First occupant: **run 6**
|
||||
the model answers in prose (3/3). The rest of the matrix (auto / required / named / parallel /
|
||||
nested schema / empty `tools: []` / streaming / tool-result round trip) was green before and
|
||||
after. `stacks/gemma4-charrp` has the same exposure and does NOT carry the flag yet.
|
||||
- **Forced tool_choice (named / `required`) is prompt-driven on EVERY Gemma-4 seat, not
|
||||
grammar-enforced — by vLLM design.** `vllm/tool_parsers/gemma4_engine_tool_parser.py` sets
|
||||
`supports_required_and_named = False` and its `adjust_request` deliberately skips the
|
||||
structured-output JSON for required/named so the model can emit its native
|
||||
`<|tool_call>call:…` syntax. A tune that weakened that syntax (this ERP tune) therefore
|
||||
honours forced calls only sometimes. Measured 2026-09-08, 3 conversations × 3 turns, real
|
||||
system prompt: v0.26.0 **1/9**; nightly `311b3513` (v0.27.2rc1, the `gen` seat's image) **6/9**
|
||||
and the tool-result round trip stays clean 3/3 — so the seat runs the nightly. tts-dev measured
|
||||
0/18 on v0.26.0 with `gen` 18/18 as the positive control, and **`response_format:
|
||||
json_schema` (guided decoding) 18/18 on this seat** — that is the deterministic path for a
|
||||
forced call; `tool_choice: auto` works normally. A parser plugin that re-enables guided JSON
|
||||
would also need JSON extraction in the engine-parser path; not attempted.
|
||||
- **GPU1 is shared** — check real usage (`nvidia-smi --query-compute-apps=pid,used_memory`) before
|
||||
raising `ERP_GPU_MEM_UTIL`; the flag sizes KV, not CUDA context.
|
||||
- **Rollback / next run:** point `ERP_MODEL` + `ERP_SERVED_NAME` at the next quant dir, keep the
|
||||
|
||||
@@ -12,7 +12,7 @@ name: erp-seat
|
||||
|
||||
services:
|
||||
vllm-erp-seat:
|
||||
image: ${ERP_IMAGE:-vllm/vllm-openai:v0.26.0}
|
||||
image: ${ERP_IMAGE:-vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0}
|
||||
container_name: ${ERP_CONTAINER:-vllm-erp-seat}
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
@@ -31,6 +31,12 @@ services:
|
||||
- --tool-call-parser
|
||||
- gemma4
|
||||
- --enable-auto-tool-choice
|
||||
# --reasoning-parser gemma4 + the enable_thinking pin are BOTH required (measured 2026-09-08):
|
||||
# without them the post-tool-response turn leaks a literal "<|channel>thought\n<channel|>"
|
||||
# prefix into content (3/3), exactly as stacks/gemma4-charrp/README.md warns. They do NOT
|
||||
# fix forced (named) tool_choice — that failure is independent of the reasoning parser
|
||||
# (measured 1/9 with it, 3/9 without): the model free-writes Gemma native call syntax
|
||||
# instead of grammar-constrained JSON. See README for the forced-tool_choice status.
|
||||
- --reasoning-parser
|
||||
- gemma4
|
||||
- --default-chat-template-kwargs
|
||||
|
||||
Reference in New Issue
Block a user