vh
dae77ee118
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
48 lines
4.5 KiB
JSON
48 lines
4.5 KiB
JSON
{
|
|
"output_dir": "/home/infra-ops/erp-tune/run-03c",
|
|
"roots_dir": "/home/infra-ops/erp-tune/datasets/derived",
|
|
"base_model_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16",
|
|
"base_model_revision": "google/gemma-4-26B-A4B-it (stock instruct, upstream) -- UNCHANGED from runs 2 and 3; run 3c varies the LEARNING RATE alone",
|
|
"recipe": "/home/infra-ops/erp-tune/recipe-r3/recipe-erp-seat-sft-r3.json",
|
|
"survivors": "/home/infra-ops/erp-tune/recipe-r3/survivors-r3.jsonl",
|
|
"chat_template_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16/chat_template.jinja",
|
|
"impersonation_mask_path": "/home/infra-ops/erp-tune/recipe-r3/lossmask-r3.jsonl",
|
|
"lora_rank": 64,
|
|
"lora_alpha": 128,
|
|
"lora_dropout": 0.0,
|
|
"max_seq_len": 16384,
|
|
"epochs": 1,
|
|
"seed": 20260824,
|
|
"per_device_batch_size": 2,
|
|
"gradient_accumulation_steps": 8,
|
|
"learning_rate": 1e-05,
|
|
"warmup_ratio": 0.1,
|
|
"lr_scheduler_type": "cosine",
|
|
"weight_decay": 0.01,
|
|
"load_in_4bit": false,
|
|
"gradient_checkpointing": true,
|
|
"loss_chunk_tokens": 1024,
|
|
"training_eligibility_override": "operator-2026-08-26-run3-go",
|
|
"overridden_blockers": [
|
|
"contamination-scan-not-implemented",
|
|
"stage-2-csam-detector-inert"
|
|
],
|
|
"substitute_controls": [
|
|
"pre-training holdout (8,002 samples, work/card/session split)",
|
|
"stage-A lexical quarantine, RP only (829 records held)",
|
|
"F1 pippa root excision (target-list change, recipe erp-seat-sft-r3)",
|
|
"F2/F3/F4 low-quality loss-mask, 5,867 turns (length floor 250w, register cap 20 asterisks/1kw, placeholder leak) -- MASKED not deleted",
|
|
"impersonation loss-mask unioned into the same file, 16 turns not covered by F2/F3/F4; 5,883 masked turns total, sha-asserted",
|
|
"kvasir subset to the recipe's stated 38.0% context share (3,498 of 12,003, seeded shuffle at run seed) -- the recipe names the share but stages no selection; see recipe-r3/RUN-03-BUILD-NOTE.md",
|
|
"DEVIATION RECORDED: recipe token accounting is not the harness tokenizer; bluemoon is NOT the largest loss contributor as trained (32.9% vs c2-logs 35.6%) -- see RUN-03-BUILD-NOTE.md section 4",
|
|
"SINGLE VARIABLE vs run 3: learning_rate 0.0002 -> 1e-05 (20x cut). Corpus BYTE-IDENTICAL (survivors-r3.jsonl, lossmask-r3.jsonl, same shas); base, max_seq_len 16384, rank 64, alpha 128, dropout 0.0, cosine, warmup 0.1, wd 0.01, batch 2, 1 epoch and target_modules ALL UNCHANGED. Rationale: the zerofata MeroMero v1 card, on this exact base, warns \"Google left the instruct model at the razor's edge of overfitting; finetune it at all and it feels like it'll rapidly lose intelligence\" -- and MeroMero v1 trains at 1e-5. Run 3 used 20x that.",
|
|
"OPERATIONAL DEVIATION (not scientific): save_steps 100 -> 50. Checkpoint cadence only; it changes nothing about the optimisation, the corpus or the single-variable lr claim above. Recorded as a separate entry so the two deviations cannot be confused. Rationale: an Anaheim site power event on 2026-08-27 killed this run's first launch at step 80 of 604 with save_steps=100, so no checkpoint existed and it restarted from zero.",
|
|
"relaunch_reason: Anaheim site power event 2026-08-27T01:2x UTC, confirmed by a 1-minute uptime on the host's return. First launch reached step 80 of 604 with save_steps=100, so NO checkpoint existed and this run starts from zero. Configured identically apart from save_steps; NOT a resumption.",
|
|
"HOST MOVE (operational, not scientific): this run executes on pfi-gx10 (NVIDIA GB10, aarch64, sm_121, 121 GB unified) instead of ana-ml2. Paths are rehomed to local NVMe; NO hyperparameter, corpus, base, mask or recipe value differs from run-03c.json. Base model verified sha256-identical to ana-ml2's copy (both shards, 2026-09-03). Rationale: an Anaheim breaker trip already killed this run once and costs a 40-minute drive plus 13 hosts down, three of them SureFire client machines; the GX10 is a ~240 W appliance at NH3 that can take nothing else down. Measured cost: ~79.4 s/it vs ana-ml2's 10.8-15.8, so 604 steps is ~13.3 h instead of ~2.5 h.",
|
|
"LIBRARY DELTA (recorded, not assumed harmless): torch 2.13.0 -> 2.14.0+cu130, transformers 5.15.1 -> 5.16.1, triton 3.7.1 -> 3.8.0; peft 0.20.0 and accelerate 1.14.0 unchanged. Attention backend flex_attention on both sides. Tokenisation equivalence across the transformers delta is asserted by comparing this host's encoded corpus against ana-ml2's byte-for-byte before launch, not inferred."
|
|
],
|
|
"unfittable": "drop",
|
|
"holdout_dir": "/home/infra-ops/erp-tune/datasets/holdout",
|
|
"save_steps": 50
|
|
}
|