# `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c ASUS Ascent GX10 arrived and was registered, converted to headless, and given a rack-move playbook. **It was NOT racked** — the operator ran out of day. It is still on his desk, on Wi-Fi, on a temporary DHCP lease. | | | |---|---| | GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** | | CPU | 20 cores, **aarch64** | | Memory | **121 GB UNIFIED** — CPU and GPU share it; not 121 GB *plus* VRAM | | Storage | 916 GB NVMe, 6% used | | OS | Ubuntu 24.04.4, kernel 6.17.0-1031-nvidia | | Access | `infra-ops` NOPASSWD sudo (operator-bootstrapped). `lkraven` has key auth but needs a password to escalate — **automation must connect as `infra-ops`**. | ## Purpose (operator, 2026-09-01) Local inference experiments **and** the failed training — run 3c. That is the whole point: run 3c died on ana-ml2 when a **kilowatt-class** training box tripped a breaker (2× 300 W GPUs + dual EPYC 9254). The GX10 is a ~240 W appliance, roughly a fifth of the draw, on a different site's circuits. **The memory arithmetic favours it strongly.** Run 3c is a LoRA (r=64, batch 2 × accum 8, gradient checkpointing) over `gemma4-26b-a4b-it-bf16` — **49 GB of base weights**, working set roughly 55–65 GB. The `gemma4-charrp` compose warns in capitals that "48.10 GiB of BF16 weights CANNOT be served here" on ana-ml2's shared GPU0. 121 GB unified turns that constraint into a non-issue. ## What has NOT been established — do not assume any of it 1. **The box is bare.** No torch, no nvcc, no CUDA stack. Docker 29.2.1 and 822 GB free. 2. **aarch64 dependency risk.** torch 2.9.1+cu130 ARM64 wheels exist; transformers/TRL/PEFT are pure Python. Compiled deps — flash-attn, bitsandbytes, liger, xformers — are open questions per-arch. 3. **Triton has NO sm_121 support** (established independently by comfy-dev the same day). Anything reaching for `torch.compile` or Triton-backed kernels is closed on this silicon. 4. **Throughput is unmeasured.** Run 3c ran 16.45 s/it on a 300 W RTX PRO 6000. GB10 will be slower; how much decides whether 604 steps is an overnight run or two days. **Measure this before porting anything** — the recommended first move is a probe: install ARM64 torch, load the base, run ten steps, report s/it. 5. **Model transfer:** 49 GB from ana-ml2 over the Anaheim↔NH3 link at a **measured 32.4 MB/s** — about 25 minutes. Note that is a third of the irv-ml1↔NH3 link's 99 MB/s. ## The headless conversion, and the lesson inside it `playbooks/gx10-headless.yaml` (`1b596c8`): multi-user.target, gnome-remote-desktop stopped, sleep/suspend/hibernate **masked**, logind ignores lid and idle, sshd keepalives, hostname corrected `gx10-a745` → `pfi-gx10`. ⚠ **`gdm` is a STATIC unit on Ubuntu** — pulled in by `display-manager.service`, never "enabled". The first version guarded on `is-enabled | grep enabled`, which always skips, and **the verify tested the same wrong property and passed**. Six green verifies having not stopped the display manager. Both now test `is-active`. This is an instance of [[2026-09-01-wrong-object-measurement]]. The playbook refuses to stop GDM while a seat session is held (`--var force_dm_stop=true` to override) — automation should not yank a display out from under someone at the machine. ⚠ **elway's `--sudo` applies only to ad-hoc `--shell`/`--upload`.** Playbook steps run as the connecting user and must carry their own `sudo`. ## The rack move, written but not run `playbooks/gx10-rack-network.yaml` (`a0c5fc6`) — target settled: **`nh3-servers` VLAN 50, static `10.100.50.60`**, clear of `.40`/`.42`/`.50`/`.90` and below the `.150` DHCP pool. Requires nothing from the operator beyond racking it. The wired NIC has its own MAC (`30:c5:99:3d:a7:45`, distinct from Wi-Fi `50:bb:b5:a2:00:a8`), so the post-move address AND switch port are discoverable from the UDM rather than relayed. **Design property worth preserving:** the playbook never leaves itself one path back. Wi-Fi stays up while the wired interface is configured beside it; Wi-Fi teardown is explicitly a separate later change. `netplan try`'s auto-rollback needs a TTY that elway cannot provide, so a live Wi-Fi link is what substitutes for it. Two preconditions are asserted as steps rather than assumed: carrier must be 1, and the MAC must match (interface names renumber across kernels; MACs do not). ⚠ **The GX10 is NOT the Ada box.** Two separate machines — see [[2026-09-01-ada-migration-branch-a]]. ## `[2026-09-01 23:09]` Liveness check — alive, idle, and STILL BARE (nothing has changed) Probed at **2026-09-01T23:09:51-07:00**, read-only: host pfi-gx10 Ubuntu 24.04.4 LTS kernel 6.17.0-1031-nvidia aarch64 uptime 11h48m (booted 2026-09-01 11:21) load 0.00 0.01 0.00 — idle GPU NVIDIA GB10 driver 580.173.02 memory 121 GB unified, 118 GB free disk /dev/nvme0n1p2 916 GB, 48 G used, 822 G free (6%) net wlP9s9 10.100.10.226/24 (DHCP) | enP7s7 DOWN, NO-CARRIER (no cable) sessions 2x lkraven desktop (seat0 + :1) — sitting at a GUI login screen **Compute stack unchanged and still absent:** no `torch`, no `nvcc`, no `uv`. Docker daemon present but `infra-ops` is **not in the docker group**. Nothing has been installed since onboarding. ⚠ **`nvidia-smi` reports FB Memory / BAR1 as `N/A`. That is CORRECT, not a fault** — GB10 is a Grace Blackwell superchip with **unified** LPDDR5X shared between CPU and GPU, so there is no discrete VRAM figure to report. Do not chase it as a driver problem. ## ⚠ RACKING IS **NOT** A PREREQUISITE — the Wi-Fi is not the bottleneck Assumed the temp Wi-Fi would gate getting a 49 GB base onto the box. **Measured, and it does not:** link SSID PFI, 6 GHz, 160 MHz, EHT (Wi-Fi 7), -48 dBm rx/tx bitrate 2401.9 Mbit/s both directions measured 300 MB nh3-dev -> gx10 in 4.5 s = 67 MB/s over SSH (SSH cipher is the limiter, not the radio — link is ~300 MB/s) implies 49 GB bf16 base transfers in ~12 min So the throughput probe (ARM64 torch, load base, ten steps, report s/it) can proceed **from the desk, today**, without racking first. Racking remains one command (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`) and is worth doing for permanence, but it does not gate the decision the probe exists to answer. ## `[2026-09-01 23:2x]` OUTFITTED (userspace) — CUDA WORKS on sm_121, full training stack imports ### ✅ RULING: BARE METAL, not Proxmox/VM. Settled, do not re-litigate. 1. **Proxmox VE has no aarch64 build.** It ships x86_64 only; the ARM ports are unofficial community work. Running an unsupported hypervisor under a compute asset is a bad trade. 2. **The GPU cannot be cleanly passed through.** GB10 is a superchip: the GPU sits on NVIDIA's own on-package root complex (bridges `22ce`/`22d0`/`22d1`, GPU at `000f:01:00.0`) and is **cache-coherent with the CPU over NVLink-C2C, sharing the same physical LPDDR5X.** Passing it to a guest means partitioning unified memory — which destroys the entire value proposition, since **121.6 GiB GPU-addressable** is the reason this box exists. 3. **Vendor's supported path is bare metal** — it already runs NVIDIA's own kernel flavour (`6.17.0-1031-nvidia`). 4. **Fleet convention agrees:** ana-ml2 and irv-ml1 are both bare-metal GPU hosts. Hypervisors are separate machines in this fleet, never under the accelerators. 5. **Single tenant.** VM isolation buys multi-tenancy nobody needs; the rollback benefit is better served by **containers + `uv` venvs**, which is what the rest of the fleet does. ### Installed and verified (all userspace, NO sudo needed) uv 0.12.9 (aarch64), ~/.local/bin, wired into .bashrc + .profile venv ~/ml/.venv (python 3.12) torch 2.14.0+cu130 cuda runtime 13.0, cudnn 92400, triton 3.8.0 stack numpy 2.5.2, transformers 5.16.1, accelerate 1.14.0, peft 0.20.0, trl 1.12.0, datasets 5.0.1, safetensors 0.8.0, hf_hub 1.29.0, bitsandbytes 0.50.2 -- ALL import clean on aarch64 **The "per-arch unknowns" warning did not materialise for the training stack.** Everything imported first try. ### ✅ MEASURED GPU COMPUTE — it works, and the memory is the point device NVIDIA GB10, capability sm_121, driver 580.173.02 GPU memory 121.6 GiB total / 110.8 GiB free <- unified, fully CUDA-addressable bf16 8192^3 ~93 TFLOP/s fp16 ~93 TFLOP/s fp32+tf32 38.2 TFLOP/s fp32 (no tf32) 19.5 TFLOP/s bf16/fp32 4.9x -> tensor cores ARE engaged ~93 TFLOP/s dense bf16 is roughly **A6000-class throughput with 2.5x the memory**. Capacity, not speed, is what this box buys. A 49 GB bf16 base fits with ~70 GB to spare. ### ⚠⚠ I NEARLY REPORTED A PHANTOM PROBLEM — NO WARMUP MEANS YOU MEASURE THE JIT First benchmark (10 iterations, **no warmup**) read **27.0 TFLOP/s bf16** — slower than a 3090 — and I was one step from reporting "tensor cores are not engaging on sm_121." Adding 5 warmup iterations moved it to **93 TFLOP/s. The first measurement was timing the PTX JIT compile.** ⚠ **This is exactly `docs/pfi/training-throughput-playbook.md` §4: a run that completes, reports plausible numbers, and is wrong.** It would have sent us chasing a cuBLAS fallback that does not exist. **Always warm up before timing anything on this box** — see the JIT note below for why it bites harder here than elsewhere. ### ⚠ sm_121 IS NOT IN TORCH'S COMPILED ARCH LIST — everything JITs from sm_120 PTX torch.cuda.get_arch_list() -> ['sm_80','sm_90','sm_100','sm_110','sm_120'] device capability -> sm_121 There is **no sm_121 cubin in the wheel**; it runs by PTX forward-compat JIT from sm_120. Consequences: - **First use of every kernel pays a JIT compile** (cached afterward in `~/.nv/ComputeCache`). This is what corrupted the benchmark above. - ⚠ **Any library shipping cubin-only binaries with no PTX will simply fail here.** That is the real shape of the "per-arch unknown" risk — not source builds, but prebuilt kernels. ### ✅ PRIVILEGED HALF DONE — and I WAS NEVER BLOCKED ⚠⚠ **The block was my own bad test.** `infra-ops` has had **NOPASSWD sudo since account creation** (`/etc/sudoers.d/infra-ops`, installed 11:31 with the account). I probed with `sudo -n -v`, which returns non-zero on **sudo >= 1.9.15** because `-v` refreshes an auth timestamp that a NOPASSWD-only rule never creates — and asked the operator for a password that was never needed. pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0 nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0 ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0 irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0 **Always `sudo -n true`.** Fixed in `scripts/elway` (which carried the same probe and would have started falsely prompting fleet-wide as hosts pass sudo 1.9.13) and in auto-memory `reference_infra_ops_sudo_identity`, which *recommended the broken test*. [[feedback_assert_effective_value_not_substring]] ⚠ **Same shape twice in one session.** I also reported "nvcc: ABSENT" from `command -v nvcc` when the CUDA toolkit was installed all along at `/usr/local/cuda/bin/nvcc`, just not on PATH. Both times I checked a proxy and reported it as the thing itself. **Credential:** the temp password was `infra-ops`'s, not `lkraven`'s. Rotated to 32 random chars and **vaulted at `pfi-gx10/infra-ops-password`** (round-trip verified). Kept rather than locked — deviating from the canonical bootstrap's no-password posture **deliberately, because this box has no iDRAC/BMC**, so a console break-glass credential is the only way back if key auth breaks. ⚠ `lkraven` (uid 1000, GUI desktop user, `sudo` group) has a password nobody knows; out of scope, but it is a gap. ### ⚠ DOCKER'S BUILDKIT DB WAS ALREADY CORRUPT — the restart surfaced it, did not cause it `nvidia-ctk runtime configure` + `systemctl restart docker` failed with `error initializing buildkit: error creating buildkit instance: invalid database`. **Not the nvidia config** — `daemon.json` was valid and `nvidia-container-runtime` was present. dockerd had been running on a process whose on-disk buildkit state was already bad; it simply could not come back. Remediated by removing `/var/lib/docker/buildkit` (108 KB of **cache**, and the box had **0 images, 0 containers**), then `daemon-reload` + `reset-failed` + start. ⚠ **A long-running daemon can be unable to restart for hours before anyone finds out.** Nothing reported unhealthy until something asked it to restart. ### ✅ FINAL STATE — playbook green, 5/5 verifies playbooks/gx10-outfit-privileged.yaml -> steps 1 ok / 1 changed / 0 failed / 6 skipped verify 5 ok / 0 failed NOPASSWD sudo OK nvcc /usr/local/cuda/bin/nvcc (NOT on default PATH) docker socket as infra-ops OK (docker group) CUDA container sees the GB10 OK (nvidia/cuda:13.0.0-base -> nvidia-smi -L reports GB10) userspace torch still sees GPU OK docker Runtimes nvidia runc io.containerd.runc.v2 **The box is fully outfitted for inference + training, bare metal, containers available.** ### Historical note — what the block looked like before it was disproved ⚠ **`infra-ops` does NOT have NOPASSWD sudo on pfi-gx10** (`sudo -n -v` -> "a password is required"), unlike the rest of the fleet — see [[reference_infra_ops_sudo_identity]], which should be corrected: gx10 is an exception. **No gx10 credential is in the vault** (`secret list --prefix pfi-gx10` -> 0 items), and `scripts/elway` prompts via `getpass`, which a non-interactive session cannot answer. Still outstanding, all needing root: - CUDA toolkit / nvcc (absent -- needed to COMPILE kernels, not to run torch) - NVIDIA Container Toolkit + add infra-ops to the docker group (docker daemon runs; infra-ops gets "permission denied" on the socket) - NOPASSWD sudo for infra-ops, so this box stops being a fleet exception - racking (playbooks/gx10-rack-network.yaml) -- optional, NOT a prerequisite **Nothing above blocks the run-3c throughput probe**, which needs only the driver and torch. ## `[2026-09-01 23:5x]` Run-3c throughput probe — IN FLIGHT, and the ana-ml2 baseline is now on record ### The baseline nobody had written down: ana-ml2's real s/it for run 3c Extracted from `/tank/erp-tune/run-03c.log` (the actual run, before the breaker killed it): step 10 10.81 s/it step 20 12.88 s/it step 24 15.76 s/it => 604 steps on ana-ml2 ~= 2.2-2.7 h **This is the number the GX10 has to be compared against**, and it was sitting in a log nobody had reduced. Anything under ~45 s/it on the GX10 makes 604 steps an overnight run. ### Exact run-3c geometry (from run-03c.json + a real adapter_config.json) base /tank/aimodels/gemma4-26b-a4b-it-bf16 49 GB gemma4, 30 layers, hidden 2816, 16 heads / 8 KV, MoE (A4B), vocab 262144, sliding_window 1024, multimodal (vision+audio configs) LoRA r 64, alpha 128, dropout 0.0, bias none, 205 target modules (q/k/v/o + gate/up/down across all 30 language-model layers) shape max_seq_len 16384, per_device_batch 2, grad_accum 8 (effective 16) other gradient_checkpointing true, bf16, cosine, warmup 0.1, wd 0.01 length 604 optimizer steps, 1 epoch, save_steps 50 ### ⚠ THE BACKEND-DELTA TRAP — CHECKED, and it is NOT present CLAUDE.md warns that two frontier-panel conclusions were voided because a benchmark and a trainer had silently different attention backends. Enumerated both sides **before** running: ana-ml2 flash_attn NOT installed in /tank/erp-tune/venv -> falls back to sdpa pfi-gx10 flash_attn not available for aarch64/sm_121 -> falls back to sdpa Same backend both sides. ⚠ **Library versions DO differ and it is recorded rather than assumed harmless:** ana-ml2 torch 2.13.0 transformers 5.15.1 peft 0.20.0 accelerate 1.14.0 pfi-gx10 torch 2.14.0+cu130 transformers 5.16.1 peft 0.20.0 accelerate 1.14.0 The probe reads the **resolved** `_attn_implementation` back off the loaded model rather than trusting the request, so the comparison is checkable rather than assumed. ### Probe design (§4-disciplined) `~/ml/probe_3c.py` on the GX10. Real geometry, synthetic batches — it measures step cost, and padding on the real run is 0.0% so the corpus does not change per-step cost materially. - **12 steps, first 2 discarded as warmup.** Non-negotiable here: an unwarmed benchmark on this box already read 27 TFLOP/s when the true figure was 93, because it was timing the PTX JIT. sm_121 is not in torch's arch list. - Reports the **step-time distribution** (median, min/max, every step), not a mean beside a settings dump. - Emits **every provenance key always, null when unused** — present-and-null, not absent. ### Transfer 49 GB relayed **ana-ml2 -> nh3-dev -> pfi-gx10**, ~33 MB/s, ~25 min. Relayed rather than direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an access grant just to save a hop.** ## `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape ⚠ **PURPOSE, stated by the operator so it is not misread later: this was a BASELINE for the box and a check that the tooling loads.** It was NOT a decision about where run 3c runs, and no port was undertaken. The placement reasoning below is a *byproduct* — real and worth keeping, but the deliverable was the number and the working stack. **Deliverable 1 — the tooling loads and runs.** aarch64 + sm_121 runs the full training stack: torch 2.14.0+cu130, transformers, accelerate, peft, trl, datasets, safetensors, bitsandbytes, **plus the harness's own `flex_attention` backend and chunked-loss path**. Nothing exotic was needed beyond `python3-dev`. This box can train. **Deliverable 2 — the baseline.** ### The number pfi-gx10 MEDIAN 79.36 s/it min/max 79.30 / 79.45 (10 timed steps, 0.19% spread; warmup 85.90 then 79.32, discarded) peak memory 75.1 / 121.6 GiB -- 46 GiB SPARE, comfortably not memory-bound attn_resolved: flex_attention (matched to the harness, read off the model) FINAL, from the probe's own provenance block on the box -> 604 steps ~= 13.3 h peak 75.1 / 121.6 GiB (PyTorch max_memory_allocated) -- 46 GiB spare ana-ml2 10.8-15.8 s/it (run-03c.log, the real run) -> 604 steps ~= 2.2-2.7 h ⚠ **6x slower where raw compute predicts 2.7x** (93 TFLOP/s bf16 measured here vs an RTX PRO 6000 Blackwell's ~250). **Hypothesis, not a measurement:** the limiter is **memory bandwidth** — GB10's unified LPDDR5X against dedicated graphics memory, on a workload built to be bandwidth-hungry (seq 16384 with gradient checkpointing, which trades compute for memory traffic by design). Confirming it needs a bandwidth-bound microbenchmark that has not been run. ⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the fleet can. It is **not** the box to move a time-sensitive run *to*. ### Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG ⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is worth more than the recommendation: **I costed the failure in the units I was measuring.** I had step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the recompute cost. That is not what a breaker trip at Anaheim costs. **Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it. it's not 25 minutes, it's a giant pain in the ass."* **The real cost of an Anaheim breaker trip:** a 40-minute drive EACH WAY, on the operator's time, whenever he notices 13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them), pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard), pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm ⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630) -- a client's production machines, under a hosting agreement ⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I claimed; it only bounds the cheapest component of the loss. **So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes is expensive and lands on the operator personally. ⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between options, check whether you priced the failure mode in the units you happened to be measuring.** A metric in hand will volunteer itself as the unit of risk. ### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it 2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it. Recorded so nobody re-derives it as a clever third option. ### Probe methodology — five launches, five distinct causes, all recorded 1. reaped SSH long remote job piped through a reapable foreground ssh. FIX: setsid nohup, log ON the box. Work survived; output did not. 2. bare LoRA target names q_proj/k_proj/... also match Gemma-4's VISION and AUDIO towers, whose projections are Gemma4ClippableLinear -- PEFT cannot wrap them. FIX: the real run's 205 fully-qualified names, sha recorded. 3. missing python3-dev Triton JITs a C shim at FIRST USE and needs Python.h. Torch imports, model loads, LoRA attaches, THEN step 1 dies with a bare CalledProcessError -- Triton devnulls the compiler's stdout. 4. naive-loss OOM (32 GiB) the 262,144-wide vocab x 32,768 positions logits tensor in fp32. ⚠ NOT a GX10 limit -- the harness docstring says it OOMs "on a 95.60 GiB card (measured)", i.e. on ana-ml2 too. 5. self-inflicted VRAM leak a stuck orphan held 80 GB while PyTorch reported 0 bytes allocated; every relaunch was doomed before it started and blamed the NEW run. ⚠ ASSERT THE GPU IS CLEAR BEFORE EVERY LAUNCH. ### ⚠ THE BACKEND-DELTA TRAP WAS PRESENT AND I FIRST DECLARED IT ABSENT CLAUDE.md records two frontier-panel conclusions voided by a benchmark and a trainer having silently different attention backends. I checked for it by asking whether **flash-attn was installed** — the wrong discriminator entirely — and reported "no delta". The harness sets **`ATTN_IMPLEMENTATION = "flex_attention"`** explicitly in code, with tuned kernel options; its own comment records peak 74.3 GiB under flex vs sdpa's 79.71. ⚠ **Read the configured value in the code. Absence of an alternative is not evidence of the default.** The probe now requests flex_attention and reads `_attn_implementation` back off the loaded model. **RESOLVED: flex_attention — it compiles and runs on sm_121**, which was a real open risk (flex compiles through Triton; sm_121 is not in torch's arch list). ### Faithfulness of the probe (what makes the number comparable) LoRA the real run's 205 fully-qualified modules, sha256 c9ee070c0d67eb08 loss path copied from the harness: trunk hidden states, DROP IGNORE_INDEX before the head, chunk lm_head+CE at loss_chunk_tokens=1024 under checkpoint, softcap 30.0 loss ratio 28.4% of positions masked to match the real 13,310,930 / 18,600,057 loss/ctx ratio -- observed 23,351-23,638 loss tokens/step against a 23,449 target tokens sampled [1000, 250000) to stay below every multimodal special token (boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4) warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use ## ⚠ NOT DONE, and deliberately: the run-3c port Scoped but **not executed** — operator stood it down; the baseline was the goal. Scope, so it does not have to be re-derived: harness eitri-smithy 26 MB corpus /mnt/smithy/.../derived 2.3 GB recipe-r3 485 KB holdout 82 MB run-03c manifests 70 MB -> ~2.5 GB, ~75 s at 33 MB/s venv 3.2 GB NOT portable (x86-64) -- rebuild on aarch64 ⚠ **No encode cache to preserve.** The launch script's "609 MB encode cache / ~4.3 hours" comment carries its own correction to 145.5 s, and run 3c's log shows **`[encode] done in 14.4s`** on 32 workers. Nothing worth moving. ⚠ **Copy the corpus; do NOT mount `/mnt/smithy` over NFS on this box.** A desk machine on Wi-Fi is the wrong place for a hard NFS dependency under a 13-hour unattended run — the fleet has already lost a host to a hard mount wedging into unkillable D-state ([[incident_esh_docker_nfs_boot_race]]). The only untested piece is the harness's own 26 MB of Python; every library it leans on is verified above.