esh-pfi-infrastructure/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md

# `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c

ASUS Ascent GX10 arrived and was registered, converted to headless, and given a rack-move
playbook. **It was NOT racked** — the operator ran out of day. It is still on his desk, on
Wi-Fi, on a temporary DHCP lease.

| | |
|---|---|
| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** |
| CPU | 20 cores, **aarch64** |
| Memory | **121 GB UNIFIED** — CPU and GPU share it; not 121 GB *plus* VRAM |
| Storage | 916 GB NVMe, 6% used |
| OS | Ubuntu 24.04.4, kernel 6.17.0-1031-nvidia |
| Access | `infra-ops` NOPASSWD sudo (operator-bootstrapped). `lkraven` has key auth but needs a password to escalate — **automation must connect as `infra-ops`**. |

## Purpose (operator, 2026-09-01)

Local inference experiments **and** the failed training — run 3c. That is the whole point:
run 3c died on ana-ml2 when a **kilowatt-class** training box tripped a breaker
(2× 300 W GPUs + dual EPYC 9254). The GX10 is a ~240 W appliance, roughly a fifth of the
draw, on a different site's circuits.

**The memory arithmetic favours it strongly.** Run 3c is a LoRA (r=64, batch 2 × accum 8,
gradient checkpointing) over `gemma4-26b-a4b-it-bf16` — **49 GB of base weights**, working
set roughly 55–65 GB. The `gemma4-charrp` compose warns in capitals that "48.10 GiB of BF16
weights CANNOT be served here" on ana-ml2's shared GPU0. 121 GB unified turns that
constraint into a non-issue.

## What has NOT been established — do not assume any of it

1. **The box is bare.** No torch, no nvcc, no CUDA stack. Docker 29.2.1 and 822 GB free.
2. **aarch64 dependency risk.** torch 2.9.1+cu130 ARM64 wheels exist; transformers/TRL/PEFT
   are pure Python. Compiled deps — flash-attn, bitsandbytes, liger, xformers — are open
   questions per-arch.
3. **Triton has NO sm_121 support** (established independently by comfy-dev the same day).
   Anything reaching for `torch.compile` or Triton-backed kernels is closed on this silicon.
4. **Throughput is unmeasured.** Run 3c ran 16.45 s/it on a 300 W RTX PRO 6000. GB10 will be
   slower; how much decides whether 604 steps is an overnight run or two days.
   **Measure this before porting anything** — the recommended first move is a probe: install
   ARM64 torch, load the base, run ten steps, report s/it.
5. **Model transfer:** 49 GB from ana-ml2 over the Anaheim↔NH3 link at a **measured
   32.4 MB/s** — about 25 minutes. Note that is a third of the irv-ml1↔NH3 link's 99 MB/s.

## The headless conversion, and the lesson inside it

`playbooks/gx10-headless.yaml` (`1b596c8`): multi-user.target, gnome-remote-desktop stopped,
sleep/suspend/hibernate **masked**, logind ignores lid and idle, sshd keepalives, hostname
corrected `gx10-a745` → `pfi-gx10`.

⚠ **`gdm` is a STATIC unit on Ubuntu** — pulled in by `display-manager.service`, never
"enabled". The first version guarded on `is-enabled | grep enabled`, which always skips, and
**the verify tested the same wrong property and passed**. Six green verifies having not
stopped the display manager. Both now test `is-active`. This is an instance of
[[2026-09-01-wrong-object-measurement]].

The playbook refuses to stop GDM while a seat session is held (`--var force_dm_stop=true` to
override) — automation should not yank a display out from under someone at the machine.

⚠ **elway's `--sudo` applies only to ad-hoc `--shell`/`--upload`.** Playbook steps run as the
connecting user and must carry their own `sudo`.

## The rack move, written but not run

`playbooks/gx10-rack-network.yaml` (`a0c5fc6`) — target settled: **`nh3-servers` VLAN 50,
static `10.100.50.60`**, clear of `.40`/`.42`/`.50`/`.90` and below the `.150` DHCP pool.

Requires nothing from the operator beyond racking it. The wired NIC has its own MAC
(`30:c5:99:3d:a7:45`, distinct from Wi-Fi `50:bb:b5:a2:00:a8`), so the post-move address AND
switch port are discoverable from the UDM rather than relayed.

**Design property worth preserving:** the playbook never leaves itself one path back. Wi-Fi
stays up while the wired interface is configured beside it; Wi-Fi teardown is explicitly a
separate later change. `netplan try`'s auto-rollback needs a TTY that elway cannot provide,
so a live Wi-Fi link is what substitutes for it. Two preconditions are asserted as steps
rather than assumed: carrier must be 1, and the MAC must match (interface names renumber
across kernels; MACs do not).

⚠ **The GX10 is NOT the Ada box.** Two separate machines — see
[[2026-09-01-ada-migration-branch-a]].


## `[2026-09-01 23:09]` Liveness check — alive, idle, and STILL BARE (nothing has changed)

Probed at **2026-09-01T23:09:51-07:00**, read-only:

    host       pfi-gx10   Ubuntu 24.04.4 LTS   kernel 6.17.0-1031-nvidia   aarch64
    uptime     11h48m (booted 2026-09-01 11:21)   load 0.00 0.01 0.00 — idle
    GPU        NVIDIA GB10   driver 580.173.02
    memory     121 GB unified, 118 GB free
    disk       /dev/nvme0n1p2  916 GB, 48 G used, 822 G free (6%)
    net        wlP9s9  10.100.10.226/24 (DHCP)  |  enP7s7 DOWN, NO-CARRIER (no cable)
    sessions   2x lkraven desktop (seat0 + :1) — sitting at a GUI login screen

**Compute stack unchanged and still absent:** no `torch`, no `nvcc`, no `uv`. Docker daemon
present but `infra-ops` is **not in the docker group**. Nothing has been installed since
onboarding.

⚠ **`nvidia-smi` reports FB Memory / BAR1 as `N/A`. That is CORRECT, not a fault** — GB10 is a
Grace Blackwell superchip with **unified** LPDDR5X shared between CPU and GPU, so there is no
discrete VRAM figure to report. Do not chase it as a driver problem.

## ⚠ RACKING IS **NOT** A PREREQUISITE — the Wi-Fi is not the bottleneck

Assumed the temp Wi-Fi would gate getting a 49 GB base onto the box. **Measured, and it does
not:**

    link       SSID PFI, 6 GHz, 160 MHz, EHT (Wi-Fi 7), -48 dBm
               rx/tx bitrate 2401.9 Mbit/s both directions
    measured   300 MB nh3-dev -> gx10 in 4.5 s = 67 MB/s over SSH
               (SSH cipher is the limiter, not the radio — link is ~300 MB/s)
    implies    49 GB bf16 base transfers in ~12 min

So the throughput probe (ARM64 torch, load base, ten steps, report s/it) can proceed **from the
desk, today**, without racking first. Racking remains one command
(`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`) and is worth doing for
permanence, but it does not gate the decision the probe exists to answer.


## `[2026-09-01 23:2x]` OUTFITTED (userspace) — CUDA WORKS on sm_121, full training stack imports

### ✅ RULING: BARE METAL, not Proxmox/VM. Settled, do not re-litigate.

1. **Proxmox VE has no aarch64 build.** It ships x86_64 only; the ARM ports are unofficial
   community work. Running an unsupported hypervisor under a compute asset is a bad trade.
2. **The GPU cannot be cleanly passed through.** GB10 is a superchip: the GPU sits on NVIDIA's
   own on-package root complex (bridges `22ce`/`22d0`/`22d1`, GPU at `000f:01:00.0`) and is
   **cache-coherent with the CPU over NVLink-C2C, sharing the same physical LPDDR5X.** Passing
   it to a guest means partitioning unified memory — which destroys the entire value
   proposition, since **121.6 GiB GPU-addressable** is the reason this box exists.
3. **Vendor's supported path is bare metal** — it already runs NVIDIA's own kernel flavour
   (`6.17.0-1031-nvidia`).
4. **Fleet convention agrees:** ana-ml2 and irv-ml1 are both bare-metal GPU hosts. Hypervisors
   are separate machines in this fleet, never under the accelerators.
5. **Single tenant.** VM isolation buys multi-tenancy nobody needs; the rollback benefit is
   better served by **containers + `uv` venvs**, which is what the rest of the fleet does.

### Installed and verified (all userspace, NO sudo needed)

    uv                0.12.9 (aarch64), ~/.local/bin, wired into .bashrc + .profile
    venv              ~/ml/.venv  (python 3.12)
    torch             2.14.0+cu130   cuda runtime 13.0, cudnn 92400, triton 3.8.0
    stack             numpy 2.5.2, transformers 5.16.1, accelerate 1.14.0, peft 0.20.0,
                      trl 1.12.0, datasets 5.0.1, safetensors 0.8.0, hf_hub 1.29.0,
                      bitsandbytes 0.50.2   -- ALL import clean on aarch64

**The "per-arch unknowns" warning did not materialise for the training stack.** Everything
imported first try.

### ✅ MEASURED GPU COMPUTE — it works, and the memory is the point

    device        NVIDIA GB10, capability sm_121, driver 580.173.02
    GPU memory    121.6 GiB total / 110.8 GiB free   <- unified, fully CUDA-addressable
    bf16 8192^3   ~93 TFLOP/s        fp16 ~93 TFLOP/s
    fp32+tf32     38.2 TFLOP/s       fp32 (no tf32) 19.5 TFLOP/s
    bf16/fp32     4.9x -> tensor cores ARE engaged

~93 TFLOP/s dense bf16 is roughly **A6000-class throughput with 2.5x the memory**. Capacity,
not speed, is what this box buys. A 49 GB bf16 base fits with ~70 GB to spare.

### ⚠⚠ I NEARLY REPORTED A PHANTOM PROBLEM — NO WARMUP MEANS YOU MEASURE THE JIT

First benchmark (10 iterations, **no warmup**) read **27.0 TFLOP/s bf16** — slower than a 3090
— and I was one step from reporting "tensor cores are not engaging on sm_121." Adding 5 warmup
iterations moved it to **93 TFLOP/s. The first measurement was timing the PTX JIT compile.**

⚠ **This is exactly `docs/pfi/training-throughput-playbook.md` §4: a run that completes, reports
plausible numbers, and is wrong.** It would have sent us chasing a cuBLAS fallback that does not
exist. **Always warm up before timing anything on this box** — see the JIT note below for why it
bites harder here than elsewhere.

### ⚠ sm_121 IS NOT IN TORCH'S COMPILED ARCH LIST — everything JITs from sm_120 PTX

    torch.cuda.get_arch_list() -> ['sm_80','sm_90','sm_100','sm_110','sm_120']
    device capability          -> sm_121

There is **no sm_121 cubin in the wheel**; it runs by PTX forward-compat JIT from sm_120.
Consequences:

- **First use of every kernel pays a JIT compile** (cached afterward in `~/.nv/ComputeCache`).
  This is what corrupted the benchmark above.
- ⚠ **Any library shipping cubin-only binaries with no PTX will simply fail here.** That is the
  real shape of the "per-arch unknown" risk — not source builds, but prebuilt kernels.

### ✅ PRIVILEGED HALF DONE — and I WAS NEVER BLOCKED

⚠⚠ **The block was my own bad test.** `infra-ops` has had **NOPASSWD sudo since account
creation** (`/etc/sudoers.d/infra-ops`, installed 11:31 with the account). I probed with
`sudo -n -v`, which returns non-zero on **sudo >= 1.9.15** because `-v` refreshes an auth
timestamp that a NOPASSWD-only rule never creates — and asked the operator for a password that
was never needed.

    pfi-gx10    sudo 1.9.15p5   sudo -n -v rc=1   sudo -n true rc=0
    nh3-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
    ana-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
    irv-ml1     sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0

**Always `sudo -n true`.** Fixed in `scripts/elway` (which carried the same probe and would
have started falsely prompting fleet-wide as hosts pass sudo 1.9.13) and in auto-memory
`reference_infra_ops_sudo_identity`, which *recommended the broken test*.
[[feedback_assert_effective_value_not_substring]]

⚠ **Same shape twice in one session.** I also reported "nvcc: ABSENT" from `command -v nvcc`
when the CUDA toolkit was installed all along at `/usr/local/cuda/bin/nvcc`, just not on PATH.
Both times I checked a proxy and reported it as the thing itself.

**Credential:** the temp password was `infra-ops`'s, not `lkraven`'s. Rotated to 32 random
chars and **vaulted at `pfi-gx10/infra-ops-password`** (round-trip verified). Kept rather than
locked — deviating from the canonical bootstrap's no-password posture **deliberately, because
this box has no iDRAC/BMC**, so a console break-glass credential is the only way back if key
auth breaks. ⚠ `lkraven` (uid 1000, GUI desktop user, `sudo` group) has a password nobody
knows; out of scope, but it is a gap.

### ⚠ DOCKER'S BUILDKIT DB WAS ALREADY CORRUPT — the restart surfaced it, did not cause it

`nvidia-ctk runtime configure` + `systemctl restart docker` failed with
`error initializing buildkit: error creating buildkit instance: invalid database`. **Not the
nvidia config** — `daemon.json` was valid and `nvidia-container-runtime` was present. dockerd
had been running on a process whose on-disk buildkit state was already bad; it simply could not
come back. Remediated by removing `/var/lib/docker/buildkit` (108 KB of **cache**, and the box
had **0 images, 0 containers**), then `daemon-reload` + `reset-failed` + start.

⚠ **A long-running daemon can be unable to restart for hours before anyone finds out.** Nothing
reported unhealthy until something asked it to restart.

### ✅ FINAL STATE — playbook green, 5/5 verifies

    playbooks/gx10-outfit-privileged.yaml -> steps 1 ok / 1 changed / 0 failed / 6 skipped
                                             verify 5 ok / 0 failed

    NOPASSWD sudo                    OK
    nvcc                             /usr/local/cuda/bin/nvcc (NOT on default PATH)
    docker socket as infra-ops       OK (docker group)
    CUDA container sees the GB10     OK (nvidia/cuda:13.0.0-base -> nvidia-smi -L reports GB10)
    userspace torch still sees GPU   OK
    docker Runtimes                  nvidia runc io.containerd.runc.v2

**The box is fully outfitted for inference + training, bare metal, containers available.**

### Historical note — what the block looked like before it was disproved

⚠ **`infra-ops` does NOT have NOPASSWD sudo on pfi-gx10** (`sudo -n -v` -> "a password is
required"), unlike the rest of the fleet — see [[reference_infra_ops_sudo_identity]], which
should be corrected: gx10 is an exception. **No gx10 credential is in the vault** (`secret list
--prefix pfi-gx10` -> 0 items), and `scripts/elway` prompts via `getpass`, which a
non-interactive session cannot answer.

Still outstanding, all needing root:

    - CUDA toolkit / nvcc (absent -- needed to COMPILE kernels, not to run torch)
    - NVIDIA Container Toolkit + add infra-ops to the docker group
      (docker daemon runs; infra-ops gets "permission denied" on the socket)
    - NOPASSWD sudo for infra-ops, so this box stops being a fleet exception
    - racking (playbooks/gx10-rack-network.yaml) -- optional, NOT a prerequisite

**Nothing above blocks the run-3c throughput probe**, which needs only the driver and torch.

## `[2026-09-01 23:5x]` Run-3c throughput probe — IN FLIGHT, and the ana-ml2 baseline is now on record

### The baseline nobody had written down: ana-ml2's real s/it for run 3c

Extracted from `/tank/erp-tune/run-03c.log` (the actual run, before the breaker killed it):

    step 10   10.81 s/it       step 20   12.88 s/it       step 24   15.76 s/it
    => 604 steps on ana-ml2 ~= 2.2-2.7 h

**This is the number the GX10 has to be compared against**, and it was sitting in a log
nobody had reduced. Anything under ~45 s/it on the GX10 makes 604 steps an overnight run.

### Exact run-3c geometry (from run-03c.json + a real adapter_config.json)

    base            /tank/aimodels/gemma4-26b-a4b-it-bf16   49 GB
                    gemma4, 30 layers, hidden 2816, 16 heads / 8 KV, MoE (A4B),
                    vocab 262144, sliding_window 1024, multimodal (vision+audio configs)
    LoRA            r 64, alpha 128, dropout 0.0, bias none, 205 target modules
                    (q/k/v/o + gate/up/down across all 30 language-model layers)
    shape           max_seq_len 16384, per_device_batch 2, grad_accum 8 (effective 16)
    other           gradient_checkpointing true, bf16, cosine, warmup 0.1, wd 0.01
    length          604 optimizer steps, 1 epoch, save_steps 50

### ⚠ THE BACKEND-DELTA TRAP — CHECKED, and it is NOT present

CLAUDE.md warns that two frontier-panel conclusions were voided because a benchmark and a
trainer had silently different attention backends. Enumerated both sides **before** running:

    ana-ml2   flash_attn NOT installed in /tank/erp-tune/venv  -> falls back to sdpa
    pfi-gx10  flash_attn not available for aarch64/sm_121      -> falls back to sdpa

Same backend both sides. ⚠ **Library versions DO differ and it is recorded rather than
assumed harmless:**

    ana-ml2   torch 2.13.0        transformers 5.15.1   peft 0.20.0   accelerate 1.14.0
    pfi-gx10  torch 2.14.0+cu130  transformers 5.16.1   peft 0.20.0   accelerate 1.14.0

The probe reads the **resolved** `_attn_implementation` back off the loaded model rather than
trusting the request, so the comparison is checkable rather than assumed.

### Probe design (§4-disciplined)

`~/ml/probe_3c.py` on the GX10. Real geometry, synthetic batches — it measures step cost, and
padding on the real run is 0.0% so the corpus does not change per-step cost materially.

- **12 steps, first 2 discarded as warmup.** Non-negotiable here: an unwarmed benchmark on
  this box already read 27 TFLOP/s when the true figure was 93, because it was timing the PTX
  JIT. sm_121 is not in torch's arch list.
- Reports the **step-time distribution** (median, min/max, every step), not a mean beside a
  settings dump.
- Emits **every provenance key always, null when unused** — present-and-null, not absent.

### Transfer

49 GB relayed **ana-ml2 -> nh3-dev -> pfi-gx10**, ~33 MB/s, ~25 min. Relayed rather than
direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs
nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an
access grant just to save a hop.**

## `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape

⚠ **PURPOSE, stated by the operator so it is not misread later: this was a BASELINE for the box
and a check that the tooling loads.** It was NOT a decision about where run 3c runs, and no port
was undertaken. The placement reasoning below is a *byproduct* — real and worth keeping, but the
deliverable was the number and the working stack.

**Deliverable 1 — the tooling loads and runs.** aarch64 + sm_121 runs the full training stack:
torch 2.14.0+cu130, transformers, accelerate, peft, trl, datasets, safetensors, bitsandbytes,
**plus the harness's own `flex_attention` backend and chunked-loss path**. Nothing exotic was
needed beyond `python3-dev`. This box can train.

**Deliverable 2 — the baseline.**

### The number

    pfi-gx10    MEDIAN 79.36 s/it   min/max 79.30 / 79.45   (10 timed steps, 0.19% spread;
                warmup 85.90 then 79.32, discarded)
                peak memory 75.1 / 121.6 GiB -- 46 GiB SPARE, comfortably not memory-bound
                attn_resolved: flex_attention (matched to the harness, read off the model)
                FINAL, from the probe's own provenance block on the box
                -> 604 steps ~= 13.3 h
                peak 75.1 / 121.6 GiB (PyTorch max_memory_allocated) -- 46 GiB spare
    ana-ml2     10.8-15.8 s/it (run-03c.log, the real run)
                -> 604 steps ~= 2.2-2.7 h

⚠ **6x slower where raw compute predicts 2.7x** (93 TFLOP/s bf16 measured here vs an RTX PRO
6000 Blackwell's ~250). **Hypothesis, not a measurement:** the limiter is **memory bandwidth**
— GB10's unified LPDDR5X against dedicated graphics memory, on a workload built to be
bandwidth-hungry (seq 16384 with gradient checkpointing, which trades compute for memory
traffic by design). Confirming it needs a bandwidth-bound microbenchmark that has not been run.

⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
fleet can. It is **not** the box to move a time-sensitive run *to*.

### Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG

⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is
worth more than the recommendation: **I costed the failure in the units I was measuring.** I had
step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the
recompute cost. That is not what a breaker trip at Anaheim costs.

**Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it.
it's not 25 minutes, it's a giant pain in the ass."*

**The real cost of an Anaheim breaker trip:**

    a 40-minute drive EACH WAY, on the operator's time, whenever he notices
    13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them),
      pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard),
      pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm
    ⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630)
      -- a client's production machines, under a hosting agreement

⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I
claimed; it only bounds the cheapest component of the loss.

**So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down
anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The
GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes
is expensive and lands on the operator personally.

⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between
options, check whether you priced the failure mode in the units you happened to be measuring.**
A metric in hand will volunteer itself as the unit of risk.

### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it

    2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W

Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but
nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it.
Recorded so nobody re-derives it as a clever third option.

### Probe methodology — five launches, five distinct causes, all recorded

    1. reaped SSH             long remote job piped through a reapable foreground ssh.
                              FIX: setsid nohup, log ON the box. Work survived; output did not.
    2. bare LoRA target names q_proj/k_proj/... also match Gemma-4's VISION and AUDIO towers,
                              whose projections are Gemma4ClippableLinear -- PEFT cannot wrap
                              them. FIX: the real run's 205 fully-qualified names, sha recorded.
    3. missing python3-dev    Triton JITs a C shim at FIRST USE and needs Python.h. Torch
                              imports, model loads, LoRA attaches, THEN step 1 dies with a bare
                              CalledProcessError -- Triton devnulls the compiler's stdout.
    4. naive-loss OOM (32 GiB) the 262,144-wide vocab x 32,768 positions logits tensor in fp32.
                              ⚠ NOT a GX10 limit -- the harness docstring says it OOMs "on a
                              95.60 GiB card (measured)", i.e. on ana-ml2 too.
    5. self-inflicted VRAM leak a stuck orphan held 80 GB while PyTorch reported 0 bytes
                              allocated; every relaunch was doomed before it started and blamed
                              the NEW run. ⚠ ASSERT THE GPU IS CLEAR BEFORE EVERY LAUNCH.

### ⚠ THE BACKEND-DELTA TRAP WAS PRESENT AND I FIRST DECLARED IT ABSENT

CLAUDE.md records two frontier-panel conclusions voided by a benchmark and a trainer having
silently different attention backends. I checked for it by asking whether **flash-attn was
installed** — the wrong discriminator entirely — and reported "no delta". The harness sets
**`ATTN_IMPLEMENTATION = "flex_attention"`** explicitly in code, with tuned kernel options; its
own comment records peak 74.3 GiB under flex vs sdpa's 79.71.

⚠ **Read the configured value in the code. Absence of an alternative is not evidence of the
default.** The probe now requests flex_attention and reads `_attn_implementation` back off the
loaded model. **RESOLVED: flex_attention — it compiles and runs on sm_121**, which was a real
open risk (flex compiles through Triton; sm_121 is not in torch's arch list).

### Faithfulness of the probe (what makes the number comparable)

    LoRA         the real run's 205 fully-qualified modules, sha256 c9ee070c0d67eb08
    loss path    copied from the harness: trunk hidden states, DROP IGNORE_INDEX before the
                 head, chunk lm_head+CE at loss_chunk_tokens=1024 under checkpoint, softcap 30.0
    loss ratio   28.4% of positions masked to match the real 13,310,930 / 18,600,057 loss/ctx
                 ratio -- observed 23,351-23,638 loss tokens/step against a 23,449 target
    tokens       sampled [1000, 250000) to stay below every multimodal special token
                 (boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4)
    warmup       first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use

## ⚠ NOT DONE, and deliberately: the run-3c port

Scoped but **not executed** — operator stood it down; the baseline was the goal. Scope, so it
does not have to be re-derived:

    harness eitri-smithy   26 MB      corpus /mnt/smithy/.../derived   2.3 GB
    recipe-r3             485 KB      holdout                           82 MB
    run-03c manifests      70 MB      -> ~2.5 GB, ~75 s at 33 MB/s
    venv                  3.2 GB      NOT portable (x86-64) -- rebuild on aarch64

⚠ **No encode cache to preserve.** The launch script's "609 MB encode cache / ~4.3 hours"
comment carries its own correction to 145.5 s, and run 3c's log shows **`[encode] done in
14.4s`** on 32 workers. Nothing worth moving.

⚠ **Copy the corpus; do NOT mount `/mnt/smithy` over NFS on this box.** A desk machine on Wi-Fi
is the wrong place for a hard NFS dependency under a 13-hour unattended run — the fleet has
already lost a host to a hard mount wedging into unkillable D-state
([[incident_esh_docker_nfs_boot_race]]).

The only untested piece is the harness's own 26 MB of Python; every library it leans on is
verified above.