The proposed shape was QLoRA r64. It cannot be run as specified. The
checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter
tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj
[128,2816,704] — no .weight suffix, so they are parameters, not modules).
bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params /
42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit
saves ~3.1 GiB of 48.07 and does not error while doing it.
Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per
additional 8192-token sequence.
Two sizing items were absent from the brief and both are load-bearing:
- vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping
30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at
~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a
95.6 GiB card — it starts, then OOMs on the first long sample. Fused or
chunked linear CE is mandatory and must be smoke-proven before a window
is booked, since Liger may not carry a Gemma-4 MoE patch.
- v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the
full-attention layers). A v_proj target silently produces no adapter
there, and k_proj adapts K and V simultaneously. 45.96M trainable at
r64 across q/k/v/o.
Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and
gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on
GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr.
Recommend standing down sec (2 aliases, last request ~5h ago) rather than
gen (7 aliases, 765 busy-engine log lines in 24h).
Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours
at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at
effective batch 16, which makes the wall-clock-checkpointing amendment
concrete rather than hypothetical.
Package as a uv venv on /tank: root is 91% full (36 GB) with
/var/lib/docker on it.
vh
c507db9ac0