vh
c507db9ac0
The proposed shape was QLoRA r64. It cannot be run as specified. The checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj [128,2816,704] — no .weight suffix, so they are parameters, not modules). bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params / 42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit saves ~3.1 GiB of 48.07 and does not error while doing it. Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per additional 8192-token sequence. Two sizing items were absent from the brief and both are load-bearing: - vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping 30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at ~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a 95.6 GiB card — it starts, then OOMs on the first long sample. Fused or chunked linear CE is mandatory and must be smoke-proven before a window is booked, since Liger may not carry a Gemma-4 MoE patch. - v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the full-attention layers). A v_proj target silently produces no adapter there, and k_proj adapts K and V simultaneously. 45.96M trainable at r64 across q/k/v/o. Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr. Recommend standing down sec (2 aliases, last request ~5h ago) rather than gen (7 aliases, 765 busy-engine log lines in 24h). Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at effective batch 16, which makes the wall-clock-checkpointing amendment concrete rather than hypothetical. Package as a uv venv on /tank: root is 91% full (36 GB) with /var/lib/docker on it.
docs/
Navigation map for the documentation tree. New session? Read
orientation.md first — it's the narrative overview
of the fleet, backup architecture, governing principles, and gotchas,
and it points at everything else.
Tree
docs/
├── orientation.md # start here — fleet overview + where-to-look guide
├── runbooks/ # ops runbooks (recovery, deployment phases)
│ ├── disaster-recovery.md
│ ├── nh3-prune-ritual.md
│ └── pbs-deployment.md
└── pfi/ # PFI-specific reference (services, models, VMs)
├── docker-stack.md
├── model-list.md
├── proxmox-vms.md
├── recommended-model-settings.md
├── vm-102-matrix-appservice.md
└── vm-102-matrix-synapse.md
What goes where
runbooks/— step-by-step ops procedures. Anything you'd reach for during an incident or while standing up new infrastructure. Examples: disaster recovery (blast-radius tiers + restoration steps), PBS deployment (9-phase rollout). New runbook → new file here.pfi/— PFI-specific reference material that's too narrow for the top-level CLAUDE.md but doesn't change incident response. AI model inventory, recommended inference settings, Matrix bridge config, Proxmox VM map. New stable reference → new file here.- Top-level (
docs/orientation.md,docs/README.md) — narrative guides about the workspace itself, not about specific infra.
Cross-references
- Fleet topology + servers table: top-level
CLAUDE.md. - Open work + recent milestones: top-level
STATUS.md. - Durable cross-session facts:
~/.claude/projects/-home-lkraven-development-eshpfi-management/memory/.
Conventions
- Markdown, GitHub-flavored. CommonMark renders fine in most viewers.
- File names are lowercase-kebab-case, descriptive. No dates in filenames — git history covers that.
- One topic per file. If a file grows past ~500 lines, look for a natural split before adding more.
- No checked-in binaries or checksums. Build/release artifacts belong
in a build pipeline or
tools/, notdocs/.