vh
7db6c44bcd
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus the operator's "keep the adapter" ruling made durable. Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq 4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on: Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96). Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect. Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused linear-attention kernel is installed on the box. Grad checkpointing is not the culprit (19%, and saves 2.6x memory). Batching is not the lever for either family -- both sit at this box's roofline at batch 1. Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B 4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune the regime exists to replace, so the carrier family is now an open decision with a recommendation for the dense Qwen3 line -- the design doc's original pin. Two further Qwen3.5 findings, both measured rather than read off the config: the Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that target_modules="all-linear" would train on text) and an MTP head, both dropped for free by loading through AutoModelForCausalLM -- which renames modules relative to the vLLM serving path, so adapter binding needs the sampled-target-changed check on the serving side; and cross-document packing is unsafe because SSM state ignores the attention mask, breaking the per-copy name-consistency invariant the design doc calls sacred. Neither exists on dense. Adapter disposition, per the operator's ruling: all five gx10-resident ERP adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter matching the layout runs 01-03 already used, byte-totals identical both sides and sha256 matching on every adapter_model.safetensors. /tank/* is deliberately excluded from ana-ml2's restic sources, so the profile gains one documented carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run to expand to exactly those eight paths. Nothing is training and nothing is queued.
92 lines
3.4 KiB
YAML
92 lines
3.4 KiB
YAML
# resticprofile config for ana-ml2.
|
|
#
|
|
# **Bare metal** — not protected by any Proxmox vzdump. This is the only
|
|
# backup for this host's config/state, so it's the highest-stakes repo in
|
|
# the fleet. Writes to the Anaheim rest-server at 10.250.50.70 as user
|
|
# `ana-ml2`. The full REST URL (with HTTP basic auth creds) lives in
|
|
# /etc/restic/restic.env. The client-side encryption passphrase lives in
|
|
# /etc/restic/password.
|
|
#
|
|
# Notable exclusions:
|
|
# - /tank/* is NOT in source paths. That's ~TB of model weights (HF
|
|
# caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
|
|
# from upstream. Backing them up would blow the repo size budget.
|
|
# ONE carve-out: /tank/erp-tune/run-*/adapter. A trained LoRA adapter is
|
|
# the only thing under /tank that upstream cannot hand back — each is
|
|
# ~300 MB and cost 7-14 h of GPU time, and the ERP line's adapters exist
|
|
# nowhere else but pfi-gx10's single NVMe and here. ~2.4 GB total.
|
|
# - No DB dumps needed. None of the stacks on this host (llama-swap,
|
|
# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
|
# dozzle-agent, dockge) store relational data.
|
|
|
|
version: "1"
|
|
|
|
global:
|
|
priority: low
|
|
ionice: true
|
|
ionice-class: 2
|
|
ionice-level: 7
|
|
min-memory: 100
|
|
|
|
default:
|
|
env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@…
|
|
env:
|
|
RESTIC_PASSWORD_FILE: /etc/restic/password
|
|
initialize: false # repo created manually by `restic init`
|
|
lock: /var/lock/restic-ana-ml2.lock
|
|
|
|
backup:
|
|
verbose: 1
|
|
run-after:
|
|
- date +%s > /var/lib/restic/last-success
|
|
source:
|
|
- /opt/docker # compose files + host-level configs — small (~100MB)
|
|
- /etc # host config (systemd units, chrony, apparmor, ssh, etc.)
|
|
- /root # root shell history, ssh keys, any ad-hoc scripts
|
|
- /var/lib/docker/volumes # named volumes (small; models live on /tank, not here)
|
|
- /tank/erp-tune/run-*/adapter # non-regenerable trained LoRA adapters (see header)
|
|
exclude:
|
|
# Docker internals we never want in a backup
|
|
- /var/lib/docker/volumes/backingFsBlockDev
|
|
- /var/lib/docker/volumes/metadata.db
|
|
# Parakeet's HF model cache — named volume (stacks that bind-mount
|
|
# from /tank/ are already outside the source list; parakeet is the
|
|
# odd one out using a docker volume). Regenerable from Hugging Face.
|
|
- /var/lib/docker/volumes/parakeet_parakeet_cache
|
|
# Ephemeral / regenerable junk
|
|
- /opt/docker/compose/*/logs
|
|
- "**/*.log"
|
|
- "**/*.log.*"
|
|
- "**/*.pid"
|
|
# Root's noise — shell caches, tmp
|
|
- /root/.cache
|
|
- /root/.local/share/Trash
|
|
- /root/.npm
|
|
- /root/.python_history
|
|
tag:
|
|
- host:ana-ml2
|
|
- site:ana
|
|
- fleet:pfi
|
|
schedule: "*-*-* 01:00:00"
|
|
schedule-permission: system
|
|
schedule-log: /var/log/restic-backup.log
|
|
|
|
forget:
|
|
keep-daily: 7
|
|
keep-weekly: 4
|
|
keep-monthly: 12
|
|
keep-yearly: 3
|
|
# NOTE: no `prune: true` — rest-server runs with --append-only.
|
|
# See README.md "Prune ceremony".
|
|
tag:
|
|
- host:ana-ml2
|
|
# Schedule removed 2026-04-21: forget against --append-only rest-server
|
|
# always fails (delete ops blocked). Run manually during the prune
|
|
# ceremony when --append-only is temporarily off.
|
|
|
|
check:
|
|
read-data-subset: 10%
|
|
schedule: "Sun *-*-* 05:00:00"
|
|
schedule-permission: system
|
|
schedule-log: /var/log/restic-check.log
|