Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
First backup run pulled in 9 GB due to /var/lib/docker/volumes/
parakeet_parakeet_cache — Parakeet is the only AI stack on ana-ml2
using a docker named volume for its HF model cache (kokoro, vibevoice,
comfyui, llama-swap, vllm-qwen3 all bind-mount from /tank which is
already outside source paths).
Excluding brings expected snapshot size back to ~100-300 MB.
rest-server-ana README now describes the /mnt/backup/restic/repo/
top-level NFS mount and its three per-site subdirs:
ana/ — live data served by this rest-server (per-host repos +
.htpasswd) — what DATA_DIR points at
esh/ — mirror destination for ESH-site backups (pending)
nh3/ — mirror destination for NH3 Synology's tree (pending)
ana-ml2 README gains a proper "Recreating the repo" section with the
correct /mnt/backup/restic/repo/ana/ana-ml2/ path for wiping the old
repo after a lost passphrase, and two paths for regenerating keys:
- interactive: type a user-generated passphrase at restic's init
prompt, then install it into /etc/restic/password via `cat > file`
+ Ctrl-D (no shell history or transcript exposure)
- scripted: openssl rand -base64 48, passphrase prints once and must
be captured into the password manager immediately
Cross-site replication snippet in rest-server-ana README updated to
use the unified /mnt/backup/restic/repo/{esh,nh3}/ destinations
instead of the earlier restic-mirror-*/ staging paths.
Original used `sudo env \$(cat /etc/restic/restic.env) …` but the `cat`
in command substitution runs as the login user, not root. Since the
env-file is root:600, the substitution silently yielded an empty
RESTIC_REPOSITORY and restic errored with "Please specify repository
location".
Wrap the whole dance in `sudo bash -c "…"` so the env-file read and
the restic call both happen as root.
An ana-ml2 user and repo were created during the original backup
pipeline pass. Reuse keeps snapshot history consolidated and avoids
duplicate infrastructure. Reworks the README to:
- Skip `restic init` (repo exists) and the htpasswd step (user exists)
- Install the two existing secrets (REST URL w/ htpasswd password,
repo passphrase) into /etc/restic/{restic.env,password}
- Verify credentials against the existing repo via `restic snapshots`
Fresh-setup flow retained below as a fallback for zero-state rebuilds.
ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it.
This closes the biggest single backup gap per the 2026-04-20 pipeline
audit.
Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes.
Excludes /tank/* (model weights — regenerable from Hugging Face and
would blow repo size budget). No pre-backup DB hook — none of the
llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks
use relational databases.
README walks through the one-time setup: rest-server .htpasswd entry,
restic init with fresh passphrase, resticprofile install, systemd timer
generation, verification against the Backrest UI.
vh