Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.
Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:
Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s
Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s
Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s
The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.
Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.
Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.
Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.
Nothing is training and nothing is queued.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
Selectively include /worktank/<stack>/ subtrees now that comfyui,
parakeet, and cosyvoice place real user state there. Bulk weights,
scratch outputs, and the ~8 GB disposable comfyui run/ venv stay
out — both by the include list being precise and by belt-and-
suspenders exclude patterns.
Added sources:
/worktank/comfyui/basedir/user workflows + per-user settings
/worktank/comfyui/basedir/custom_nodes hand-installed extensions
/worktank/comfyui/basedir/input user-uploaded source images
/worktank/cosyvoice/voices cloned speaker profiles
Belt-and-suspenders excludes (inert under current sources; guards
against a future wholesale-add of /worktank):
/worktank/comfyui/basedir/{models,output,temp}
/worktank/comfyui/run
/worktank/parakeet/models
/worktank/cosyvoice/{input,output}
Verified by `resticprofile backup --dry-run` on irv-ml1 — 1.5 GiB
scanned across all 9 sources, 13 MiB new to the repo. If any bulk
dir had leaked in, the total would be multi-GB.
Also fixed a stale /home row in the README (profile only sources
/home/lkraven; llmuser + sduser are explicitly excluded).
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
nh3-dev is the author's active dev workstation at the NH3 site
(nh3-dev.phasefinal.com). 22 GB home with 12 GB of development code,
shell history, dotfiles, Claude Code memory, and scripts that weren't
being captured by any fleet backup.
Writes site-local to the Synology rest-server (10.100.50.50:8000)
matching the nh3-docker pattern.
Excludes trim ~8 GB of regenerable caches + build outputs:
- language toolchains (.cargo, .rustup, .npm, .m2, .gradle, go/pkg, etc.)
- editor caches (.vscode-server, .vscode)
- project build directories (node_modules, __pycache__, .venv, target,
dist, build, .pytest_cache, .tox, .next, .nuxt) via `**/` patterns
- trash / browser caches / steam / snap
Expected first snapshot ~14 GB, incrementals much smaller due to
content-defined dedup across dev code.
README walks through the full setup: resticprofile install, Synology-
side htpasswd addition (via DSM File Station or Container Manager since
the DSM SSH setup is tabled), repo init, profile deploy, timer enable,
first backup. Schedule is nightly 01:00 with systemd's Persistent=true
behavior so missed runs catch up at next boot (workstation is often
off/sleeping at backup time).
README.md fleet-coverage table updated to reflect 6/6 file-level
coverage (ana-docker, ana-ml2, nh3-docker, esh-docker-vm, vm-esh-nas,
nh3-dev).
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:
- paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
- home-assistant → local SQLite ~50MB (host sqlite3 .backup)
- calibre-web-automated → local SQLite (in-container sqlite3)
- pgadmin → local SQLite (host sqlite3)
- uptime-kuma → local SQLite (host sqlite3; container name may vary
after force-recreate, detect by label)
Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.
Cross-site writes to rest-server-ana since ESH has no local rest-server.
NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).
Found in audit (non-blocking but noted in README follow-ups):
- paperless-ngx Postgres password is literally "paperless-ng" —
trivially weak, rotate at next opportunity.
The Anaheim NAS exports /mnt/backup with root_squash, so sudo-as-root
on ana-docker becomes nobody on the NFS side and `sudo tee` gets EACCES.
The repo tree is owned by lkraven:lkraven precisely because of this —
all writes go through the lkraven UID.
Drop sudo from the example; plain `>>` append as lkraven works.
Second ESH Docker host — light (filezilla + dockge + agents, /opt/docker
is 56 KB). Cross-site writes to the Anaheim rest-server since ESH has
no local rest-server.
Critical detail: vm-esh-nas NFS-mounts /mnt/{share,music,books,media}
from 10.0.50.50 (~400 TB). Profile's exclude patterns explicitly
reject those paths as a safety net on top of the source list not
including them — a careless future edit to sources can't nuke the
backup repo by pulling in 400 TB.
Sources include /home/ (lkraven/nas/user dirs) in addition to the
usual /opt/docker /etc /root /var/lib/docker/volumes — this host has
multiple user accounts worth preserving dotfiles for.
No pre-backup hook: no relational DBs.
README walks through the full setup: install restic (not present on
this host), add htpasswd entry on rest-server-ana, install creds,
init, install resticprofile, schedule timers. Also flags the 3.8 GB
RAM constraint.
The compressed zip defeated restic's content-defined chunking: each
day's dump looked completely different to restic even when the repo
content barely changed, causing repo growth of ~full dump size (821 MB
at last measurement) every day until forget/prune aged snapshots out.
Uncompressed tar is dedup-friendly. After the first snapshot, daily
incrementals cost only the actual new-data delta — typically a few
MB for an active repo.
Tradeoff: stage file on the client host is ~2-3x the zip size while
the dump is in flight, but that's transient (purged at the start of
each run). Repo-side storage is much smaller over time.
Adds file-level restic for the NH3 Docker VM. Targets the Synology
rest-server (rest-server-nh3 at 10.100.50.50:8000) as user nh3-docker —
site-local writes matching the fleet pattern.
No pre-backup hook needed: none of the stacks on this host (adguard,
dockge, beszel-agent, dozzle-agent, portainer) run relational DBs. The
SQLite state in their named volumes is WAL-mode and restores cleanly
from raw restic capture.
Source paths mirror ana-ml2's template (/opt/docker, /etc, /root,
/var/lib/docker/volumes) with the same exclude list (docker internals,
logs, pids, root's shell/caches). ~1.6 MB of compose files + small
volumes — expected snapshot size a few hundred MB at most, dominated
by any AdGuard query log history.
README walks through reuse (existing repo + htpasswd) vs fresh init
paths, resticprofile install, timer generation, and verification via
the Backrest UI.
Audit surfaced three DB-backed services not being dumped consistently
by the existing pre-backup.sh:
- vaultwarden — migrated to external Postgres on PFI-Postgres
(10.250.50.80); old sqlite block was dumping stale pre-migration
files. Replace with pg_dump against the live database. Requires
postgresql-client on ana-docker and VW_PG* creds in
/etc/restic/dbcreds.env.
- gitea — also on PFI-Postgres; no hook existed at all. Use
`gitea dump` for a single zip that captures DB + repos + config +
LFS + attachments consistently. No explicit creds needed; the
container reads its own GITEA__database__* env.
- openwebui — two local SQLite databases (webui.db + the ChromaDB
vector store). .backup command if sqlite3 is in the image, volume-
level fallback otherwise.
Refactor: each block now logs a WARN on failure instead of aborting the
whole script — partial dumps > no dumps when one service has an issue.
dbcreds.env.example committed as a template; real file goes to
/etc/restic/dbcreds.env root:600 on the host and is never in the repo.
Mattermost retired (user confirmed 2026-04-21); removed from the
pre-backup.sh list and flagged in README's stacks section. Mattermost
container was not running regardless; the audit surfaced that it was
already effectively gone. Compose dir on ana-docker can be removed as
separate cleanup.
First backup run pulled in 9 GB due to /var/lib/docker/volumes/
parakeet_parakeet_cache — Parakeet is the only AI stack on ana-ml2
using a docker named volume for its HF model cache (kokoro, vibevoice,
comfyui, llama-swap, vllm-qwen3 all bind-mount from /tank which is
already outside source paths).
Excluding brings expected snapshot size back to ~100-300 MB.
rest-server-ana README now describes the /mnt/backup/restic/repo/
top-level NFS mount and its three per-site subdirs:
ana/ — live data served by this rest-server (per-host repos +
.htpasswd) — what DATA_DIR points at
esh/ — mirror destination for ESH-site backups (pending)
nh3/ — mirror destination for NH3 Synology's tree (pending)
ana-ml2 README gains a proper "Recreating the repo" section with the
correct /mnt/backup/restic/repo/ana/ana-ml2/ path for wiping the old
repo after a lost passphrase, and two paths for regenerating keys:
- interactive: type a user-generated passphrase at restic's init
prompt, then install it into /etc/restic/password via `cat > file`
+ Ctrl-D (no shell history or transcript exposure)
- scripted: openssl rand -base64 48, passphrase prints once and must
be captured into the password manager immediately
Cross-site replication snippet in rest-server-ana README updated to
use the unified /mnt/backup/restic/repo/{esh,nh3}/ destinations
instead of the earlier restic-mirror-*/ staging paths.
Original used `sudo env \$(cat /etc/restic/restic.env) …` but the `cat`
in command substitution runs as the login user, not root. Since the
env-file is root:600, the substitution silently yielded an empty
RESTIC_REPOSITORY and restic errored with "Please specify repository
location".
Wrap the whole dance in `sudo bash -c "…"` so the env-file read and
the restic call both happen as root.
An ana-ml2 user and repo were created during the original backup
pipeline pass. Reuse keeps snapshot history consolidated and avoids
duplicate infrastructure. Reworks the README to:
- Skip `restic init` (repo exists) and the htpasswd step (user exists)
- Install the two existing secrets (REST URL w/ htpasswd password,
repo passphrase) into /etc/restic/{restic.env,password}
- Verify credentials against the existing repo via `restic snapshots`
Fresh-setup flow retained below as a fallback for zero-state rebuilds.
ana-ml2 is not on any Proxmox hypervisor, so vzdump doesn't touch it.
This closes the biggest single backup gap per the 2026-04-20 pipeline
audit.
Sources: /opt/docker (~110 MB), /etc, /root, /var/lib/docker/volumes.
Excludes /tank/* (model weights — regenerable from Hugging Face and
would blow repo size budget). No pre-backup DB hook — none of the
llama-swap / vllm / comfyui / kokoro / parakeet / vibevoice stacks
use relational databases.
README walks through the one-time setup: rest-server .htpasswd entry,
restic init with fresh passphrase, resticprofile install, systemd timer
generation, verification against the Backrest UI.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh