pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, and kernel
6.8.12-42 staged on the /boot LV. dpkg clean, nothing outstanding for
apt -f install, all PVE services active, cluster quorate, no unapplied
conffiles. Reboot deliberately deferred at operator request, so the host
still runs 6.8.12-13 until a chosen window.
This validates the GRUB fix from 061c4b7 under the exact condition it
was written for. update-grub regenerated entries for the new kernel and
entry 0 -- what GRUB_DEFAULT=0 selects -- is now
/vmlinuz-6.8.12-42-pve with root=ZFS=nvme/ROOT/pve-1, supplied by the
grub.d drop-in since grub-mkconfig cannot derive the pool name itself.
The old kernel keeps correct entries as a fallback and the ext4 rollback
entry is untouched. Had the fix not landed first, saved_entry would
still be pinned to 6.8.12-13 and the host would boot the old kernel
indefinitely -- 161 security updates installed and never run.
/boot holds both kernel sets at 176M used of 488M, confirming the 512M
LV carved out of swap was sized correctly.
Adds a ZFS snapshot step to the upgrade playbook, taken automatically on
ZFS-root nodes before any package lands. That is the first real use of
the boot-environment upside the migration was meant to unlock: rollback
for this upgrade is now `zfs rollback -r
nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z && reboot` rather than
archaeology in dpkg. Also documents that the corosync bump restarts
corosync mid-upgrade, which on a 2-node cluster is a brief quorum event.
171 packages, pve-manager 8.4.11 -> 8.4.20, kernel 6.8.12-16 -> 6.8.12-42,
corosync 3.1.9 -> 3.1.10-pve2. dpkg clean, no unapplied conffiles, no
failed units, cluster quorate with both nodes visible after the reboot.
Adds a reusable pve-node-upgrade playbook (upgrade only -- reboot stays a
separate deliberate step, since it has cluster and NFS consequences the
playbook cannot see). It guards on quorum and free space, snapshots
/etc/pve and friends first, uses --force-confdef/--force-confold, and
surfaces any .dpkg-dist files that policy left unapplied so they are not
silently ignored.
The reboot needed a forced guest stop, operator-authorised after the risk
was surfaced. Two obstacles, only one of them ours:
- A vzdump had been hung since 14 April -- 126 days, stalled at 0% of
256 GiB -- holding lock: backup on VM 102, which had therefore been
sitting in QEMU prelaunch that entire time. Killed by explicit PID; 102
is now cleanly stopped rather than half-alive.
- esh-vm-db would not shut down: its guest agent had died and ACPI went
unanswered. Most likely ours -- it hard-mounts /mnt/backup from CT 103,
which we deliberately left mounted through the NAS reboots.
PostgreSQL survived the hard stop. It had checkpointed five minutes
prior, so recovery replayed 56 bytes of WAL in 0.02s and came up ready;
all four databases present and queryable. That was lucky timing as much
as anything -- a hard stop mid-checkpoint on a busy database would not
read the same way.
The reboot also repaired esh-vm-db, which had silently lost sshd, mongod
and its guest agent. All three are back.
The cutover left saved_entry=pve-zfs-root, a hand-authored 40_custom
entry hardcoding /vmlinuz-6.8.12-13-pve. The pending upgrade installs
proxmox-kernel-6.8.12-42, which made that a trap with two exits: if -13
were autoremoved the default entry would point at a missing kernel and
the host would need console recovery it has no IPMI for; if -13 survived
the host would silently keep booting the old kernel, so 161 security
updates including a kernel would install and never run.
That entry was written as a one-time cutover target. It was never fit to
be the standing default across kernel upgrades, and this is remediation
of that, caught before the upgrade rather than after.
Fix is to stop hand-authoring the ZFS entry: GRUB_DEFAULT=0 boots the
first auto-generated entry, which grub-mkconfig regenerates for the
newest kernel on every install, and which /etc/default/grub.d/zfs-root.cfg
already corrects to the pool-qualified root=ZFS=nvme/ROOT/pve-1. grubenv
is cleared so nothing overrides it.
The rollback entry stays pinned, which is correct rather than an
oversight: it boots the untouched ext4 root on the DOM, whose /boot is
never regenerated because update-initramfs writes only to the /boot LV.
That kernel genuinely never changes.
Also adds a reusable safe-reboot playbook for this host, carrying the
constraints that are easy to forget: quiesce the hard-NFS clients first,
the other cluster node goes read-only while this one is down (quorum 2,
no qdevice), and a failed boot has no auto-fallback and no remote
console.
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.
The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.
The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.
It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.
Fixes and guards:
- --make-rslave after every rbind, plus a guard that refuses to proceed
while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
five NFS clients, not the two documented. The new one that mattered is
esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
which GRUB reads but cannot write, so next_entry survived the boot
that consumed it. Steady state is saved_entry=pve-zfs-root with no
next_entry. There is no auto-fallback on this host and no IPMI.
Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
Worldtree #401: a slow fd accrual in worldtree-personal hit the 1024 soft
nofile ceiling and converted into a hard deadlock. Operator authorized the
raise 2026-08-17 (relayed via worldtree-dev); sizing 65536 agreed.
Applied at the daemon layer rather than compose because /opt/worldtree-*/
compose.yaml on corviduo-dev is written by the team CI deploy identity -- a
host-side compose edit reverts on the next deploy and would leave a false
'raised' record. Daemon config is infra-ops-owned and covers all 13
containers on the box. worldtree-dev shipped a redundant compose-level pin
(e41b139) as the belt to this braces.
daemon.json is written and valid, but the floor is STAGED, NOT ACTIVE:
default-ulimits is not in dockerd's SIGHUP-reloadable set. Measured on
29.4.3 -- the post-reload 'Reloaded configuration' log enumerates the live
config without default-ulimits, and a fresh container still reports
ulimit -n 1024. Activation needs a full dockerd restart, which bounces every
container; not taken, since #401 is not urgent at fd ~100 and the compose
pin already covers worldtree. The playbook documents this and its verify
step 3 fails by design until a restart happens.
The prior fix (_netdev,nofail + docker.service After=remote-fs.target) looked
correct but silently failed — paperless still Exited(255) on the 2026-07-14
reboot. Root cause: `nofail` drops a mount out of remote-fs.target's blocking
set, so ordering docker After=remote-fs.target does NOT wait for the nofail
NFS mounts. Fix: add x-systemd.before=docker.service,x-systemd.mount-timeout=30
to the 4 NFS fstab lines (direct mount->docker ordering, nofail-safe). Applied
+ verified live (systemctl show docker -p After now lists all 4 mnt-*.mount).
Playbook + verify updated to canonicalize.
On a train reaching succeeded, IN ADDITION to output/{name}.safetensors
(unchanged download source), COPY it into ComfyUI's loras search path at
/storetank/arbo/models/loras/trained/{train_id}/{name}.safetensors and
return published_lora_name (the ComfyUI-relative LoraLoader string) in the
terminal GET /train/{id} payload (arbo Phase 2 auto-registration, §4.1/§7).
- Copy not move; a publish failure NEVER fails the train (keeps succeeded,
omits published_lora_name, logs the reason to the tailable run log).
- INV-T7-safe: a copy to a fixed computed path, no new free-form args.
- train_id derived from the handoff layout (output_dir.parent.name).
- Provisions loras/trained/ (arbotrain 2775, group-write per the Phase-1
lesson; world-readable/traversable for ComfyUI) via the deploy playbook.
- ComfyUI verified to resolve nested loras subfolders (no flat fallback).
- Pure path helper unit-tested; 16 tests green.
Host service (runs as llmuser, owns /opt/fluxgym + GPU access) that runs
sd-scripts SDXL LoRA training on demand for arbo — the infra-ops half of the
in-arbo LoRA training Phase 1 ownership split (vh/arbo
docs/contracts/in-arbo-lora-training-phase1.contract.md §4.1/§2).
- Fixed-invocation only (INV-T7): bounded params -> one sd-scripts command
shape; every param range/allowlist/path-containment checked before spawn;
bad request = 422, never a silent downgrade. 14 unit tests green.
- Thin supervisor: never imports torch; subprocesses the fluxgym venv's
accelerate. 1-job-at-a-time (arbo lease is the serializer, 409 is backstop).
Durable job records + boot reconciliation (§4.3).
- API: POST /train, GET /train/{id}[/log], POST /train/{id}/cancel,
GET /gpu-status (per-device VRAM + tts_on_3090 co-OOM signal), GET /healthz.
- Wire-shape (§7 resolved with comfy-dev): shared /worktank/arbo/train handoff
(group arbotrain, setgid 2770); worker binds 0.0.0.0:8203, arbo reaches via
host.docker.internal:host-gateway (reachability proven on 172.20.0.1:8203);
device-aware TTS steering via /gpu-status.
Deployed to irv-ml1 via playbooks/deploy-lora-training-worker.yaml (elway,
idempotent); systemd unit active; /healthz + /gpu-status verified live.
Add a live-consumer streaming path and text sanitation to the OmniVoice
wrapper, so it can front speech-to-speech chat engines (not just the
asset-engine's batch WAV use).
- POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by
the adaptive buffer-ratchet scheduler. Emits the first sentence
immediately, then ratchets chunk size up on OmniVoice's ~40x realtime
headroom -> sub-second time-to-first-audio. Wire-compatible with
chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is
unchanged for asset/file callers.
- scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure-
Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1).
Vendor-copy over a shared package (operator call 2026-06-19): the module
has no GPU deps, so reuse it without dragging chatterbox-fast's torch
tree into this image. Promote to a shared package only on a 3rd consumer
or real drift.
- sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips
markdown, <think> blocks, HTML, and model control tokens; deliberately
SKIPS the fork's English-only number/phone normalization that would
corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags.
- Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest;
single GEN_LOCK serializes generation (single-stream interactive).
- Dockerfile/playbook: copy + upload the two new modules; build-time
`import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 on ana-ml2 GPU 0 (heretic downed):
- stacks/qwen3.5-122b/ — vLLM serve via the repo's text-only patch (Qwen3.5 MoE is a
multimodal arch but this checkpoint is text-only weights), --reasoning-parser qwen3,
GPU 0 pin, :8013; entrypoint+patch mounted from the model dir.
- serve-qwen3.5-122b.yaml — displace heretic + serve + verify.
- litellm: REMOVED dead mistral-small-4 / -reasoning; added qwen3.5-122-a10b[-reasoning]
+ aliases qwen-large[-reasoning] + repointed gen[-reasoning] -> qwen (thinking split via
chat_template_kwargs.enable_thinking + --reasoning-parser qwen3).
Verified live: qwen healthy on :8013; gen / qwen-large / qwen3.5-122-a10b route, and
gen-reasoning returns reasoning_content; mistral-small-4 removed.
NOTE: Worldtree character backend (was bound to mistral-small-4) is dark until repointed
(operator-acknowledged).
- app.py: thin FastAPI wrapper exposing OpenAI /v1/audio/speech (+ /v1/audio/voices,
/healthz) around OmniVoice's Python API; precomputes a voice-clone prompt per voice
at startup (loaded Whisper auto-transcribes each reference). Replaces the Gradio demo.
- Dockerfile/compose: run the uvicorn wrapper, /healthz healthcheck, project name pinned
to "omnivoice" so the asset-engine liveness probe matches.
- deploy-omnivoice.yaml: stage chatterbox /refs/*.wav as clone voices (skip _* artifacts)
+ verify the API surface.
- services.yaml: catalog entry (id omnivoice, :8199/v1/audio/speech, voice list sourced
live from /v1/audio/voices) + reproducibility_audit row.
Verified live on irv-ml1: /healthz ok, 33 voices loaded, test synth -> 24kHz PCM_16 WAV.
Zero-shot, massively-multilingual (600+ language) voice-cloning + voice-design
TTS (diffusion-LM, Apache-2.0). No official image, so a thin CUDA container
around the pip package running upstream's own Gradio demo (no FastAPI wrapper).
Pinned to GPU 0 (3090) — the A6000 is ComfyUI-exclusive — port 8199. Built +
verified live on irv-ml1 (Gradio 200, container healthy). Surface is the Gradio
UI + Gradio API, NOT OpenAI-compat /v1/audio/speech (wrap later if asset-engine
should consume it). deploy-omnivoice.yaml builds local + verifies.
ana-ml2 ran overcommit_memory=0 with zero swap, capping the CommitLimit at
~RAM/2 (~283 GB of 566 GB). The resident vLLM services commit ~224 GB, so a
large model-file mmap (the 50 GB NVFP4 shard during HF->native conversion, or
a vLLM model load) failed with ENOMEM despite ~393 GB of RAM actually free.
overcommit_memory=1 is the conventional setting for ML hosts that mmap large
files. A drop-in under /etc/sysctl.d/ makes it reboot-durable. Operator-directed
permanent (2026-06-17). Idempotent via when:; sudo tee for the root-owned path
(elway runs steps as the SSH user, so a shell > redirect can't write there).
The irv-ml1-arbo Gitea Actions runner (host-executor as lkraven under
systemd) gets the bare service PATH (/usr/local/bin:/usr/bin:/bin), which
omits ~/.local/bin — so the CI uv bootstrap failed with "uv: not found".
Symlink uv/uvx into /usr/local/bin (on the systemd PATH) to fix it and
retire the per-run curl|sh bootstrap. Idempotent (creates: guard);
re-applies cleanly after a runner rebuild. Authorized by comfy-dev
(engine owner) per althing thread 01KV94VTS27B.
Append 10.100.10.50:8391 to BIFROST_CLIENT_ALLOWED_HOSTS on the personal
Worldtree (.env) so the consumer may bind the memory provider at session-create
(affect :8390 was already listed; the url-guard 422s un-allowlisted endpoints).
Idempotent elway playbook; surgical worldtree-api recreate that auto-derives the
image pin from the matrix sibling to avoid the stale-:latest crash-block footgun.
Repoint servers/corviduo-dev/ssh-target to infra-ops (operator granted durable
NOPASSWD admin on corviduo-dev 2026-06-15).
Operator decision 2026-06-13 (relayed by comfy-dev, confirmed in-session):
turn off the prod arbo engine's bearer auth and rely on the WireGuard
perimeter. Reverses ADR-0001's open-auth-hole-closed posture (comfy-dev owns
the ADR update on the vh/arbo side).
The app's protected-gate no-ops only when ENGINE_TOKEN is ABSENT — an empty
string still gates (verified: ENGINE_TOKEN="" -> /workflows 401). So both
inject paths are removed: the compose environment line is commented out and
the .env line deleted on the host. Result: tokenless GET /workflows 200 (was
401), matching the dev engine. Original token preserved in the host's
.env.pre-auth-off.bak for re-enable.
playbooks/arbo-disable-engine-token.yaml captures the reversible procedure.
Default gcc is gcc-11 (CUDA/torch R14 pin) but the only gfortran is 12,
whose libgfortran-12-dev puts the unversioned libgfortran.so only in the
gcc-12 dir. R links shared objects with gcc-11 + -lgfortran, so the whole
Fortran dependency chain (quadprog/tseries/RcppArmadillo -> soundgen) failed
'cannot find -lgfortran'. Adds an idempotent step installing libgfortran-11-dev
(guarded on 'gcc -print-file-name=libgfortran.so' not resolving). Additive;
default toolchain untouched.
No-sudo phase: uv, R16 world venv + pyworld (setuptools<81 pin for
pyworld's pkg_resources import). Sudo phase (run as infra-ops): R +
soundgen system deps, and pin the Ollama service to the A6000 BY UUID
(CUDA FASTEST_FIRST ordering != nvidia-smi PCI order on this box, so an
index pin is ambiguous). R15 models share the A6000's free headroom —
yt-voice-clipper + R14 keep running, card not freed.
Adds a host-agnostic elway play + fleet driver that stand up an
`infra-ops` system user (dedicated ed25519 key, NOPASSWD sudo with
log_output audit, docker group) so the infra-ops agent completes
DevOps work end-to-end instead of handing sudo steps back to the
operator. Scoped to PFI-owned Linux boxes; tiered (compute/app/
sensitive-infra) with SureFire/corviduo/esh/Synology explicitly
excluded. Validated live on irv-ml1.
reference_id=<name> resolves against the DIRECTORY references/<name>/
(audio + same-basename .lab), not a flat references/<name>.wav. Voices
were staged flat with the per-name dirs left empty, so every
reference_id resolved to nothing and Fish fell back to its default
speaker — every dropdown voice produced byte-identical audio (proven:
Abigail == Imogen == no-ref, same text+seed). This was the real "no
accent" root cause, independent of the asset-engine "undefined" select
bug.
Server fix (applied to irv-ml1): populated references/<name>/<name>.wav
+ <name>.lab for all 32 voices; re-test confirms Imogen/Eleanor/
Beatrice/Abigail/no-ref now all distinct.
Durable hardening + record correction:
- playbook: normalize-layout step (flat <name>.wav -> nested dir, cp -u
idempotent, when-gated on count mismatch) + an A/B verify gate that
hard-fails the deploy if two reference_ids yield identical output.
- services.yaml: correct the reference_id resolution doc (dir + .lab,
not flat wav).
- README + persistent-memory: correct the "reference_id-by-name is THE
working path, verified" claim — it was a no-op until this fix; the
prior ECAPA 0.79 result came through the inline base64 path.
override.conf already existed on esh-docker-vm (dockerd ExecStart), so
the creates: guard skipped the step and the After=remote-fs.target
directive was never written. Use a distinct filename
(10-after-remote-fs.conf) that systemd merges alongside override.conf,
and guard on content rather than file existence so a stale file can't
silently skip.
Adds _netdev,nofail to the 4 10.0.50.50 NFS lines in /etc/fstab and a
docker.service drop-in (After=remote-fs.target) so Docker waits for the
NFS mounts before starting NFS-bind containers. Root cause of paperless
Exited(255) on every reboot. Idempotent; backs up fstab; no reboot.
The verify step ran 0.09s after `docker compose up -d --force-recreate`,
well before the container's 30s start_period elapsed — so 'healthy'
isn't yet in docker ps's Status string and the grep failed. False
negative; container was operationally up (verify 1/2 already confirmed
/health 200).
Greping for '^Up' instead catches the case verify 2/2 actually wants
to catch (container exited entirely) without racing the healthcheck.
Registry-pull pattern matching Worldtree: CI on vh/skaldsong builds and
pushes gitea.phasefinal.com/vh/skaldsong:<sha>, this playbook pulls +
recreates. SHA-pin only per current preference; no :latest moving-tag
advance yet (revisit once /health exercises Worldtree + Kokoro reach).
Host port 8300 (host) → 8000 (container). Persistent state under
/opt/docker/conf/skaldsong/{db,runs}.
Bifrost endpoint URL 10.250.50.70:8300 will need a paired
BIFROST_CLIENT_ALLOWED_HOSTS update on corviduo-dev Worldtree at first
deploy.
Phase 3.1 closes the cross-process gap the Phase 3 smoke surfaced —
streaming events (msg_start/thinking/delta/complete/curated) flow
from agent-runner → chamber via valkey pub/sub rather than the
SQLite bridge (too high-volume + ephemeral for the DB).
New service: `althing-valkey` (stock `valkey/valkey:8-alpine`).
Internal-only — no exposed port, no volume. chamber + agent-runner
reach via docker DNS at `valkey:6379` on the compose default
network. healthcheck via `valkey-cli ping` (5s interval). chamber
+ agent-runner gain `depends_on: valkey: service_healthy` so the
bridge is up before either side starts publishing or subscribing.
Forseti unchanged — never publishes Phase 3 events.
Operational properties (per forseti's deployment notes):
- Mixed-state safe at every step. Missing valkey.url config key
→ chamber + runner stay on v3.0 / Phase 2 equivalent paths.
- Backward path is single config-key delete + restart.
- streaming_enabled: true (set on agent-runner 2026-05-17) is
unaffected by this change.
README's services table + playbook header + verify section all
extended to reflect the four-service shape. Forseti's contract
at vh/althing:docs/contracts/phase3_1_valkey_bridge.contract.md
carries the wire-protocol spec.
Phase 2 daemon added to the althing-chamber stack per forseti's request
(vh/althing@5cd088a..ad1d025). Polls floor_grants WHERE consumed_at IS
NULL AND agents.driver='worldtree', claims via atomic UPDATE, calls
Worldtree's conversation API, posts the response back through the bus
as a broadcast.
Shape matches the existing forseti daemon:
- Same ${ALTHING_IMAGE} (the binary is already in [project.scripts]
as of ad1d025)
- command: ["althing-agent-runner"]
- Same shared SQLite bind-mount at /app/data
- No port, no healthcheck (CLI doesn't expose one; same liveness
story as forseti)
Safe to enable preemptively per forseti — when no driver=worldtree
handles are declared in config, the runner sleeps at
poll_interval_seconds. Multi-instance safe via the atomic claim
primitive (no flock needed).
Compose top comment, README "Services in this stack" table, playbook
header + verify steps all extended to reflect the three-service
shape. Will land on ana-docker on vh/althing's next push (compose
deployed via the elway playbook's upload step; image already carries
the binary).
Two-service compose (chamber + forseti sidecar daemon) sharing a single
SQLite store via bind-mount under /opt/docker/conf/althing-chamber/data.
eventbus.bridge_from_db is the cross-process glue — forseti's commits
reach chamber's SSE subscribers via the bridge.
Pattern matches task-board's build-on-host deploy:
- elway playbook clones vh/althing into /opt/docker/build/
- docker build -t althing-chamber:local . (no registry)
- playbook uploads compose + seeds .env one-time, brings both
services up, polls /health
- Gitea Actions workflow lives in vh/althing; reference copy here.
Internal tooling — host port 7881 (chamber's default of 7878 collides
with task-board). LAN-direct, no Traefik. Container always listens on
8000 internally.
Scaffold will fail to bring the chamber container up healthy until
galdrabok-side commits land:
- Dockerfile at vh/althing repo root (two-stage: uv-bookworm-slim
build → python:3.12-slim runtime, locked per open_questions §2
of the v1 contract).
- GET /health endpoint on the chamber app (200, no DB read).
- ALTHING_BIND / ALTHING_PORT env-var support in
core.cli.chamber_serve / core.chamber.cli (env > config.yaml >
defaults precedence).
Coordinated via althing thread 01KRMAK7RD7TP6C8DF4KXV31RT.
Playbook handles models, datasets, and spaces (via --var repo_type=...)
since 3025d49 — the "-model" suffix was misleading. Renaming to match
actual scope.
Updates the single in-repo reference (changelog comment in
stacks/llama-swap/conf/config.yaml). config.yaml was scp'd to ana-ml2;
no docker compose restart needed (comment-only).
Adds optional repo_type var (default "model"; valid: model/dataset/space)
that's forwarded to `hf download --repo-type` and threaded through the
verify step (cache-dir prefix tracks repo_type: models--, datasets--,
spaces--).
The playbook was previously model-only because hf download defaults
repo_type=model — pulling a dataset surfaced as a 404 against the wrong
HF API path. Now generic.
Smoke-tested 2026-05-13:
scripts/elway ana-ml2 --playbook playbooks/pull-hf-model.yaml \
--var hf_repo=Skywork/Skywork-Reward-Preference-80K-v0.2 \
--var repo_type=dataset
→ 200 MB parquet cached at
/tank/aimodels/huggingface/hub/datasets--Skywork--Skywork-Reward-Preference-80K-v0.2/
Filename is misleading now (playbook handles more than models); leaving
the rename for a future cleanup since pull-hf-model.yaml is referenced
from the llama-swap config.yaml's 2026-05-13 changelog entry.
Codifies the previously-manual workflow described in
stacks/llama-swap/README.md: install hf CLI via pipx (one-time),
inject hf_transfer for fast multi-connection downloads,
`hf download` into the shared HF cache at /tank/aimodels/huggingface
with optional --include filter.
Model-format-agnostic by design — same playbook handles GGUFs for
llama-swap and safetensors for vLLM (both stacks read the same cache
dir via HF_HOME=/hfcache). Does NOT edit any consumer's config.yaml;
per-model run params (ctx-size, sampler defaults, quant choice,
chat template, etc.) stay human-curated.
Usage:
scripts/elway ana-ml2 --playbook playbooks/pull-hf-model.yaml \
--var hf_repo=<user>/<repo> \
[--var allow_patterns='*Q6_K*']
Idempotent: hf CLI skips already-cached blobs; re-runs are
sub-second when the snapshot is already complete.
Smoke-tested 2026-05-13 against:
- mradermacher/Selene-1-Mini-Llama-3.1-8B-GGUF (Q6_K, ~6.5 GB)
- Skywork/Skywork-Reward-V2-Llama-3.1-8B (full safetensors, ~16 GB)
Three changes prepping infra for asset_engine's orchestration feature
(SSH-driven bring-up / bring-down of irv-ml1 inference services with
per-device VRAM gating, contract in vh/asset-engine commit 5a36f8c):
1. asset-engine compose + .env.example + playbook gain a read-only
bind-mount for /app/runtime/ssh — the dedicated ed25519 keypair
(generated on ana-docker, not in the repo) plus a pinned known_hosts
for irv-ml1's host fingerprint. Env vars SSH_KEY_PATH and
SSH_KNOWN_HOSTS are exposed for the app to consume.
2. docs/asset-engine/services.yaml gains a `lifecycle: { stack, vram_gb,
gpu_device_id }` block on each of 12 orchestratable irv-ml1 services
(kokoro, chatterbox, index-tts, qwen3-tts, cosyvoice, fish-s2,
kyutai-tts, vibevoice, voxtral, parakeet, stable-audio-open, ace-step).
VRAM numbers are estimates from model footprint at fp16 — tune from
real nvidia-smi measurements once the gate is live. comfyui and
kokoro-captioned are deliberately excluded (variable-VRAM and
shared-container respectively).
3. servers/irv-ml1/README.md docker-stacks table now lists all 13
inference stacks (was only dockge + agents + comfyui) with port +
GPU pinning columns.
Pubkey deployed to ~lkraven/.ssh/authorized_keys on irv-ml1;
end-to-end SSH from ana-docker → irv-ml1 verified with strict
host-key checking.
Internal tooling — accessed at http://10.250.50.70:8200, not through
Traefik. Removes the unused traefik labels (router rule, TLS, crowdsec
middleware, loadbalancer port) and the traefik-net network membership;
homepage.href now points at host:port for direct discovery, matching
task-board's pattern. Playbook verify drops the traefik-net membership
check.
Mirrors task-board's build-on-host pattern: elway playbook clones
vh/asset-engine into /opt/docker/build/, docker build, install compose +
seed .env, up -d, verify /health. No registry.
Internal-only tool — LAN port 8200 (bind 0.0.0.0) is primary; Traefik
labels additionally route asset-engine.phasefinal.com with TLS via the
anaprod cert resolver. DB and outputs are separate bind-mounts under
/opt/docker/conf/asset-engine/ so outputs/ can move volumes later
without touching DB state. INFERENCE_HOST defaults to 10.100.79.3
(irv-ml1 over WG). OIDC env seam is pre-allocated empty for v2.
Wrapper only enumerates one voice directory (settings.voices_dir,
default /app/api/src/voices/v1_0 — inside the container's writable
layer, not bind-mounted). Override via VOICES_DIR=/app/user_voices
(host bind mount) and add a command shim that cp -r's built-ins from
the in-image v1_0 into user_voices on every start. Built-ins re-seed
fresh from the image (so upgrades that add voices propagate); custom
.pt files in user_voices are preserved (cp -r is additive).
Also adds scripts/blend_kokoro_voice.py + a playbook around it that
mirrors the wrapper's request-time voice="a(w)+b(w)" math but writes
the result as a named .pt to user_voices, making it discoverable via
GET /v1/audio/voices and persistent across recreate. Defaults to
athena = af_bella(2)+af_aoede(1) normalized.
Same shape as task-board: build-on-host from vh/vor, bind-mounted
persistence for sessions/ and responses/ (the user-published markdown
files), exposed at port 7879 (adjacent to task-board's 7878 since both
are claude-tooling sidecars).
Workflow template assumes the same DEPLOY_SSH_KEY + MGMT_REPO_TOKEN
secrets at user scope; nothing new to provision. Playbook accepts SHA
or branch refs (same fix as deploy-task-board.yaml) so manual runs
and CI runs share the same code path.
Centralized vs upstream-local: README documents the trade. Claude
fetches response markdown via /api/sessions/{id} JSON instead of a
local file read — the only API-flow change from the upstream README.
CI passes --var ref=<github.sha> (a full SHA), but the playbook
hardcoded `git reset --hard origin/{{ ref }}` which only works for
branch names — `origin/<sha>` is invalid syntax. Resolve ref via
git rev-parse with `^{commit}` (try origin/<ref> first for branch
names, fall back to bare <ref> for SHAs/tags) so manual runs (ref=main)
and CI runs (ref=<sha>) both work.
Same fix applied to the changed_when comparison so no-op reruns still
report ok instead of changed.
Central runner on ana-docker (gitea is local; existing fleet tooling
already SSHes from there). Playbook is parameterized so future
site-local runners (nh3-docker, esh-docker-vm) drop in via --var
overrides instead of copy-paste.
Includes a workflow template for vh/task-board that calls the existing
deploy-task-board.yaml playbook — keeps the playbook as the single
source of truth for "how task-board is deployed", manual or automated.
Labels embed `:docker://node:20-bookworm-slim` schema; without it,
act_runner v0.6+ silently falls back to host-mode and runs job steps
inside the Alpine runner container (no apt/python/node), breaking any
real workflow. node:20-bookworm-slim is small + has git + node so
actions/checkout works out of the box.
The applet outgrew "stack alongside the infra-management workspace" —
it has its own pyproject, multi-tenant deploy story, separate
release cadence, and isn't actually about managing infrastructure.
Lives at https://gitea.phasefinal.com/vh/nevermore now, with
provenance noted in its initial commit.
This commit removes:
stacks/news-digest/ (full stack tree)
playbooks/deploy-news-digest.yaml
scripts/add-digest-user.sh
The existing ana-docker deployment continues running on its baked
local/news-digest:v5 image — nothing changes for the live install
until you choose to redeploy from the new repo. Migration steps
(rename data dir, redeploy, retire old compose dir) are in
nevermore's README.
Updated:
README.md — Current stacks listing now points at the new repo
STATUS.md — milestones entry for the extraction
Stock neosmemo/memos:stable, port 5230, SQLite at
/opt/docker/conf/memos/data/. Joins traefik-net and ships homepage
labels (group=Notes) so it auto-appears on the dashboard via docker
discovery — no edit to configs/homepage/services.yaml needed.
First-run bootstrap is via the UI: visit http://10.250.50.70:5230
and create the Host account through the sign-up form.
Playbook idiom note: docker compose pull lines need the literal
block scalar (|) when the grep pattern contains colons — bare-string
shell value made YAML parse the colon as a mapping separator and
elway choked on first try.
Three upstream gaps surfaced once /generate was actually exercised:
1. infer-api.py builds an 18-arg positional tuple but the pipeline
expects 24 — first missing arg is `format`, so audio_duration
shifts into format's slot and the pipeline calls len() on an
int. Ship a patched copy of infer-api.py and COPY over upstream's
in the Dockerfile. Also handle empty lora_name_or_path -> "none"
(empty string trips HF Hub's repo-id validator).
2. torchcodec + ffmpeg are required by the WAV save path but neither
is in upstream requirements.txt. Without them every /generate
runs to completion and then 500s at write-time.
3. ACE-Step caches checkpoints at /root/.cache/ace-step/checkpoints
(HARDCODED, not honored by HF_HOME). Mount our persistent dir
there so the ~7 GB model survives container recreates.
Bench on A6000 (cached model, lo-fi hip hop, 60-step euler/apg):
10s @ 27 steps -> 9.4s (0.94x)
30s @ 60 steps -> 11.2s (0.37x, ~2.7x realtime)
60s @ 60 steps -> 14.8s (0.24x, ~4x realtime)
Two new audio-generation stacks alongside the TTS slate:
ace-step :8210 — Apache 2.0 music generation foundation model
(hybrid diffusion + LLM). Lyric-aware multi-minute songs. ~10-12 GB
VRAM during inference, A6000-pinned. Custom Dockerfile patches
upstream's torch/cu126 resolution bug (--extra-index-url cu126 was
falling back to pypi-default cu13 wheels, mismatching torchvision).
stable-audio-open :8211 — Stability AI 1.21B latent-diffusion SFX +
ambience. Up to 47s clips at 44.1 kHz. ~6 GB VRAM in fp16,
A6000-pinned. Custom FastAPI shim around diffusers' StableAudioPipeline
(no upstream HTTP server). Dockerfile pins torchsde explicitly —
diffusers doesn't pull it as a hard dep but
CosineDPMSolverMultistepScheduler needs it.
Three deploy iterations + four backend attempts (subprocess CUDA,
resident-server CUDA, Vulkan rebuild) all failed to deliver speedup
over fish-s2:
* CUDA path: ggml_cuda_init succeeded, weights loaded onto GPU per
s2's logs, but nvidia-smi showed 0% utilization during synthesis.
Wall time 20s/long phrase vs fish-s2's 7.5s. The "CUDA get_rows
unsupported for type q6_K" warning hints at incomplete op coverage
in s2.cpp's alpha CUDA backend for fish-speech architecture.
* Vulkan path: vk::IncompatibleDriverError on container init. NVIDIA
Vulkan ICD not accessible inside the container despite
NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics. Would need
host-side nvidia-utils-vulkan installation or manual ICD bind
mount. Didn't pursue.
Both are fixable — CUDA needs op coverage upstream (author actively
working on it; "selective embedding dequant" commit landed 16 days
ago), Vulkan needs host-side ICD setup. Neither is a config-flip,
both are real work for marginal-or-zero return. Better to delete the
stack and revisit when s2.cpp matures or when we tackle FP8
quantization on ana-ml2's RTX 6000 Ada (sm_89, native FP8 hardware).
Local image rmi'd, /opt/docker/compose/fish-cpp removed on irv-ml1.
/worktank/fish-cpp left for user-side sudo cleanup.
Future Fish acceleration paths (in order of decreasing certainty):
1. Wait for s2.cpp CUDA op coverage to mature (track upstream commits).
2. Quantize Fish BF16 → FP8 via TransformerEngine, deploy on
ana-ml2's RTX 6000 Ada (Ada has native FP8 tensor cores, A6000
doesn't). ~2x speedup if it works.
3. vLLM port of Fish (no upstream support today).
Subprocess-per-request architecture forced CUDA + model load on every
/v1/tts call (~10-20s init, then 5-15s generation). Even though CUDA
is now actually being used (`-c 0` fix landed), 32s for "Verify."
proved per-request init was the bottleneck.
s2.cpp ships a built-in HTTP server (`--server -H -P`) that keeps the
model resident on the GPU. Refactor:
* entrypoint.sh — backgrounds `s2 --server -P 3030 -c 0 -m ... -t ...`,
waits for it to bind 3030, then foregrounds uvicorn. tini supervises
via `wait -n` so either child dying takes down the container.
* server.py — drops subprocess.run; instead httpx-POSTs Fish-shaped
/v1/tts JSON to s2's localhost:3030/generate (multipart form: text
+ optional prompt_text/prompt_audio for cloning). Model load + CUDA
init now happen once at container start, not per-request.
* Dockerfile — added httpx (shim dep), curl (entrypoint readiness
probe), and the entrypoint.sh COPY+chmod. CMD now invokes
entrypoint.sh instead of uvicorn directly.
* deploy-fish-cpp.yaml — uploads entrypoint.sh alongside server.py.
Two issues from the first deploy attempt:
1) Build failure (real): linker errors on s2.cpp's CUDA build —
undefined references to cuMemSetAccess, cuDeviceGet, etc. These
are CUDA Driver API symbols (in libcuda.so), not Runtime API
(libcudart.so). The driver lib is provided by NVIDIA's container
runtime at RUN time, not BUILD time.
Fix: nvidia/cuda:devel images ship a stubs library at
/usr/local/cuda/lib64/stubs/libcuda.so that provides the symbols
for linking but is non-runnable. Adding that path via
LIBRARY_PATH + CMAKE_LIBRARY_PATH lets the linker resolve while
leaving runtime unchanged (real libcuda.so comes from the
driver mount).
2) Verify false positive: the /v1/tts verify step's last command was
`rm -f "$out"` — which always exits 0. This made the shell's
final exit code 0 regardless of whether curl/file/grep succeeded,
so verify reported OK even when nothing was running on host_port.
Fix: `set -e` at top + trap-based cleanup. Failures now propagate;
the rm still runs on either path via EXIT trap.
New stack scaffolding for the Fish quantized-realtime experiment. Not
deployed yet — this commit lands the canonical files; deploy follows.
Architecture decisions made in Phase 1:
* CUDA backend, NOT Vulkan. s2.cpp's CMakeLists exposes both
-DS2_VULKAN and -DS2_CUDA; the most recent upstream commit
(2026-04-12) was specifically about CUDA improvements, and CUDA
on the A6000 will be substantially faster than Vulkan for ML
matmul. -DS2_CUDA=ON in the Dockerfile build args.
* Pinned to s2.cpp commit e48ce8e02d8335bd9a0ba94679f605724b31d12
(2026-04-12 HEAD of main). Repo is alpha software per README;
pin tightly so future churn doesn't break our build. Bump
deliberately when wanting upstream improvements.
* Multi-stage Dockerfile: nvidia/cuda:12.6.0-devel for build (needs
CMake + ninja + git + the CUDA toolchain) → nvidia/cuda:12.6.0-runtime
for serve (slimmer; just the s2 binary + GGML libs + a small Python
shim). Cuts image size by ~50% vs single-stage devel.
* FastAPI shim (server.py) wraps s2.cpp CLI in Fish's `/v1/tts`
contract so the same bench harness + clients work against fish-cpp
with no changes. Per-request flow: decode optional reference WAV
from base64 → write to temp → subprocess.run the s2 binary → stream
resulting WAV back. Adds ~50-100ms per-request fork+exec overhead;
negligible vs the multi-second generation cost.
* `streaming: true` accepted in request body but IGNORED — s2.cpp
writes a complete WAV before returning, so chunked output isn't
available. Unlike fish-s2 (HF wrapper) where streaming drops TTFB
to 26ms, fish-cpp's TTFB ≈ total wall time. Speed depends entirely
on raw generation throughput.
* q6_k as default quant — sweet spot per typical GGUF guidance:
near-bf16 quality at ~5GB. Other variants (q4_k_m, q5_k_m, q8_0,
f16) selectable via FISH_CPP_MODEL env.
* Pinned to GPU 1 (A6000) by default to share with fish-s2 for
direct A/B benching. q6_k weights ~5GB + runtime ~3GB ≈ 8GB —
comfortable on either GPU.
* Port 8199 (next free in the irv-ml1 TTS slate).
Phase 2 (next) is the actual deploy + first build. Reserved 30-45 min
for cold-cache build + weights pull.
Three fixes from the second-wave deploy attempts:
* voxtral: vllm/vllm-omni doesn't publish a `latest` tag — pull
failed with "manifest unknown". Pinned VOXTRAL_VLLM_TAG to v0.18.0
(released 2026-03-29, the day after the Voxtral 4B TTS release —
first cut with Voxtral support).
* kyutai-tts: NillPointer wrapper exposes ONLY /health (root) and
POST /v1/audio/speech. No /v1/models, no /v1/audio/voices —
those return 404. Verified by /openapi.json against the live
container. Compose healthcheck + playbook wait + verify steps
all repointed at the actual paths. POST /v1/audio/speech is now
smoke-tested with a RIFF WAV assertion (same pattern as fish-s2).
* fish-s2: added FISH_S2_MODEL env var so the model variant is
swappable via .env without rebuilding. Both s2-pro (default) and
s1-mini are pre-pulled into the bind-mount; LLAMA_CHECKPOINT_PATH
+ DECODER_CHECKPOINT_PATH now use ${FISH_S2_MODEL:-s2-pro}.
s1-mini was originally gated on fishaudio's HF org (401), but
niobures/OpenAudio-S1 mirrors the same files openly — pulled
from there via a one-shot snapshot_download.
After getting fish-s2 finally healthy on attempt #5, the playbook's
verify still failed because /v1/audio/voices doesn't exist. Discovery:
the Fish wrapper has a custom API surface, not OpenAI-compatible.
Real endpoints:
POST /v1/tts — synthesis (text body, optional `references`
field for voice cloning, returns audio/wav)
GET /v1/health — liveness (used by Docker healthcheck)
GET /heartbeat — alternate liveness signal
GET / — Swagger Editor UI for the OpenAPI spec
No /v1/audio/speech, /v1/audio/voices, /v1/models — those return 404.
Updated:
* Playbook verify — replaced the JSON-shape /v1/audio/voices check
with a POST /v1/tts smoke that asserts a real RIFF WAV comes back.
* README API section — replaced the OpenAI-compat examples with
Fish's actual {"text":"...","references":[...]} body shape.
* README disk footprint — corrected ~9 GB → ~11 GB (codec.pth was
larger than I estimated; 1.9 GB + 9 GB safetensors).
* README Lessons learned section — recorded the 5-iteration deploy
story so the next time we touch a Fish-style upstream we don't
re-walk the dockerfile / target / pre-pull / API-shape traps.
Third deploy attempt got past the build but crashlooped at container
start: Fish's start_server.sh validates checkpoints/s2-pro/ exists
and exits cleanly (rc=0) if missing — no auto-download, no helpful
message. /worktank/fish-s2/checkpoints/ was empty, so the container
exited every ~52s under restart policy.
Added an idempotent pre-pull step using the same one-shot
python:3.12-slim + huggingface_hub.snapshot_download + hf_transfer
pattern we used for the Qwen 3.6 GGUFs earlier today. Pulls the 9
relevant files (~11 GB total: codec.pth + 2 safetensors shards +
config + tokenizer/template) directly into the bind-mount at
/worktank/fish-s2/checkpoints/s2-pro/ — gated by `creates:` on
codec.pth so the pre-pull step is a no-op on reruns.
~83 s wall-clock for the 11 GB pull on first deploy.
vh