Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
Profiling the index-tts deploy log (2057 lines) showed ~25% was just
pip's per-package Downloading / Collecting / Requirement-already /
progress-bar spam — useless for ops, hard to scan when something
actually breaks.
Three changes across the four TTS deploy playbooks:
1. Pulls (Kokoro): add --quiet. 6.5 GB pull no longer floods the log
with per-layer progress redraws. Final "X Pulled" still prints.
2. Builds (VibeVoice, Chatterbox, IndexTTS-2): add --progress=plain
to stop the BuildKit TUI from littering the captured log with
carriage-return overdraws, then pipe through a grep filter that
drops pip's noisy lines but keeps:
- buildkit step transitions (#NN [stage])
- DONE / CACHED / ERROR markers
- apt + build-stage messages
set -o pipefail keeps a real build failure from being swallowed
by the grep's exit code.
Net effect: ~25% smaller logs, much more scannable; full visibility
into step progress and errors preserved.
"docker compose pull (first run: ~6.5 GB from GHCR)" had an unquoted
colon-space inside a plain scalar value, which YAML parses as a
nested mapping — elway aborted on load. Single-line fix: wrap the
value in double quotes.
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
IndexTTS-2's tts.infer(stream_return=True) is a generator that yields
audio chunks per text segment as they finish, plus inter-segment
silence. Expose this via the existing POST /v1/audio/speech with a new
"stream": true field on the request body.
Wire-up:
- 44-byte WAV header emitted up front with placeholder data length
(0xFFFFFFFF) so chunks can be written before total samples are
known. Players that read until EOF (mpv, ffplay, aplay, sox,
browsers via <audio>) handle this fine.
- Each yielded chunk goes through _chunk_to_pcm_bytes(), which
handles torch tensors / numpy arrays in either int16 or float
(-1..1) form.
- 22050 Hz mono int16 — IndexTTS-2's hardcoded output shape.
Time-to-first-audio drops from full-file latency to ~one-segment
latency. Single-sentence inputs barely benefit; long passages /
multi-paragraph reads benefit a lot. Strict metadata parsers may
balk at the placeholder size — request without stream for a
closed-length WAV in that case.
INDEX_TTS_TAG bumped to v2 to force a rebuild.
budget hazard + media-CDN workaround
Tried adding git-lfs install + git lfs pull to the build to get
real example WAVs into the image — failed with:
Error downloading object: examples/emo_hate.wav: Smudge error:
batch response: This repository exceeded its LFS budget. The
account responsible for the budget should increase it to
restore access.
The index-tts org's LFS bandwidth quota is exhausted upstream and
out of our control. Reverting the Dockerfile change. The examples
aren't needed for the wrapper to work; emotion_text and
emotion_vector are sufficient for end-to-end testing without any
WAV file at all.
For users who want the bundled example clips as starter audio,
README now documents the media-CDN URL trick — same LFS objects
served via a different code path that doesn't count against the
LFS API budget. INDEX_TTS_TAG stays at v1.
The IndexTTS-2 repo stores examples/emo_*.wav and examples/voice_*.wav
as Git LFS objects. v1 of our image cloned the repo without an LFS
pull, leaving those paths as ~130-byte pointer text files — unusable
for `docker cp` into /worktank/index-tts/{voices,emotions}/ as starter
references. (Caught when an emotion_voice="hate" call returned audio
that was actually the pointer text round-tripped through file IO.)
v2 adds git-lfs to the apt list, calls `git lfs install --system`
once, and `git lfs pull` after the checkout. Adds ~1-2 MB to the
image (the examples are small audio clips). INDEX_TTS_TAG bumped to
v2 to force a clean rebuild.
Adds a third TTS to the irv-ml1 fleet. IndexTTS-2 is Bilibili's
emotion-controllable zero-shot TTS (paper 2506.21619). Distinguishing
capability vs the existing two: timbre and emotion are disentangled —
clone a voice's timbre from one reference and the emotion from a
different reference, OR set emotion via 8-vector, OR derive it from a
text description. Neither CosyVoice 3 nor Qwen3-TTS-1.7B-Base does
this cleanly in English.
Wrapper is owned end-to-end (~150 lines in app.py) — the only existing
FastAPI fork (csllpr/index-tts-fastapi) targets v1 and is a dormant
single-commit repo. Upstream IndexTTS-2 ships only a Gradio webui.
Layout follows the qwen3-tts pattern:
stacks/index-tts/
Dockerfile — CUDA 12.8 base, IndexTTS pinned to a SHA
app.py — FastAPI: POST /v1/audio/speech + /v1/voices
entrypoint.sh — one-time HF snapshot_download of the weights
compose.yaml — env-driven, GPU pinning support, bind mounts
.env.example — port 8192, fp16, paths
README.md — API examples + comparison vs the other TTS
playbooks/deploy-index-tts.yaml — elway playbook for irv-ml1
Voice and emotion libraries are flat host dirs of WAVs, bind-mounted.
Drop a new <name>.wav and /v1/voices picks it up immediately.
License caveat: IndexTTS-2 weights ship under a custom Bilibili
license (free at our scale, not OSI-open). README documents it.
Toggles --append-only off on the rest-server via a temporary
docker-compose.override.yaml (canonical compose untouched), runs
resticprofile forget --prune --verbose on each client of that
rest-server, then restores --append-only. The restore is wrapped in
a trap so a partial-failure prune still leaves the rest-server in
its safe configuration.
ANA side is fully automated against ana-docker (5 clients:
ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db).
NH3 side currently prints a manual DSM ceremony — Synology Container
Manager doesn't expose docker on the expected paths and syncuser
sudo isn't NOPASSWD, so the toggle isn't safely scriptable from
this workstation. The instructions cover the same flow in DSM web
UI + interactive ssh on each NH3 client (nh3-docker, nh3-dev,
irv-ml1).
Usage:
scripts/restic-prune.sh ana # ANA only (auto)
scripts/restic-prune.sh nh3 # NH3 instructions
scripts/restic-prune.sh all # both
scripts/restic-prune.sh -h # help
scripts/restic-prune.sh --dry-run ana # show every command
- STATUS.md: marks items 11 (mattermost dir cleanup verified gone),
13 (UniFi UDM cards added to homepage), 15 (docs first + second
pass — README + chromadb-setup deletion + VM-102 frontmatter
strip), and 16 (drift discipline rule, no auto-regen) as done.
New session-milestones section captures the four task-board
iterations (v0.1.1 session disambiguation + dormant-timer fix,
v0.1.2 favicon + 1s tick rate, v0.1.3 case-insensitive session
names with real ALTER migration), the parakeet healthcheck fix,
the AIPA-MCP session relabel + DB rename to Architect, the
homepage Toolchain dedup, and the qwen3-tts variant flip.
- .gitignore: ignore .claude/scheduled_tasks.lock and any other
.claude/*.lock — runtime per-machine state, not interesting to
cross-checkout.
The task-board compose carries homepage.group=Toolchain. With no
matching entry in settings.yaml's layout: map, homepage placed it
on the default tab (Main) AND it appeared under the Toolchain tab,
producing duplicate cards. Declare the Toolchain group explicitly
with tab: Toolchain so it renders in exactly one place.
Second pass on docs/ cleanup (item #15 in STATUS.md):
- pfi/chromadb-setup.md: deleted. References configs/pfi-ana/... and
scripts/setup-chromadb.sh, neither of which exist in this repo
(artifacts of an earlier project layout). ChromaDB is already live
per docker-stack.md; the operational truth lives there.
- pfi/docker-stack.md: removed the cross-link to the deleted file and
pulled the bit of useful content from it (auth-token generation +
client Settings example) into the inline ChromaDB section.
- pfi/vm-102-matrix-{synapse,appservice}.md: stripped the YAML
frontmatter. The `path:` values pointed at docs/pfi-ana/... which
doesn't exist in this repo; no toolchain consumed the metadata.
- README.md: tree updated to reflect the deletion.
VM-102 docs kept separate by design — each is right-sized; merging
would push past the ~500-line guideline.
- parakeet/compose.yaml: healthcheck was using curl which isn't in the
image (only wget is, via apt). 2,190 failing checks — switched to
`wget -q -O /dev/null`, container went healthy on recreate.
- qwen3-tts/.env.example: variant annotation was reversed. The upstream
wrapper's runtime error is unambiguous: voice cloning requires the
-Base variant, not -CustomVoice. Corrected the comment block and
flipped the default to Qwen/Qwen3-TTS-12Hz-1.7B-Base.
- qwen3-tts/README.md: 0.6B switch snippet now suffixes -Base too,
since plain `Qwen/Qwen3-TTS-12Hz-0.6B` isn't published on HF.
Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
- services.yaml: PFI-UDMSE (10.100.0.1, NH3 edge) under Infra - NH3,
ESH-UDMPM (10.0.0.1) under Infra - ESH. PFI-UDMSE replaces the
retired Fortigate 101F at NH3 — comment updated. Both use si-ubiquiti.
Diff also folds in the previously-deployed-but-uncommitted PBS-ANA /
PBS-NH3 / IRV section / retired NH3-SW1 cleanup, bringing the tracked
copy in sync with what's live on esh-docker-vm.
- docs/README.md: navigation map of the docs/ tree (orientation /
runbooks / pfi) with what-goes-where conventions.
- docs/runbooks/tea-0.14.0-linux-amd64.sha256: deleted (build artifact,
not a runbook).
- STATUS.md: items 11 / 13 / 15 marked done; 15 leaves Matrix and
chromadb consolidation candidates as future focused-session work.
Three fixes from the first deploy attempt on irv-ml1:
- build.target=production. Upstream Dockerfile is multistage; the last
stage `cpu-base` was selected by default, producing a CPU-only image
with no flash-attn and `torch ... whl/cpu`.
- user: "0:0". Upstream image declares USER appuser but writes runtime
state under /root (mode 0700). appuser cannot traverse /root, so
/v1/voices 500s on PermissionError. Run as root to sidestep.
- QWEN3_TTS_MODEL=Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice. The bare
`1.7B` id we had isn't a real HF identifier; upstream publishes
-CustomVoice / -Base variants of each size. Use -CustomVoice so
`voice="clone:<name>"` works.
Tag bumped to v2 to keep the v1 cpu image distinguishable in the local
registry.
After: all 5 verify steps pass, GPU synthesis ~5s for 3-4s of audio,
three contrasting English `instructions` produce three distinct
hashes — emotion steering actually works (unlike CosyVoice's English
path).
Avoids the "paste the full output" friction. Every elway run now
writes its full streamed output to two files in addition to the
terminal:
/tmp/elway-last.log
always overwritten — the easy "what just happened" target
~/.cache/elway/runs/<ts>-<host>-<playbook>.log
timestamped permanent record; accumulates across runs
Implementation: small _Tee class wraps sys.stdout for the duration
of main() so all `print(...)` calls fan out to the real terminal +
both file handles. Subprocess output already goes through print()
via _stream_process, so the build/healthz/etc. text is captured.
ANSI color codes are kept in the file so colors are preserved in
log viewers that handle them (less -R, modern tail). Strip with
`sed 's/\x1b\[[0-9;]*m//g'` for paste-elsewhere.
New flags:
--log <path> override path; replaces both default destinations
--no-log terminal-only, skip both files
Path of the permanent log is printed at the top of every run so
you know where it landed without remembering the timestamp pattern.
Alibaba's open-weight TTS (Apache 2.0, Jan 2026), deployed via
groxaxo/Qwen3-TTS-Openai-Fastapi wrapper. Built locally from a
pinned git SHA via docker buildx's git context — no source
vendored. 1.7B flagship model by default; 0.6B available via
QWEN3_TTS_MODEL env override.
Why we need a second TTS stack: cosyvoice 3 emits Chinese phonemes
for English content per upstream FunAudioLLM/CosyVoice#1790
(unfixed). Qwen3-TTS is from the same Alibaba team but with
English first-class in the checkpoint — 10 languages, 97 ms
streaming TTFB, instruction-driven emotion. Coexists with cosyvoice
on irv-ml1 (port 8191; cosyvoice keeps 8190).
Voice cloning shape DIFFERS from cosyvoice: profile-based, not
voice-id. Profiles live under voice_library/profiles/<name>/ and
are referenced as voice="clone:<name>".
Path layout: /worktank/qwen3-tts/{cache,voices}/, with cache excluded
from restic (regenerable from HF Hub) and voices included (cloned
profiles need original reference audio to recreate).
playbooks/deploy-qwen3-tts.yaml: 10 steps + 5 verify, idempotent;
the wait step polls /health for up to ~10 min to absorb first-run
model download.
Stack only — restic profile update for /worktank/qwen3-tts/voices/
to follow when this is empirically validated against the GLaDOS
voice (the "did Qwen inherit the Chinese-bias bug?" question).
Verified empirically (gitea #7): the neosun wrapper's \`instruct\`
field routes through CosyVoice's inference_instruct2 which expects
a Chinese template. English directives produce Chinese-phonemed
speech of the directive itself — the model is stuck in Chinese
context regardless of the directive language.
XML inline tags (<angry>, <sad>, etc.) bypass that path entirely
and work cleanly for English emotion control.
README updated:
- Added Gotcha section entry explaining the instruct trap with
the specific failure mode and the byte-identical-response
evidence
- Smoke-test recipes (steps 4 + 5) reworked to use <angry>...</angry>
in the input instead of an English instruct value
- "Emotion / style control" table now marks instruct as Chinese-
only with ⚠️ and XML tags as ✅
STATUS.md:
- Mark 4b done (both Postgres migration + NFS decoupling)
- Add arch decisions for gitea remote + prefer-elway policy
- Add tooling entries for elway + tea CLI
- Document 2026-04-24 session milestones (irv-ml1 AI stacks,
elway, task-board, 4b finish)
- Expand memory-pointer list with the files added this session
CLAUDE.md:
- Tell new sessions to use elway for SSH-driven work, point at
the smoke playbook template
- Document the task-board plugin + MCP-tool contract so assistant
sessions with the plugin enabled know the assistant should call
task_start / task_update / task_wait / task_complete at
meaningful checkpoints
.claude/settings.json:
- Project-level env: TASK_BOARD_SESSION=Infra so every Claude Code
session opened here labels its task-board cards "Infra"
playbooks/decouple-pfi-postgres-from-ana-nas.yaml:
- Finishes the DB-off-NFS migration on pfi-postgres. Already ran
against prod today; fstab clean, unmounted, no systemd mnt-db
unit. Verify 3 was mis-expressed on first run (`grep -q active`
matched "inactive") — fixed to invert systemctl exit code
directly.
Previously the startup logic ran `probe_sudo()` if any step in the
playbook declared `sudo: true`, regardless of whether that step's
creates:/when: gates would actually let it fire. The result on the
task-board deploy rerun was a spurious password prompt followed by
six SKIPPED lines — the prompt served no purpose.
New flow:
- Remove the upfront probe in main().
- SSHContext.sudo_password defaults to None; new sudo_probed flag
tracks whether we've already prompted this session.
- run_shell_step + run_upload_step call ensure_sudo(ctx) only at
the point a sudo step is actually executing — i.e. after its
skip conditions have been evaluated and passed. Idempotent:
probes at most once per playbook run.
Tradeoff accepted: if the user fat-fingers the password, they see it
mid-run on the first sudo step rather than upfront. `stop_on_fail`
(default true) halts cleanly; they rerun. Lower friction for the
common idempotent-rerun case, same recoverability.
Verified against playbooks/deploy-task-board.yaml — prior run
prompted + completed in 1.7s; new run completes in 1.7s with no
prompt because every sudo step skip-gated.
stacks/task-board/compose.yaml + .env.example describe the runtime —
image tag is task-board:local (built on the host), 7878 on host maps
to 7878 in container, SQLite lives at /opt/docker/conf/task-board/data/
(bind mount, uid 1000 friendly), homepage auto-card labels under
Toolchain group, on traefik-net like the rest of the fleet.
playbooks/deploy-task-board.yaml is the first real elway playbook —
exercises everything we built tier-1 + tier-2 idempotency for:
- `creates:` on mkdir + first-time clone + compose dir + data dir
- `when:` to chown /opt/docker/build only if it came up root-owned
- `when:` to seed .env only if one doesn't already exist (never
clobbers user edits on rerun)
- `changed_when:` on the `git reset --hard` step so repeat runs
against the same ref report `ok` instead of `changed`
- `changed_when: "false"` on every verify step (they attest, not
change)
- `upload:` with mode for compose.yaml + .env
Post-up the playbook polls /api/health for 30s before handing off to
the verify phase, so verification doesn't race the healthcheck's
start_period. Verify covers: /api/health 200, /api/tasks shape, /mcp
reachable, container on traefik-net.
Prereqs documented in the playbook header: Docker + compose plugin,
traefik-net network, git SSH access to gitea from the target host.
Tier 1 — pre-step skip conditions:
when: <remote shell expr> skip unless expr exits 0
creates: <remote path> skip if path already exists
removes: <remote path> skip if path is already absent
Any of the three saying "skip" marks the step `skipped` and moves on.
Evaluated under bash -c on the remote so `!`, `[[`, pipes etc. behave
consistently regardless of the default remote shell.
Tier 2 — post-step change detection:
changed_when: <remote shell expr>
Evaluated after a successful step. Exit 0 → step counts as `changed`
(default). Exit != 0 → `ok` (ran, nothing actually different).
Without this field, successful steps default to `changed`, matching
Ansible's shell/command defaults. Useful on verify steps:
`changed_when: "false"` reports them as `ok` since they only attest.
Status model moved from pass/fail to four states:
ok / changed / failed / skipped
Summary reports each count; overall outcome is CHANGED if any step
changed, OK if none did, FAILED on any non-skipped failure.
Rerunnable smoke: playbooks/elway-smoke.yaml now proves it. On a
clean target the cold run reports 4 changed, 3 ok. Rerunning with
the same vars reports 2 skipped / 2 changed (upload + log-record
have no idempotency hooks and are always `changed`). Overriding
--var greeting=... re-runs the gated step exactly as intended.
Doc block at the top of the script updated with the new schema
fields and state machine.
`scripts/elway` is a ~600-line Python tool (stdlib + python3-yaml) for
driving one-off ssh commands, ad-hoc file uploads, and YAML playbooks
against a single host. Fills the gap between "single ssh one-liner"
and "reach for Ansible."
Highlights:
- Three invocation modes: --shell, --upload (LOCAL:REMOTE[:MODE]),
and --playbook <path>
- Playbook schema: inline vars, list of steps, optional verify block.
Template via {{ var }}; CLI --var overrides inline defaults
- stop_on_fail global (default on), per-step override. Verify phase
always runs, even after a halt — you see end-state regardless
- Sudo handled once: probes NOPASSWD; if not, prompts locally via
getpass, validates up-front, then feeds via `sudo -S` per step.
Password never written to disk/logs. Upload-with-sudo stages to
/tmp then sudo-mv + sudo-chmod
- SSH connection reuse via ControlMaster (60s persist) keeps
multi-step playbooks responsive (~30ms/step reuse vs ~550ms cold)
- Live interleaved stdout/stderr with per-step prefix and colored
pass/fail summary. --dry-run prints the plan without executing
- Shebang pinned to /usr/bin/python3 to bypass venv-shadowing
when python3-yaml lives in the system site-packages
Smoke test (playbooks/elway-smoke.yaml) covers vars + upload + verify;
drove out a YAML-scalar-coercion bug before first commit (`shell: false`
parsed to Python bool, crashed the templater — now coerced to string
at load time with a clear error on nulls).
The original smoke-test curl used voice="default" which doesn't
exist — the neosun wrapper ships zero preset voices, and the
built-in SFT speakers (中文男/女 etc.) are not surfaced. Calling
/v1/audio/speech with any unregistered voice returns a 400 whose
JSON body curl happily writes into the .wav (124-byte phantom).
Replaced the smoke test with the full clone → synthesize flow and
added a gotchas section covering:
- No default voice; /v1/voices/create is mandatory
- Reference audio ≤30s (frontend asserts; longer clips 500 at
synthesis time, not at upload)
- Providing an explicit transcript beats the auto-ASR fallback
- voice_id is the handle, not name
- Both cosyvoice-v3 and cosyvoice-v2 ship in the image
Also documented streaming: available via /api/tts with stream=true
(~150ms TTFB), NOT on /v1/audio/speech. Clarified field-name
differences between the OpenAI-compat and native endpoints in a
table. No WebSocket / SSE in this wrapper despite upstream support.
The Shadowfita FastAPI wrapper hit two unfixed upstream bugs on the
first real /transcribe call — chunker return-shape mismatch (open
issue #16) and a `torchaudio.tensor` that doesn't exist (open #10).
Rather than babysit someone else's half-tested code, switched to
sherpa-onnx with the prebuilt int8 Parakeet-TDT tarball from k2-fsa,
and wrote our own ~60-line FastAPI wrapper.
Moving parts now owned in-tree:
Dockerfile CUDA 12.8 + cuDNN 9 runtime base, installs
sherpa-onnx==1.12.39+cuda12.cudnn9 + fastapi +
soundfile + libasound2 (sherpa-onnx links to ALSA
at load time even when we never touch a mic).
app.py OfflineRecognizer.from_transducer() once at startup;
/transcribe and /v1/audio/transcriptions both accept
multipart uploads and return {"text": ...}.
entrypoint.sh Idempotent model download to /models on first run
(~400 MB int8 tarball), then exec uvicorn.
Smoke test: 0.wav (bundled in the tarball, The House of the Seven
Gables excerpt) transcribes cleanly in ~1.2s on GPU.
PARAKEET_MODEL_URL in .env lets you swap to the v3 (25-language)
tarball without touching any other files. Wipe *.onnx + tokens.txt
from the models dir and the entrypoint re-downloads.
Selectively include /worktank/<stack>/ subtrees now that comfyui,
parakeet, and cosyvoice place real user state there. Bulk weights,
scratch outputs, and the ~8 GB disposable comfyui run/ venv stay
out — both by the include list being precise and by belt-and-
suspenders exclude patterns.
Added sources:
/worktank/comfyui/basedir/user workflows + per-user settings
/worktank/comfyui/basedir/custom_nodes hand-installed extensions
/worktank/comfyui/basedir/input user-uploaded source images
/worktank/cosyvoice/voices cloned speaker profiles
Belt-and-suspenders excludes (inert under current sources; guards
against a future wholesale-add of /worktank):
/worktank/comfyui/basedir/{models,output,temp}
/worktank/comfyui/run
/worktank/parakeet/models
/worktank/cosyvoice/{input,output}
Verified by `resticprofile backup --dry-run` on irv-ml1 — 1.5 GiB
scanned across all 9 sources, 13 MiB new to the repo. If any bulk
dir had leaked in, the total would be multi-GB.
Also fixed a stale /home row in the README (profile only sources
/home/lkraven; llmuser + sduser are explicitly excluded).
Two new speech stacks on irv-ml1, both on the /worktank/<stack>/
pattern, no tnet (irv-ml1 is local-endpoints-only for now).
parakeet — ASR via Shadowfita/parakeet-tdt-0.6b-v2-fastapi:
- docker buildx git context pinned to SHA 31c5652; no source
vendored. Rebuild on SHA bump.
- GPU-capable FastAPI + Silero VAD + WS streaming.
- API: POST /transcribe, WS /ws/transcribe, GET /healthz. Not the
literal OpenAI `/v1/audio/transcriptions` path — note in README.
- HF cache at /worktank/parakeet/models/ (excluded from restic).
- Build ~158s first time; steady-state start ~40s.
cosyvoice — TTS via neosun/cosyvoice:v1.3.2 shipping
Fun-CosyVoice3-0.5B-2512 (CosyVoice 3, chosen over v2 for the
expanded 5,000-hour instruction-following data covering emotions,
speed, tones, dialects, accents, role-playing; ~150ms streaming
TTFB matches v2). API: /v1/audio/speech (OpenAI drop-in),
/v1/voices/create (cloning), /health.
- Host port 8190 (container 8188; host 8188 already taken by comfyui).
- /worktank/cosyvoice/{voices,input,output}/; voices include in
restic (precious — reproducing a clone needs the original ref
audio), input+output excluded (scratch).
- Model weights (~2-3 GB) live inside image layer; re-download on
tag bump, persist across `compose up -d`.
Both healthy on first deploy.
New stack mirroring PFI convention (stacks/comfyui/) using
mmartial/comfyui-nvidia-docker:ubuntu24_cuda12.8-20260312. Both GPUs
exposed, pinned to CUDA 12.8 to match the host's 570.x driver and the
native cuda-toolkit already in place.
Layout — single tree under /worktank/comfyui/ (462G dedicated, 1%
used pre-deploy):
- basedir/ → /basedir user state (models, workflows, custom_nodes,
input, output); owned 1000:1000 so external
tools can edit workflow JSON directly.
- run/ → /comfy/mnt ComfyUI source + venv + pip cache (~7.8G
after bootstrap). Bind mount instead of
named volume — the image refuses to chown
mounted paths at startup, so keeping this
lkraven-owned avoids the sudo dance.
servers/irv-ml1/README.md refreshed: Docker upgraded to 29.4.1 with
traefik-net in place; dockge + beszel-agent + dozzle-agent already
present; /storetank dropped 92% → 64%; restic coverage to
rest-server-nh3 is operational (not "currently none" as prior text).
VMIDs aren't globally unique across PVE hosts (esh-pve and sfsrv-ana
both have VM 100, etc.). Without namespaces, per-hypervisor backups
collide under the same /vm/<vmid>/ path in the shared datastore.
Changes:
- New Phase 1.4b: create one namespace per hypervisor up front
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana).
- Phase 2.1 storage-entry template now lists Namespace as a required
field, set to the hypervisor's own name.
- Critical-note explaining the collision risk so future deployers
don't skip this step.
ACL grants remain at the datastore level; they apply across all child
namespaces so the existing fleet-vzdump token continues to work. Sync
job (Phase 6) preserves namespace tree to PBS-NH3 automatically.
Discovered during Phase 1.3 (mount NAS datastore). The Debian NAS's
/mnt/backup is on ZFS with casesensitivity=insensitive; NFSv4 writes
fail with EACCES even for root with no_root_squash. Known-bad
combination at the ZFS-on-Linux + NFSv4 layer.
Workaround: mount the NFS share with vers=3. PBS's chunk-based
access pattern works fine over NFSv3.
Added a section 0.5 to Phase 0 documenting the issue + both fixes
(quick: use NFSv3; cleaner: create case-sensitive child dataset).
Future deployments against this NAS won't rediscover the same wall.
Adds a new `pinned` group with swap: false (models coexist in VRAM),
exclusive: false (group shares with other groups), persistent: true
(never unload). Each member also gets ttl: 0 so the per-model
idle-timeout can't drop them either — belt + suspenders.
Pair is currently qwen3.5-9b (~6 GB Q4) + qwen3.6-35-a3b (~29 GB Q6).
Plus the 128K KV caches, roughly 50-60 GB VRAM resident. Appropriate
for an A6000/H100-class card; verify fit after deploy.
Committed as a canonical change; push + restart still needed on
ana-ml2.
Current NFS exports on the Debian NAS (10.250.50.50) use root_squash,
which blocks PBS from writing its datastore metadata (chunks, locks,
GC state — all root-owned operations). Rest-server-ana worked around
this by running its container as UID 1000, but PBS's service model
doesn't accommodate that pattern cleanly.
Solution baked into Phase 0: create a dedicated NFS export for the
PBS-ANA datastore subtree, scoped to only the PBS-ANA VM's IP, with
no_root_squash. Bounded exposure (single client), kept in a separate
.exports file so Cockpit's File Sharing module doesn't clobber it.
Flag world-scoped export on /mnt/pve-VMStorage as a non-blocking
hygiene item for a later Cockpit pass.
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.
Files:
configs/restic/ana-docker/profiles.yaml
configs/restic/ana-ml2/profiles.yaml
configs/restic/nh3-docker/profiles.yaml
configs/restic/esh-docker-vm/profiles.yaml
configs/restic/vm-esh-nas/profiles.yaml
configs/restic/nh3-dev/profiles.yaml
Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.
STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
BatchMode=yes (which the scripts set) implies strict host key checking
and refuses to prompt — so first-time SSH to a host that isn't in
known_hosts fails with "Host key verification failed". Every new host
we register needs a manual `ssh <host>` round-trip first to store the
key before the refresh scripts can reach it.
accept-new fixes that: unknown hosts are auto-accepted into
known_hosts on first connect; subsequent key CHANGES still fail loudly
(as they should — that'd be a MITM signal).
Matches the pattern already used by deploy-stack.sh.
Affects only refresh-server-info.sh and refresh-proxmox-info.sh;
deploy-stack.sh + sync-stacks.sh use their own targets that may or may
not want the same treatment (leaving alone for now).
Initial framing was wrong. PFI runs these under a managed-hosting
agreement: SSH, OS ops, backups are all PFI's responsibility. Hardware
and data belong to the client.
Changes:
- ssh-target files added for sfsrv-ana (root@10.250.250.115 — same
pattern as other PVE nodes) and sf-ana-container
(lkraven@10.250.150.100 guess, adjust if different user).
- sf-r630 still lacks an ssh-target — the OS-side LAN IP isn't in
FortiGate DHCP (static config somewhere). Will fill in once
identified; README flags that gap.
- READMEs rewritten: dropped "tenant-scoped" / "not SSH-managed"
language, added "client context" section that explains the
managed-hosting relationship. Backup coverage now listed as
planned rather than blocked on tenant coordination.
- CLAUDE.md fleet table: SF rows re-labeled "SureFire client
(PFI-managed)". Placement-rules section updated to note that
SF hosts are first-class PFI-ops targets, just client-owned.
- Memory (project_surefire_tenant.md) rewritten to reflect
managed-services reality + hosts-file entries needed for name
resolution since these aren't in PFI DNS.
Snapshot of what's in place and what's outstanding as of end of
2026-04-20/21 session. Grouped by urgency (red/orange/yellow/green/blue)
so a glance tells you what's next regardless of who's picking it up.
Notable open items:
- Backrest esh-docker-vm URI mismatch (still pointed at NH3 Synology
instead of rest-server-ana)
- ssh-target verification on the 9 newly-added host entries
- Forget schedules need patching (fail nightly against --append-only)
- ~6 secrets captured in this session's transcripts need rotation
- SureFire tenant backup plan pending decision
Lists session milestones (homepage reorg, 6/6 restic coverage, discovery
scripts, CWA migration, 9 host registrations, etc.) and memory
pointers so future sessions have context without re-reading the full
chat log.
Six PFI VMs/LXCs previously known only via proxmox_inspect.sh —
covered by vzdump but not in servers/, so operational context
(roles, backup posture, ssh target) was missing:
pfi-ana-webhost (VMID 110) — web workload
ana-filebot (LXC 112) — file-task automation
pfi-pteradactyl (VMID 107) — Pterodactyl game panel
pfi-tacticalrmm (VMID 111) — TacticalRMM remote-management
pfi-postgres (VMID 105) — shared Postgres (vaultwarden/gitea/
paperless backends)
ana-wg (LXC 113) — WireGuard VPN gateway
Plus three SureFire tenant hosts at the Anaheim colo:
sfsrv-ana — tenant Proxmox hypervisor (10.250.250.115:8006)
sf-ana-container — container workload on that Proxmox
sf-r630 — physical R630 (iDRAC 10.250.250.110 for PFI-side
hardware mgmt; OS is tenant-scoped)
Each server dir has README + ssh-target where applicable. SureFire
entries explicitly document tenancy scope: PFI provides hosting,
SureFire owns the OS; management actions need tenant coordination.
SureFire hosts have no ssh-target by default.
Homepage Infra - ANA gains two new cards:
- SFsrv-ANA (https://10.250.250.115:8006, si-proxmox icon)
- SF-R630-iDRAC (https://10.250.250.110, si-dell icon)
PFI-ANA-ML2 BMC gained an href since it has a usable web UI.
CLAUDE.md fleet table extended with all 9 new rows. Placement-rules
section notes the SureFire tenant boundary.
Memory: new project_surefire_tenant.md so future sessions know sf-*
hosts are tenant-scoped by default.
First real run surfaced 31 gap rows, ~20 of which were noise. These
changes reduce the output to actionable signal.
1. discover-unifi: /ea/devices now filters out
- IPs outside the fleet LAN range (UDM's WAN IP appearing as a
"device", ISP uplink records with public IPs)
- UDM self-records (isConsole=true, or IP matches wans[].ipv4)
- UCI records (UniFi Cable Internet = ISP modem tracking)
LAN filter regex defaults to ^10\. (matches 10.0.0.0/8); override
via UNIFI_LAN_FILTER env var if you run other private ranges.
2. discover-gaps: new --ignore-unifi flag drops rows where the final
SOURCE column starts with "unifi:". Useful for "show me servery
things to manage, not the fleet's network hardware."
3. discover-gaps: known-IP set now pulls IPs from
servers/*/proxmox-details.txt AND servers/*/system-details.txt in
addition to README.md and ssh-target. Consequence: VMs tracked by
proxmox_inspect.sh are automatically counted as known without
needing a separate servers/<vmname>/ dir. Also strips meaningless
addresses (127.*, 0.0.0.0, 169.254.*) so they can't false-positive
a "known" match.
4. MAC normalization: both discover-fortigate and discover-unifi now
emit xx:xx:xx:xx:xx:xx lowercase. Previously FortiGate used colon
format, UniFi used no-separator uppercase — same MAC looked
different per source. Fortigate does tolower() in awk; UniFi uses
a shared jq `norm_mac` function.
Raw dumps of /ea/hosts and /ea/devices surfaced the actual JSON:
- /ea/hosts: LAN IP isn't at top-level ipAddress (that's WAN public);
it's buried in reportedState.ipAddrs[] mixed with WAN + link-local.
Have to pick the first RFC1918 entry that ISN'T also a WAN interface
IP (reportedState.wans[].ipv4). Name/mac/model all live under
reportedState.{hostname,mac,hardware.shortname}.
- /ea/devices: outer records are per-host wrappers; real AP/switch
records are in the nested `devices` array with top-level `ip`, `mac`,
`name`, `model` fields. Previous parser was reading the wrapper and
getting all `-`.
Reorder all TSV outputs so IP is column 1 — makes discover-gaps.sh
work uniformly against both FortiGate and UniFi sources. Sites TSV
dropped its IP slot since sites have no meaningful IP (metadata only).
Verified against the real payloads the user captured: ESH-UDMPM now
surfaces as 10.0.0.1 (LAN) instead of 192.168.200.111 (WAN2, RFC1918
but excluded via the wans cross-check). A sample device record
(E7-ESH-Media at 10.0.250.176) flattens correctly into a single TSV row.
Rewrite to use Ubiquiti's public cloud API at api.ui.com instead of
logging into individual controllers via session cookies. Benefits:
- One API key covers every UniFi OS device on the account (no
per-controller login logic, no cookie jar lifecycle).
- Read-only by design (auth keys are scoped).
- Works across sites transparently.
Three endpoints wired up: hosts (controllers / Cloud Keys), sites,
and devices (APs / switches). Each emits a distinct TSV shape so the
output can be concatenated and still parsed.
`all` mode runs all three and prints section markers on stderr so
the stdout stream stays clean TSV suitable for discover-gaps.sh.
Pagination handled via nextToken. Rate limit not enforced locally;
Ubiquiti documents generous defaults for read endpoints.
Note: Site Manager API (early access) doesn't appear to expose a
connected-client list directly. For endpoint discovery (IP + MAC of
connected clients like laptops, IoT, etc.) we'd still need to hit
each local controller's REST API — follow-up if the infrastructure-
level data isn't enough.
Requires: curl (present), jq (apt install jq).
Bug: FortiOS 7.x ana-gw replied to 'execute dhcp lease-list all' with
"Interface name 'all' does not exist." — my error-pattern grep didn't
include that phrase, so the script thought it got valid data, bailed
out of the retry loop, and handed empty/garbage to the parser, which
produced zero output with no error.
Fix: try the plain `execute dhcp lease-list` form first (works across
versions we've seen), fall back to the `all` variant only if the plain
form returns nothing. Validate acceptance by grepping for an actual
IP-shaped token — the parser needs IPs anyway, so "got real data"
and "has at least one IP" are equivalent conditions.
Device has been removed from the NH3 site. Drop the homepage card
and the corresponding example in discover-fortigate.sh.
Note left in services.yaml so whoever adds the replacement edge
device knows where the old entry lived.
First live run against ana-fw.phasefinal.com surfaced two bugs:
1. Script double-prefixed user@ when the arg already contained it
(e.g. `admin@10.250.250.1` became `admin@admin@10.250.250.1` →
auth prompt loop). Accept either "host" or "user@host" and only
prepend the default user if missing.
2. Parser assumed the wrong output format. Real FortiOS (tested on
7.x) emits:
<prompt> # <iface>
IP MAC-Address Hostname VCI SSID AP SERVER-ID Expiry
10.x.x.x ...
<next-iface>
IP MAC-Address ...
- Interface names are flush-left (no "Interface:" prefix)
- First line has the shell prompt embedded before the iface
- Hostnames don't contain spaces in practice
- 8 columns, not 4; VCI can contain "udhcp 1.32.1" etc.
Rewrote awk to detect interfaces via indentation (flush-left = iface,
indented = header or lease) and extract IP/MAC/Hostname from the
first three tokens of each lease row.
Verified against a captured sample; emits clean TSV.
First real run returned empty and we had no idea why — the script was
silently swallowing stderr via `2>/dev/null`. Remove the suppression
and try both `execute dhcp lease-list all` and the no-arg form, keeping
whichever returns non-error output.
Also emit a clearer diagnostic when both fail, pointing the user at
an interactive SSH to poke at command syntax.
vh