Revert the priming feature from d707439. Live A/B caught an audible artifact: the
context-priming discard-cut left part of the throwaway prefix in the output, so a
clause ("...without a trace of sarcasm,") was spoken an extra time.
Root cause is structural: generate() returns one finished waveform with no marker
for where the prefix ends, and the model renders the same prefix with different
timing when followed by content than when generated solo — so the duration-estimate
+ energy-minimum cut is a guess and can leave a sliver (or a whole clause) of prefix
in. A reliable cut would need token-level access (the abandoned native-streaming
arc) or a per-chunk ASR/alignment pass (heavy, still imperfect, eats the latency
budget). Fails the agreed bar: "keep only if it closes the gap without a seam."
Kept from d707439: the .gitignore (build artifacts). NOT re-applied: the bundled
margin_first fix — wiring it would shrink chunk 1 (more joins = worse coherence),
against the operator's priority, and margin=0.8 there is already starvation-safe.
Coherence loss at joins stays an accepted limitation; cold streaming was judged
"really good". Phase 1 + Phase 2 parity/perf untouched. Next: Phase 3 deploy.
Prime early joins by prepending the prior sentence as backward prosodic context,
generating context+content together, then discarding the context audio. The cut
snaps to the inter-sentence pause (energy-minimum search around the context's
solo duration) with a 5ms fade-in to kill any seam click (app: _cut_at_pause /
_fade_in / Engine.generate_primed). Opt-in via request `prime` (default off).
Scheduler: priming is AFFORDABILITY-GATED so it can never starve. A primed chunk
costs ~(2·context + content)/rtf (a 2nd context-solo pass); a chunk is only primed
when buffer ≥ prime_buffer_factor (1.5) × that cost, else it falls back to a cold
generate. Consequences proven in the GPU-free sim (17 tests):
- fires on early joins for any GPU at/above rtf_prior (3.4 = 3090; A6000 ~3.8-4.0)
- self-skips (degrades to cold) on a slower-than-fleet GPU rather than starving
- never primes chunk 0 (latency-critical)
Also fixed a latent Phase-1 bug: margin_first was applied at chunk 0 (budget always
0 there) so it never did anything — now applied at chunk 1 (the first transition).
Live A/B on irv-ml1 (A6000, GLaDOS): TTFB unaffected (445 vs 467ms), no starvation;
priming fired on chunk 2 (gen 1.6s for the doubled pass). On typical text exactly
ONE early join safely primes — priming chunk 2 flattens the buffer so later/larger
chunks no longer clear the safety gate. Samples: ~/chatterbox-ab/_p2_{cold,primed}.wav.
- /voices endpoint lists predefined voice stems (excludes `_`-prefixed bench/A-B
scratch wavs); shared _predefined_wavs() also feeds default-voice discovery.
- Perf levers: TF32 matmul/cudnn + flash/mem-efficient SDPA, default ON, env-gated
(CBF_TF32 / CBF_SDPA_FLASH). Startup logs model dtype.
Measured on irv-ml1 (turbo, A6000): the model loads FLOAT32 (not the fp16 older
notes assumed). TF32+SDPA do NOT move TTFA (489->514ms, noise) — first-sentence
latency is bound by the sequential AR token decode at batch-1, not matmul
throughput. bf16 (the lever that would help) is DEFERRED: from_pretrained() has no
dtype arg and turbo's fp32 conditioning path + dtype-sensitive vocoder make a
clean cast nontrivial; not worth the quality risk at ~0.5s TTFA. torch.compile
also deferred (batch-1 regression). Findings recorded in README.
Voice management parity (predefined dir + per-request clone refs) was already in
the Phase-1 resolve path; /voices completes the surface.
Build the streaming TTS server MVP per docs/design/chatterbox-fast-plan.md §4.
- scheduler.py: adaptive buffer-ratchet chunker (the meat) — GPU-free pure
logic. First sentence emitted alone for low TTFA, then chunks ratchet ~3x by
packing whole sentences to margin x buffered-audio; drives off measured RTF +
sec/char (EMA). relieve_leader() clause-splits a too-big mid-stream sentence
to avoid starvation (joins land on commas); a long comma-less sentence is the
one honored-but-flagged limitation.
- test_scheduler.py: GPU-free simulation, 13 tests — asserts no-starvation
(incl. overestimated RTF) and the ratchet.
- app.py: FastAPI model holder + POST /tts StreamingResponse (raw PCM s16le
default, wav optional, stream/oneshot) + GET /health.
- bench.py: client — ground-truth TTFB + real 1x-consumer starvation check.
Live test on irv-ml1 (turbo, A6000, GLaDOS voice): streaming TTFB 499ms vs
oneshot 5230ms (~10x), stayed ahead of a 1x player (no starvation), ratchet
1.64->4.08->8.60->8.60s audio, measured RTF self-corrected 3.38->4.01.
Kill the superseded docs/design/chatterbox-fast.md — its §5 windowed-token
streaming was the abandoned native-frame-streaming arc; the adaptive-chunk plan
supersedes it. Repoint persistent-memory + README at the canonical plan.
reference_id=<name> resolves against the DIRECTORY references/<name>/
(audio + same-basename .lab), not a flat references/<name>.wav. Voices
were staged flat with the per-name dirs left empty, so every
reference_id resolved to nothing and Fish fell back to its default
speaker — every dropdown voice produced byte-identical audio (proven:
Abigail == Imogen == no-ref, same text+seed). This was the real "no
accent" root cause, independent of the asset-engine "undefined" select
bug.
Server fix (applied to irv-ml1): populated references/<name>/<name>.wav
+ <name>.lab for all 32 voices; re-test confirms Imogen/Eleanor/
Beatrice/Abigail/no-ref now all distinct.
Durable hardening + record correction:
- playbook: normalize-layout step (flat <name>.wav -> nested dir, cp -u
idempotent, when-gated on count mismatch) + an A/B verify gate that
hard-fails the deploy if two reference_ids yield identical output.
- services.yaml: correct the reference_id resolution doc (dir + .lab,
not flat wav).
- README + persistent-memory: correct the "reference_id-by-name is THE
working path, verified" claim — it was a no-op until this fix; the
prior ECAPA 0.79 result came through the inline base64 path.
Tear down the parked CSM stack (status: down, never successfully built).
Bring-up attempts failed at the image build: upstream
phildougherty/sesame_csm_openai pins no huggingface_hub version, which now
resolves to 1.17.0 where the `huggingface-cli` the Dockerfile relies on has
been removed (replaced by `hf`). Building would require vendoring + patching
the upstream Dockerfile.
Deep-research verdict (primary + community sourced) confirmed it isn't worth
that: the acclaimed Maya/Miles demo runs a fine-tuned, larger CSM variant
Sesame never open-sourced; the open csm-1b is the un-fine-tuned 1B base
(only the smallest of 1B/3B/8B shipped, no newer checkpoint as of mid-2026).
Ships no usable voices, can't generate text, English-only, can't stream
real-time out of the box; absent from current TTS leaderboards and dominated
by Kokoro/Dia2/Fish-S2/IndexTTS for narration.
Removes: stacks/csm/, playbooks/deploy-csm.yaml, the csm catalog service +
reproducibility_audit entries. Host state (compose dir, /worktank/csm) torn
down on irv-ml1; no container/image existed.
Replace the single dia entry (legacy Dia 1.6B, retired) with two fixed-model
Dia2 entries (dia2-2b :8200, dia2-1b :8202), status ready (both exercised),
image local/dia:v2. Matching reproducibility_audit rows. catalog_version
unchanged (add/remove services = no vocab change).
Also fix a port collision I introduced earlier: the zonos-api adapter and
csm both claimed 8201 — move zonos-api to 8203 (catalog endpoint + voices
source_url, zonos .env.example, README).
NOTE FOR CONSUMERS: removing the dia id is a breaking catalog change for
asset_engine (it vendored dia in v0.1.4) — re-vendor + drop the dia tile,
add the two dia2 tiles.
The devnen wrapper is single-model and ignores the OpenAI model field, so
offering both Dia2 models to asset-engine as real per-request choices means
one fixed-model instance per model. Rework the dia stack to run two services
from a dia2-capable image:
* dia2-2b (:8200, best quality), dia2-1b (:8202, streaming) — both GPU 0
* each pins its model via a mounted /opt/docker/conf/dia2-*/config.yaml
Retire the legacy Dia 1.6B service.
New dia2-image/Dockerfile builds local/dia:v2 = upstream devnen wrapper +
the dia2 package (copied into site-packages; its pyproject build backend
yields an empty UNKNOWN wheel under the base's old setuptools) + the three
missing runtime deps (transformers/sphn/whisper-timestamped); torch 2.12 /
numpy 2.2 in the base already satisfy Dia2. Both instances verified
end-to-end (HTTP 200, Ogg/Opus 24 kHz).
Upstream Zonos ships only Gradio + Python SDK — no REST surface — so
asset-engine (which routes a clean JSON POST to /v1/audio/speech) can't
target it directly. Add a thin FastAPI adapter (stacks/zonos/adapter/):
POST /v1/audio/speech in front of the Zonos SDK, built FROM local/zonos
to reuse torch/CUDA/SDK. Returns a JSON envelope {audio, audio_format,
seed} — the seed rides back so asset-engine regenerate/fork can pin it
(Zonos is the fleet's first genuinely seedable TTS). compose gains a
zonos-api service on 8201; .env.example gains the port + voices dir.
Upstream Zyphra/Zonos ships no CMD in its Dockerfile (it launches the
app from its own compose), so our container ran the NVIDIA entrypoint,
printed the CUDA banner, exited 0, and restart-looped — nothing ever
bound 7860/8199. Add command: python3 gradio_interface.py to match
upstream, plus an explicit GRADIO_SHARE=False. Built + deployed to
irv-ml1; 8199 now serves HTTP 200 and the transformer model loads.
Sesame CSM-1B via phildougherty/sesame_csm_openai — OpenAI-compat
/v1/audio/speech, context-aware conversational speech (voice-agent
layer, not a plain reader). Port 8201 on irv-ml1. Gated model:
requires CSM_HF_TOKEN (license acceptance) — placeholder in .env.example,
real token only in host .env.
dia: Nari Labs dialogue TTS (Dia 1.6B / Dia2-1B / Dia2-2B) via
devnen/Dia-TTS-Server — OpenAI-compat, fills the multi-speaker
dialogue-scene slot for skaldsong. Port 8200 on irv-ml1.
zonos: Zyphra Zonos-v0.1 (Apache-2.0, 44kHz, emotion sliders) via the
official Gradio interface. Audition surface only — no OpenAI-compat
endpoint yet (needs the FastAPI fork to become skaldsong-pluggable).
Port 8199 on irv-ml1.
Both follow the chatterbox/fish-s2 convention: local image built from a
pinned wrapper SHA via buildx git-context, .env-driven port/GPU, python
healthcheck, homepage labels.
Worldtree-dev's --reasoning-format gemma suggestion isn't supported in
the deployed llama.cpp build (accepts only none|deepseek|deepseek-legacy).
Falling back to deepseek, which also populates reasoning_content — the
field Worldtree's GemmaProvider fallback path checks. Verified via test
inference: 608 reasoning_content deltas + 199 content deltas + 0 raw
<|channel> marker leaks.
Made the host-stacks bind-mount path configurable via
DOCKGE_HOST_STACKS_ROOT (default /opt/docker, unchanged for the
existing five hosts). Override on corviduo-dev to /home/vh/docker
because that host's /opt/ is owned by deploy:deploy (Worldtree team)
and vh lacks passwordless sudo for the fleet-standard path — same
reasoning as the beszel + dozzle agent placement earlier today.
Deployed to corviduo-dev. Reachable at http://10.250.50.152:5001
(first probe 200 — Docker's port-mapping route through iptables
worked without firewall changes, unlike beszel's network_mode: host).
Scoped to PFI-managed stacks only (/home/vh/docker/compose/) — does
NOT see /opt/worldtree*/ deployments. Keeps the management boundary
clean: dockge can restart/recreate PFI's beszel+dozzle+itself but
not the Worldtree-team-owned containers.
Operator-approved fleet monitoring extension. Both agents up + healthy:
- beszel-agent on host port 45876 (KEY-mode, hub at ana-docker:8090
SSH-polls inbound; seeded with hub's ed25519 pubkey).
- dozzle-agent on host port 7007 (mTLS auto-generated; hub at
ana-docker:8088 connects inbound).
Compose lives at /home/vh/docker/compose/{beszel,dozzle-agent}/ rather
than the fleet-standard /opt/docker/compose/{...}/ because corviduo-dev's
/opt/ is owned by deploy:deploy (Worldtree team) and vh lacks
passwordless sudo to create the fleet path. Functionally identical;
documented in servers/corviduo-dev/README.md so future infra-ops
sessions find them.
Created an empty traefik-net external docker network on corviduo-dev
as a side effect of dozzle-agent's compose (which declares it external).
Future PFI services landing here can reuse it.
Dozzle hub on ana-docker had 10.250.50.152:7007 appended to
DOZZLE_REMOTE_AGENT (a host-side change to a non-tracked .env, not
canonical). Beszel hub still needs corviduo-dev added via the UI's
"Add System" action — one-time operator step, flagged in the corviduo
README.
Temporary diagnostic for the class of bug story 83ff386d47c6 hit
2026-05-23: POST /generation/start returned 202, then total silence
— no log, no DB state update, py-spy showed event loop idle with no
GenerationRunner frame anywhere. Strongly suggests a created_task()
result not held → GC'd → silent destroy.
PYTHONASYNCIODEBUG=1 emits "Task was destroyed but it is pending"
and "Task exception was never retrieved" warnings to stderr; that
should distinguish lost-task from cancelled-task on the next attempt.
Per skaldsong-dev's note, remove once they wire proper task-exception
capture upstream.
Diagnosis thread: althing 01KSBGKQBXA756JWW1KD4MPGXM
The compose set SKALDSONG_DB_PATH + SKALDSONG_RUNS_DIR, but skaldsong's
app reads SKALDSONG_HOST_SQLITE_PATH + SKALDSONG_HOST_RUNS_ROOT (per
its Dockerfile ENV defaults). Our values were orthogonal — the app
fell back to Dockerfile defaults pointing at /app/data/... which is
NOT bind-mounted, so every --force-recreate wiped the SQLite DB +
runs/ tree along with the ephemeral container layer.
Surfaced by skaldsong-dev (althing thread 01KS4DPF6SXTBP4Q360JZVWPNT)
after the operator noticed stories vanishing on every deploy.
Confirmed on ana-docker: container had a 40KB skaldsong-ui.db sitting
in /app/data/, while /opt/docker/conf/skaldsong/db/ on the host was
empty. Rescued the live DB to the bind-mount target before recreate.
Fix: rename env vars to match what the app reads. Bind targets stay
at /app/state/{db,runs} (parent-dir mount for SQLite WAL+SHM).
Two corrections surfaced by the first end-to-end deploy that didn't
land in the pre-flight align:
- SPA static assets are at /app/spa, not /app/web/dist (Dockerfile
COPYs the SvelteKit build output flat into /app/spa, not into
/app/spa/dist). Mismatch caused /health to 500 with
"RuntimeError: File at path /app/web/dist/index.html does not
exist."
- SKALDSONG_HOST_CORS_ORIGINS must be a JSON array literal in .env.
Pydantic-settings parses complex-typed env vars via json.loads();
bare URL string fails first-boot with SettingsError.
Container now reports Up (healthy) on ana-docker; /health 200.
skaldsong-dev surfaced three contract corrections before the first
deploy:
- WORLDTREE_TOKEN (outbound HTTP Bearer) was missing — separate code
path from SKALDSONG_BIFROST_JWT_KEY (inbound HS256 verify) but
same secret value.
- WORLDTREE_BASE_URL replaces SKALDSONG_WORLDTREE_API_URL (the
former is what the app actually reads).
- SKALDSONG_HOST_WIZARD_AGENT_ID was missing entirely — must pin to
skaldsong:wizard-v2 to inherit the existing Worldtree agent slot;
blank would burn another slot of the 50-per-key Heimdall quota.
Registry-pull pattern matching Worldtree: CI on vh/skaldsong builds and
pushes gitea.phasefinal.com/vh/skaldsong:<sha>, this playbook pulls +
recreates. SHA-pin only per current preference; no :latest moving-tag
advance yet (revisit once /health exercises Worldtree + Kokoro reach).
Host port 8300 (host) → 8000 (container). Persistent state under
/opt/docker/conf/skaldsong/{db,runs}.
Bifrost endpoint URL 10.250.50.70:8300 will need a paired
BIFROST_CLIENT_ALLOWED_HOSTS update on corviduo-dev Worldtree at first
deploy.
Phase 3.1 closes the cross-process gap the Phase 3 smoke surfaced —
streaming events (msg_start/thinking/delta/complete/curated) flow
from agent-runner → chamber via valkey pub/sub rather than the
SQLite bridge (too high-volume + ephemeral for the DB).
New service: `althing-valkey` (stock `valkey/valkey:8-alpine`).
Internal-only — no exposed port, no volume. chamber + agent-runner
reach via docker DNS at `valkey:6379` on the compose default
network. healthcheck via `valkey-cli ping` (5s interval). chamber
+ agent-runner gain `depends_on: valkey: service_healthy` so the
bridge is up before either side starts publishing or subscribing.
Forseti unchanged — never publishes Phase 3 events.
Operational properties (per forseti's deployment notes):
- Mixed-state safe at every step. Missing valkey.url config key
→ chamber + runner stay on v3.0 / Phase 2 equivalent paths.
- Backward path is single config-key delete + restart.
- streaming_enabled: true (set on agent-runner 2026-05-17) is
unaffected by this change.
README's services table + playbook header + verify section all
extended to reflect the four-service shape. Forseti's contract
at vh/althing:docs/contracts/phase3_1_valkey_bridge.contract.md
carries the wire-protocol spec.
Phase 2 daemon added to the althing-chamber stack per forseti's request
(vh/althing@5cd088a..ad1d025). Polls floor_grants WHERE consumed_at IS
NULL AND agents.driver='worldtree', claims via atomic UPDATE, calls
Worldtree's conversation API, posts the response back through the bus
as a broadcast.
Shape matches the existing forseti daemon:
- Same ${ALTHING_IMAGE} (the binary is already in [project.scripts]
as of ad1d025)
- command: ["althing-agent-runner"]
- Same shared SQLite bind-mount at /app/data
- No port, no healthcheck (CLI doesn't expose one; same liveness
story as forseti)
Safe to enable preemptively per forseti — when no driver=worldtree
handles are declared in config, the runner sleeps at
poll_interval_seconds. Multi-instance safe via the atomic claim
primitive (no flock needed).
Compose top comment, README "Services in this stack" table, playbook
header + verify steps all extended to reflect the three-service
shape. Will land on ana-docker on vh/althing's next push (compose
deployed via the elway playbook's upload step; image already carries
the binary).
Two-service compose (chamber + forseti sidecar daemon) sharing a single
SQLite store via bind-mount under /opt/docker/conf/althing-chamber/data.
eventbus.bridge_from_db is the cross-process glue — forseti's commits
reach chamber's SSE subscribers via the bridge.
Pattern matches task-board's build-on-host deploy:
- elway playbook clones vh/althing into /opt/docker/build/
- docker build -t althing-chamber:local . (no registry)
- playbook uploads compose + seeds .env one-time, brings both
services up, polls /health
- Gitea Actions workflow lives in vh/althing; reference copy here.
Internal tooling — host port 7881 (chamber's default of 7878 collides
with task-board). LAN-direct, no Traefik. Container always listens on
8000 internally.
Scaffold will fail to bring the chamber container up healthy until
galdrabok-side commits land:
- Dockerfile at vh/althing repo root (two-stage: uv-bookworm-slim
build → python:3.12-slim runtime, locked per open_questions §2
of the v1 contract).
- GET /health endpoint on the chamber app (200, no DB read).
- ALTHING_BIND / ALTHING_PORT env-var support in
core.cli.chamber_serve / core.chamber.cli (env > config.yaml >
defaults precedence).
Coordinated via althing thread 01KRMAK7RD7TP6C8DF4KXV31RT.
Stack was retired and replaced by the vllm stack (originally vllm-qwen3,
renamed 2026-05-13). Its README still framed it as a current solution
while ana-ml2's README + vllm's README both documented the retirement.
stacks/vllm/README.md "Migrating off Infinity" step 3 explicitly said
"Delete stacks/infinity/ from this workspace" — actioning that now.
No backwards-compat shims (PRACTICES §4): contract of a deleted system
has no historical value the next contributor needs; the replacement
path is documented in stacks/vllm/README.md.
Surfaced by /tend-docs audit 2026-05-14.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Playbook handles models, datasets, and spaces (via --var repo_type=...)
since 3025d49 — the "-model" suffix was misleading. Renaming to match
actual scope.
Updates the single in-repo reference (changelog comment in
stacks/llama-swap/conf/config.yaml). config.yaml was scp'd to ana-ml2;
no docker compose restart needed (comment-only).
AtlaAI's Selene-1-Mini judge model for evaluation/scoring tasks.
Llama 3.1 8B base, mradermacher imatrix-quantized Q6_K (~6.5GB,
quality-leaning quant). Apache-2.0. Per Atla cookbook these defaults
hit 84% on RAGTruth hallucination eval.
New 'JUDGE / EVAL MODELS' section between the dense chat models and
the embedding models — separate category from chat/reasoning since
the run-params shape is different (deterministic-leaning: temp 0.01,
top-p 1.0, no repeat penalty).
q8_0 KV cache to fit 32K ctx cleanly on the 3090 with headroom.
Pre-pulled into the shared HF cache via the new
playbooks/pull-hf-model.yaml playbook (canonical replacement for
ad-hoc huggingface_hub.snapshot_download calls; see CHANGELOG).
Smoke-tested 2026-05-13: GET /v1/models lists selene-1-mini-8b,
POST /v1/chat/completions returns expected output cleanly.
Three changes prepping infra for asset_engine's orchestration feature
(SSH-driven bring-up / bring-down of irv-ml1 inference services with
per-device VRAM gating, contract in vh/asset-engine commit 5a36f8c):
1. asset-engine compose + .env.example + playbook gain a read-only
bind-mount for /app/runtime/ssh — the dedicated ed25519 keypair
(generated on ana-docker, not in the repo) plus a pinned known_hosts
for irv-ml1's host fingerprint. Env vars SSH_KEY_PATH and
SSH_KNOWN_HOSTS are exposed for the app to consume.
2. docs/asset-engine/services.yaml gains a `lifecycle: { stack, vram_gb,
gpu_device_id }` block on each of 12 orchestratable irv-ml1 services
(kokoro, chatterbox, index-tts, qwen3-tts, cosyvoice, fish-s2,
kyutai-tts, vibevoice, voxtral, parakeet, stable-audio-open, ace-step).
VRAM numbers are estimates from model footprint at fp16 — tune from
real nvidia-smi measurements once the gate is live. comfyui and
kokoro-captioned are deliberately excluded (variable-VRAM and
shared-container respectively).
3. servers/irv-ml1/README.md docker-stacks table now lists all 13
inference stacks (was only dockge + agents + comfyui) with port +
GPU pinning columns.
Pubkey deployed to ~lkraven/.ssh/authorized_keys on irv-ml1;
end-to-end SSH from ana-docker → irv-ml1 verified with strict
host-key checking.
Adds VOR_WORLDTREE_KEY + VOR_WORLDTREE_BASE + VOR_WORLDTREE_MODEL to vor's
compose environment with sane defaults. Empty key falls back to the
in-process MockWorldtree (the /mockup/ surface returns canned fixtures);
a real key issued by architect routes LLM calls at the demo Saga instance.
Key itself lives in ana-docker:/opt/docker/compose/vor/.env (not in the
repo).
Internal tooling — accessed at http://10.250.50.70:8200, not through
Traefik. Removes the unused traefik labels (router rule, TLS, crowdsec
middleware, loadbalancer port) and the traefik-net network membership;
homepage.href now points at host:port for direct discovery, matching
task-board's pattern. Playbook verify drops the traefik-net membership
check.
Mirrors task-board's build-on-host pattern: elway playbook clones
vh/asset-engine into /opt/docker/build/, docker build, install compose +
seed .env, up -d, verify /health. No registry.
Internal-only tool — LAN port 8200 (bind 0.0.0.0) is primary; Traefik
labels additionally route asset-engine.phasefinal.com with TLS via the
anaprod cert resolver. DB and outputs are separate bind-mounts under
/opt/docker/conf/asset-engine/ so outputs/ can move volumes later
without touching DB state. INFERENCE_HOST defaults to 10.100.79.3
(irv-ml1 over WG). OIDC env seam is pre-allocated empty for v2.
The pre-fix wrapper at stacks/ace-step/infer-api.py returned a JSON
{output_path: "..."} reference to a file written inside the
container at /app/outputs/. That path was unreachable from outside
the container — every consumer got 134 bytes of JSON-pretending-to-
be-WAV instead of audio. Surfaced by the asset_engine consumer's
end-to-end smoke (althing thread 01KRCJF7NGMXYE9F62Q1A6KFD4 msg 5);
my own earlier smoke missed it because I checked HTTP=200 and stopped
reading instead of inspecting the response body.
Wrapper now reads back the file the pipeline writes and streams the
bytes via fastapi.responses.Response with media_type set from the
audio_format request field (audio/wav | audio/mpeg | audio/flac).
The in-container path is exposed via X-Output-Path header for log
correlation but is no longer load-bearing.
Verified end-to-end against live ace-step on irv-ml1:
POST /generate -> HTTP 200 in 80s
content-type: audio/wav
content-length: 945226
x-output-path: /app/outputs/output_cfe87d1d....wav
$ file response.wav
RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit,
stereo 48000 Hz
Catalog: ace-step bumped version 3 -> 4. Dropped
response.output_field (no longer applicable). reproducibility.notes
expanded to record both the v2 18-arg-tuple fix and this v4
inline-streaming change so the history is auditable from the
catalog itself.
Stale ACEStepOutput Pydantic model left in infer-api.py for now —
unused but small; future cleanup.
StableAudioPipeline isn't reentrant — concurrent requests share the
scheduler's step_index counter and corrupt each other mid-run
(observed: IndexError in cosine_dpmsolver_multistep when two requests
overlap). Wrap the pipeline call + audio decode in a single
asyncio.Lock created at startup, and run the (sync, GPU-bound)
pipeline call via asyncio.to_thread so the event loop stays
responsive. Concurrent requests now queue cleanly instead of racing.
Verified: 5 parallel POSTs at steps=50 all return 200, clear ~4s
serialization spacing (4, 8, 12, 16, 20s wall time), distinct
output hashes per seed.
server.py accepted cfg_scale in the request schema and the README
documented its 0–20 range, but the pipeline call never received it
— so changing cfg_scale between requests silently produced identical
output (the pipeline ran at its own default of 7.0 every time). Add
guidance_scale=req.cfg_scale to the pipe(...) call.
Verified: (prompt, seed, steps) held constant, cfg_scale=3.0 vs 15.0
now produce different SHA256s; same triple at cfg_scale=7.0 is
deterministic across repeated calls.
Wrapper only enumerates one voice directory (settings.voices_dir,
default /app/api/src/voices/v1_0 — inside the container's writable
layer, not bind-mounted). Override via VOICES_DIR=/app/user_voices
(host bind mount) and add a command shim that cp -r's built-ins from
the in-image v1_0 into user_voices on every start. Built-ins re-seed
fresh from the image (so upgrades that add voices propagate); custom
.pt files in user_voices are preserved (cp -r is additive).
Also adds scripts/blend_kokoro_voice.py + a playbook around it that
mirrors the wrapper's request-time voice="a(w)+b(w)" math but writes
the result as a named .pt to user_voices, making it discoverable via
GET /v1/audio/voices and persistent across recreate. Defaults to
athena = af_bella(2)+af_aoede(1) normalized.
Same shape as task-board: build-on-host from vh/vor, bind-mounted
persistence for sessions/ and responses/ (the user-published markdown
files), exposed at port 7879 (adjacent to task-board's 7878 since both
are claude-tooling sidecars).
Workflow template assumes the same DEPLOY_SSH_KEY + MGMT_REPO_TOKEN
secrets at user scope; nothing new to provision. Playbook accepts SHA
or branch refs (same fix as deploy-task-board.yaml) so manual runs
and CI runs share the same code path.
Centralized vs upstream-local: README documents the trade. Claude
fetches response markdown via /api/sessions/{id} JSON instead of a
local file read — the only API-flow change from the upstream README.
Runner is now re-registered with `:docker://node:20-bookworm-slim`
schema in its labels, so workflows targeting `pfi-fleet` get that
image automatically. Saves a few lines per workflow and gives us one
place (the runner config) to bump the default image when a new
node/debian release lands.
Runner's .runner registration cached :host mode at first start; env-var
label updates aren't sticky once the runner is registered. Until we
re-register with docker-schema labels, workflows must declare their
own container. node:20-bookworm-slim has node (for actions/checkout)
and apt (for python3-yaml + openssh-client install).
debian:bookworm-slim lacks node, so actions/checkout@v4 (a JS action
running dist/index.js) fails with `exec: "node": executable file not
found in $PATH`. Dropping the explicit `container:` directive lets
the runner use its label-default — node:20-bookworm-slim has node +
git out of the box. Install step shrinks to python3 + pyyaml +
openssh-client.
Central runner on ana-docker (gitea is local; existing fleet tooling
already SSHes from there). Playbook is parameterized so future
site-local runners (nh3-docker, esh-docker-vm) drop in via --var
overrides instead of copy-paste.
Includes a workflow template for vh/task-board that calls the existing
deploy-task-board.yaml playbook — keeps the playbook as the single
source of truth for "how task-board is deployed", manual or automated.
Labels embed `:docker://node:20-bookworm-slim` schema; without it,
act_runner v0.6+ silently falls back to host-mode and runs job steps
inside the Alpine runner container (no apt/python/node), breaking any
real workflow. node:20-bookworm-slim is small + has git + node so
actions/checkout works out of the box.
The applet outgrew "stack alongside the infra-management workspace" —
it has its own pyproject, multi-tenant deploy story, separate
release cadence, and isn't actually about managing infrastructure.
Lives at https://gitea.phasefinal.com/vh/nevermore now, with
provenance noted in its initial commit.
This commit removes:
stacks/news-digest/ (full stack tree)
playbooks/deploy-news-digest.yaml
scripts/add-digest-user.sh
The existing ana-docker deployment continues running on its baked
local/news-digest:v5 image — nothing changes for the live install
until you choose to redeploy from the new repo. Migration steps
(rename data dir, redeploy, retire old compose dir) are in
nevermore's README.
Updated:
README.md — Current stacks listing now points at the new repo
STATUS.md — milestones entry for the extraction
The compose-side default was still pinning qwen3.5-35-a3b — broken
on launch since its GGUF stopped working months ago. Real .env on
ana-docker overrides to granite-4-small so live deploys are unaffected,
but the default was misleading for anyone forking the stack. Found
via /tend-docs.
The original default model in .env.example was changed to
granite-4-small months ago when qwen3.5-35-a3b's GGUF file started
exiting on launch, but the README still named the old one as
"current". Also bumped the summarization-style description from
"one sentence" to "2-3 sentences" to match the post-trafilatura
prompt rewrite. Found via /tend-docs.
Stock neosmemo/memos:stable, port 5230, SQLite at
/opt/docker/conf/memos/data/. Joins traefik-net and ships homepage
labels (group=Notes) so it auto-appears on the dashboard via docker
discovery — no edit to configs/homepage/services.yaml needed.
First-run bootstrap is via the UI: visit http://10.250.50.70:5230
and create the Host account through the sign-up form.
Playbook idiom note: docker compose pull lines need the literal
block scalar (|) when the grep pattern contains colons — bare-string
shell value made YAML parse the colon as a mapping separator and
elway choked on first try.
Adds a "Customizing the run schedule" section (DIGEST_CRON_AM/PM env
vars, edit-and-recreate flow) and a "Multi-tenant: one instance per
teammate" section covering scripts/add-digest-user.sh end to end:
what it does, the per-user file layout on ana-docker, idempotent
schedule/password updates, and the teardown path.
Updated the stale "two editions per day" intro line to note the
schedule is now configurable.
vh