LLM observability for the fleet — pretty trace UI over the gateway: prompts,
completions, reasoning, latency, token counts. The pretty layer LiteLLM's
spend_logs lacked.
- stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/
minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000).
Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on
langfuse-web so it's reachable via the published port while also on tnet.
- litellm: enabled success_callback/failure_callback: ["langfuse"] (the
passthrough env was already wired); keys + host go in the litellm .env.
Verified: stack healthy, project keys authenticate, and a real gateway call
landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in
the server .env (never committed).
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's
the new production summarizer/dreamer for nevermore.
- vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8,
compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot.
- GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a
5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18,
freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and
keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora).
- LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004);
explicit entry shadows the '*' wildcard's llama-swap route.
- nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live.
Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok),
KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
ComfyUI's default cudaMallocAsync allocator phantom-OOMs ('allocation
would exceed allowed memory', 0 bytes allocated) when the A6000 is
shared with the pinned TTS services (~22 GB used). --disable-cuda-malloc
switches to PyTorch's native allocator. --fp8_e4m3fn-text-enc loads the
FLUX.2 Qwen3-8B text encoder as fp8 (~8.7 GB) instead of upcasting the
fp8 file to fp16 (~16 GB), matching the box's Ampere-fp8 posture.
Applied via COMFY_CMDLINE_EXTRA in the canonical compose; verified the
allocator flipped to 'native' and both flags are on the live cmdline.
IBM Granite 4.1 dense instruct (3B + 8B), unsloth Q4_K_M GGUF via -hf
syntax, 64K ctx with q8 KV. Auto-exposed through the LiteLLM gateway
wildcard (ana-docker:4000) and direct on llama-swap (:9292).
Replaces the retired irv-ml1 Ollama granite4.1 — Ollama is now banned
fleet-wide; serving consolidates onto the sanctioned llama-swap/vLLM
substrate.
Adds explicit gateway entries for the four z.ai GLM models (glm-5.1,
glm-5-turbo, glm-4.7, glm-4.5-air) routed to api.z.ai with Z_AI_API_KEY,
plus the compose env passthrough + .env.example doc. Explicit entries
win over the llama-swap wildcard (distinct IDs, no collision). Extends
the gateway's unified logging to cloud inference, not just local
vLLM/llama-swap.
Cost note: paid API — only gateway-keyed callers reach these, but calls
spend z.ai credits (documented in config + compose comments).
Adds a `model_name: "*"` entry routing any unmatched model to llama-swap
(ana-ml2:9292) so its whole swappable LLM zoo logs through the gateway
without per-model registration — add/swap models in llama-swap freely,
litellm logs them all. Exact entries (phi4-mini/qwen3-embedding/
qwen3-reranker → vLLM) still win; the wildcard only catches the rest.
litellm does no inference; llama-swap keeps loading + serving. Enables
routing worldtree-personal's generative chat through the gateway for
full req/resp logging while preserving llama-swap's on-demand swapping.
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).
- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
(:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
full prompt/completion text. reward classifier (:8003) stays direct
(no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.
Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
Operator chose option (ii): keep the OFFICIAL Phi-4 format globally rather than
impose Ollama's leaner scaffold on every phi4 consumer. Removes the
--chat-template override + the conf/phi4-chat-template.jinja file (90e08f0).
vLLM now uses the tokenizer's built-in template (system <|end|> present);
verified 7-token render via tokenize/detokenize. brokkr re-baselines its R15
canonical on the official scaffold so baseline == production.
vLLM's official Phi-4 tokenizer template emits <|end|> after the system turn;
Ollama's does not. That single boundary token regressed brokkr's R15 P02
admission eval (type macro-F1 -33pp) vs the Ollama-measured canonical, while
valid_format held at 1.0. Operator chose to make vLLM match Ollama's leaner
scaffold globally (baseline == production). Adds conf/phi4-chat-template.jinja
(drops the system <|end|>) + mounts it + --chat-template on vllm-phi4. Applied
prompt verified via tokenize/detokenize; brokkr re-smokes probe_vllm.yaml.
chatterbox-fast is authored software with a test suite, not a config-mirror stack —
so it moves to its own MIT-licensed, versioned, CI'd repo (gitea vh/chatterbox-fast,
v0.1.0) following the sister-repo pattern. Replace stacks/chatterbox-fast/ with a
pointer README; the moved code (scheduler/app/bench/tests/Dockerfile/compose) now
lives in the new repo. The deployed :8197 service is unaffected (still runs the
legacy devnen-based image; self-contained-image migration is an optional follow-up).
The fleet catalog entry stays in docs/asset-engine/services.yaml.
- chatterbox-fast experimental -> ready: browser audition verified end-to-end
(operator confirmed progressive playback "excellent" 2026-06-02).
- vibevoice ready -> down: no container running on irv-ml1 (connection refused);
catalog status was stale.
- voxtral: NOT a stale typo — its stack genuinely claimed :8197, the port now held
by the live chatterbox-fast. voxtral is down, so moved IT to :8201 (catalog
endpoint + source_url, stacks/voxtral/.env.example + README, host .env) rather
than disturb the live service. No live clash existed (voxtral down) but it was a
latent deploy-time collision I introduced by placing chatterbox-fast on 8197.
No catalog_version bump (status changes + endpoint correction, additive). Validates
against the schema.
TTSRequest gains `seed` (0=random); seeded once per request under the lock via
torch.manual_seed + cuda.manual_seed_all. One-shot output is then byte-reproducible
for a fixed seed+params (verified: seed=42 -> identical sha256 across runs).
Streaming stays non-reproducible by design — adaptive-chunk boundaries depend on
live-measured RTF. Needed for the asset-engine catalog reproducibility contract
(parity with the chatterbox sibling, which exposes seed).
stream=false + format=wav emitted the streaming 0xFFFFFFFF-length header, so a
buffered consumer reading a complete wav got bogus RIFF/data sizes. One-shot knows
the full length, so emit correct sizes; streaming keeps the open-ended header
(length genuinely unknown up front). Verified remote: one-shot wav data size ==
bytes-44, python wave.open() reads 2.20s cleanly; streaming still 0xFFFFFFFF.
Deployed on irv-ml1 beside live chatterbox (:8196): healthy on :8197, TTFB ~0.5s,
no starvation. Measured VRAM 5.34 GB (fp32) settles the placement: the 3090's
~3.8 GB free does NOT fit, A6000 (device 1) is the only viable card.
Revert the priming feature from d707439. Live A/B caught an audible artifact: the
context-priming discard-cut left part of the throwaway prefix in the output, so a
clause ("...without a trace of sarcasm,") was spoken an extra time.
Root cause is structural: generate() returns one finished waveform with no marker
for where the prefix ends, and the model renders the same prefix with different
timing when followed by content than when generated solo — so the duration-estimate
+ energy-minimum cut is a guess and can leave a sliver (or a whole clause) of prefix
in. A reliable cut would need token-level access (the abandoned native-streaming
arc) or a per-chunk ASR/alignment pass (heavy, still imperfect, eats the latency
budget). Fails the agreed bar: "keep only if it closes the gap without a seam."
Kept from d707439: the .gitignore (build artifacts). NOT re-applied: the bundled
margin_first fix — wiring it would shrink chunk 1 (more joins = worse coherence),
against the operator's priority, and margin=0.8 there is already starvation-safe.
Coherence loss at joins stays an accepted limitation; cold streaming was judged
"really good". Phase 1 + Phase 2 parity/perf untouched. Next: Phase 3 deploy.
Prime early joins by prepending the prior sentence as backward prosodic context,
generating context+content together, then discarding the context audio. The cut
snaps to the inter-sentence pause (energy-minimum search around the context's
solo duration) with a 5ms fade-in to kill any seam click (app: _cut_at_pause /
_fade_in / Engine.generate_primed). Opt-in via request `prime` (default off).
Scheduler: priming is AFFORDABILITY-GATED so it can never starve. A primed chunk
costs ~(2·context + content)/rtf (a 2nd context-solo pass); a chunk is only primed
when buffer ≥ prime_buffer_factor (1.5) × that cost, else it falls back to a cold
generate. Consequences proven in the GPU-free sim (17 tests):
- fires on early joins for any GPU at/above rtf_prior (3.4 = 3090; A6000 ~3.8-4.0)
- self-skips (degrades to cold) on a slower-than-fleet GPU rather than starving
- never primes chunk 0 (latency-critical)
Also fixed a latent Phase-1 bug: margin_first was applied at chunk 0 (budget always
0 there) so it never did anything — now applied at chunk 1 (the first transition).
Live A/B on irv-ml1 (A6000, GLaDOS): TTFB unaffected (445 vs 467ms), no starvation;
priming fired on chunk 2 (gen 1.6s for the doubled pass). On typical text exactly
ONE early join safely primes — priming chunk 2 flattens the buffer so later/larger
chunks no longer clear the safety gate. Samples: ~/chatterbox-ab/_p2_{cold,primed}.wav.
- /voices endpoint lists predefined voice stems (excludes `_`-prefixed bench/A-B
scratch wavs); shared _predefined_wavs() also feeds default-voice discovery.
- Perf levers: TF32 matmul/cudnn + flash/mem-efficient SDPA, default ON, env-gated
(CBF_TF32 / CBF_SDPA_FLASH). Startup logs model dtype.
Measured on irv-ml1 (turbo, A6000): the model loads FLOAT32 (not the fp16 older
notes assumed). TF32+SDPA do NOT move TTFA (489->514ms, noise) — first-sentence
latency is bound by the sequential AR token decode at batch-1, not matmul
throughput. bf16 (the lever that would help) is DEFERRED: from_pretrained() has no
dtype arg and turbo's fp32 conditioning path + dtype-sensitive vocoder make a
clean cast nontrivial; not worth the quality risk at ~0.5s TTFA. torch.compile
also deferred (batch-1 regression). Findings recorded in README.
Voice management parity (predefined dir + per-request clone refs) was already in
the Phase-1 resolve path; /voices completes the surface.
Build the streaming TTS server MVP per docs/design/chatterbox-fast-plan.md §4.
- scheduler.py: adaptive buffer-ratchet chunker (the meat) — GPU-free pure
logic. First sentence emitted alone for low TTFA, then chunks ratchet ~3x by
packing whole sentences to margin x buffered-audio; drives off measured RTF +
sec/char (EMA). relieve_leader() clause-splits a too-big mid-stream sentence
to avoid starvation (joins land on commas); a long comma-less sentence is the
one honored-but-flagged limitation.
- test_scheduler.py: GPU-free simulation, 13 tests — asserts no-starvation
(incl. overestimated RTF) and the ratchet.
- app.py: FastAPI model holder + POST /tts StreamingResponse (raw PCM s16le
default, wav optional, stream/oneshot) + GET /health.
- bench.py: client — ground-truth TTFB + real 1x-consumer starvation check.
Live test on irv-ml1 (turbo, A6000, GLaDOS voice): streaming TTFB 499ms vs
oneshot 5230ms (~10x), stayed ahead of a 1x player (no starvation), ratchet
1.64->4.08->8.60->8.60s audio, measured RTF self-corrected 3.38->4.01.
Kill the superseded docs/design/chatterbox-fast.md — its §5 windowed-token
streaming was the abandoned native-frame-streaming arc; the adaptive-chunk plan
supersedes it. Repoint persistent-memory + README at the canonical plan.
reference_id=<name> resolves against the DIRECTORY references/<name>/
(audio + same-basename .lab), not a flat references/<name>.wav. Voices
were staged flat with the per-name dirs left empty, so every
reference_id resolved to nothing and Fish fell back to its default
speaker — every dropdown voice produced byte-identical audio (proven:
Abigail == Imogen == no-ref, same text+seed). This was the real "no
accent" root cause, independent of the asset-engine "undefined" select
bug.
Server fix (applied to irv-ml1): populated references/<name>/<name>.wav
+ <name>.lab for all 32 voices; re-test confirms Imogen/Eleanor/
Beatrice/Abigail/no-ref now all distinct.
Durable hardening + record correction:
- playbook: normalize-layout step (flat <name>.wav -> nested dir, cp -u
idempotent, when-gated on count mismatch) + an A/B verify gate that
hard-fails the deploy if two reference_ids yield identical output.
- services.yaml: correct the reference_id resolution doc (dir + .lab,
not flat wav).
- README + persistent-memory: correct the "reference_id-by-name is THE
working path, verified" claim — it was a no-op until this fix; the
prior ECAPA 0.79 result came through the inline base64 path.
Tear down the parked CSM stack (status: down, never successfully built).
Bring-up attempts failed at the image build: upstream
phildougherty/sesame_csm_openai pins no huggingface_hub version, which now
resolves to 1.17.0 where the `huggingface-cli` the Dockerfile relies on has
been removed (replaced by `hf`). Building would require vendoring + patching
the upstream Dockerfile.
Deep-research verdict (primary + community sourced) confirmed it isn't worth
that: the acclaimed Maya/Miles demo runs a fine-tuned, larger CSM variant
Sesame never open-sourced; the open csm-1b is the un-fine-tuned 1B base
(only the smallest of 1B/3B/8B shipped, no newer checkpoint as of mid-2026).
Ships no usable voices, can't generate text, English-only, can't stream
real-time out of the box; absent from current TTS leaderboards and dominated
by Kokoro/Dia2/Fish-S2/IndexTTS for narration.
Removes: stacks/csm/, playbooks/deploy-csm.yaml, the csm catalog service +
reproducibility_audit entries. Host state (compose dir, /worktank/csm) torn
down on irv-ml1; no container/image existed.
Replace the single dia entry (legacy Dia 1.6B, retired) with two fixed-model
Dia2 entries (dia2-2b :8200, dia2-1b :8202), status ready (both exercised),
image local/dia:v2. Matching reproducibility_audit rows. catalog_version
unchanged (add/remove services = no vocab change).
Also fix a port collision I introduced earlier: the zonos-api adapter and
csm both claimed 8201 — move zonos-api to 8203 (catalog endpoint + voices
source_url, zonos .env.example, README).
NOTE FOR CONSUMERS: removing the dia id is a breaking catalog change for
asset_engine (it vendored dia in v0.1.4) — re-vendor + drop the dia tile,
add the two dia2 tiles.
The devnen wrapper is single-model and ignores the OpenAI model field, so
offering both Dia2 models to asset-engine as real per-request choices means
one fixed-model instance per model. Rework the dia stack to run two services
from a dia2-capable image:
* dia2-2b (:8200, best quality), dia2-1b (:8202, streaming) — both GPU 0
* each pins its model via a mounted /opt/docker/conf/dia2-*/config.yaml
Retire the legacy Dia 1.6B service.
New dia2-image/Dockerfile builds local/dia:v2 = upstream devnen wrapper +
the dia2 package (copied into site-packages; its pyproject build backend
yields an empty UNKNOWN wheel under the base's old setuptools) + the three
missing runtime deps (transformers/sphn/whisper-timestamped); torch 2.12 /
numpy 2.2 in the base already satisfy Dia2. Both instances verified
end-to-end (HTTP 200, Ogg/Opus 24 kHz).
Upstream Zonos ships only Gradio + Python SDK — no REST surface — so
asset-engine (which routes a clean JSON POST to /v1/audio/speech) can't
target it directly. Add a thin FastAPI adapter (stacks/zonos/adapter/):
POST /v1/audio/speech in front of the Zonos SDK, built FROM local/zonos
to reuse torch/CUDA/SDK. Returns a JSON envelope {audio, audio_format,
seed} — the seed rides back so asset-engine regenerate/fork can pin it
(Zonos is the fleet's first genuinely seedable TTS). compose gains a
zonos-api service on 8201; .env.example gains the port + voices dir.
Upstream Zyphra/Zonos ships no CMD in its Dockerfile (it launches the
app from its own compose), so our container ran the NVIDIA entrypoint,
printed the CUDA banner, exited 0, and restart-looped — nothing ever
bound 7860/8199. Add command: python3 gradio_interface.py to match
upstream, plus an explicit GRADIO_SHARE=False. Built + deployed to
irv-ml1; 8199 now serves HTTP 200 and the transformer model loads.
Sesame CSM-1B via phildougherty/sesame_csm_openai — OpenAI-compat
/v1/audio/speech, context-aware conversational speech (voice-agent
layer, not a plain reader). Port 8201 on irv-ml1. Gated model:
requires CSM_HF_TOKEN (license acceptance) — placeholder in .env.example,
real token only in host .env.
dia: Nari Labs dialogue TTS (Dia 1.6B / Dia2-1B / Dia2-2B) via
devnen/Dia-TTS-Server — OpenAI-compat, fills the multi-speaker
dialogue-scene slot for skaldsong. Port 8200 on irv-ml1.
zonos: Zyphra Zonos-v0.1 (Apache-2.0, 44kHz, emotion sliders) via the
official Gradio interface. Audition surface only — no OpenAI-compat
endpoint yet (needs the FastAPI fork to become skaldsong-pluggable).
Port 8199 on irv-ml1.
Both follow the chatterbox/fish-s2 convention: local image built from a
pinned wrapper SHA via buildx git-context, .env-driven port/GPU, python
healthcheck, homepage labels.
Worldtree-dev's --reasoning-format gemma suggestion isn't supported in
the deployed llama.cpp build (accepts only none|deepseek|deepseek-legacy).
Falling back to deepseek, which also populates reasoning_content — the
field Worldtree's GemmaProvider fallback path checks. Verified via test
inference: 608 reasoning_content deltas + 199 content deltas + 0 raw
<|channel> marker leaks.
Made the host-stacks bind-mount path configurable via
DOCKGE_HOST_STACKS_ROOT (default /opt/docker, unchanged for the
existing five hosts). Override on corviduo-dev to /home/vh/docker
because that host's /opt/ is owned by deploy:deploy (Worldtree team)
and vh lacks passwordless sudo for the fleet-standard path — same
reasoning as the beszel + dozzle agent placement earlier today.
Deployed to corviduo-dev. Reachable at http://10.250.50.152:5001
(first probe 200 — Docker's port-mapping route through iptables
worked without firewall changes, unlike beszel's network_mode: host).
Scoped to PFI-managed stacks only (/home/vh/docker/compose/) — does
NOT see /opt/worldtree*/ deployments. Keeps the management boundary
clean: dockge can restart/recreate PFI's beszel+dozzle+itself but
not the Worldtree-team-owned containers.
Operator-approved fleet monitoring extension. Both agents up + healthy:
- beszel-agent on host port 45876 (KEY-mode, hub at ana-docker:8090
SSH-polls inbound; seeded with hub's ed25519 pubkey).
- dozzle-agent on host port 7007 (mTLS auto-generated; hub at
ana-docker:8088 connects inbound).
Compose lives at /home/vh/docker/compose/{beszel,dozzle-agent}/ rather
than the fleet-standard /opt/docker/compose/{...}/ because corviduo-dev's
/opt/ is owned by deploy:deploy (Worldtree team) and vh lacks
passwordless sudo to create the fleet path. Functionally identical;
documented in servers/corviduo-dev/README.md so future infra-ops
sessions find them.
Created an empty traefik-net external docker network on corviduo-dev
as a side effect of dozzle-agent's compose (which declares it external).
Future PFI services landing here can reuse it.
Dozzle hub on ana-docker had 10.250.50.152:7007 appended to
DOZZLE_REMOTE_AGENT (a host-side change to a non-tracked .env, not
canonical). Beszel hub still needs corviduo-dev added via the UI's
"Add System" action — one-time operator step, flagged in the corviduo
README.
Temporary diagnostic for the class of bug story 83ff386d47c6 hit
2026-05-23: POST /generation/start returned 202, then total silence
— no log, no DB state update, py-spy showed event loop idle with no
GenerationRunner frame anywhere. Strongly suggests a created_task()
result not held → GC'd → silent destroy.
PYTHONASYNCIODEBUG=1 emits "Task was destroyed but it is pending"
and "Task exception was never retrieved" warnings to stderr; that
should distinguish lost-task from cancelled-task on the next attempt.
Per skaldsong-dev's note, remove once they wire proper task-exception
capture upstream.
Diagnosis thread: althing 01KSBGKQBXA756JWW1KD4MPGXM
The compose set SKALDSONG_DB_PATH + SKALDSONG_RUNS_DIR, but skaldsong's
app reads SKALDSONG_HOST_SQLITE_PATH + SKALDSONG_HOST_RUNS_ROOT (per
its Dockerfile ENV defaults). Our values were orthogonal — the app
fell back to Dockerfile defaults pointing at /app/data/... which is
NOT bind-mounted, so every --force-recreate wiped the SQLite DB +
runs/ tree along with the ephemeral container layer.
Surfaced by skaldsong-dev (althing thread 01KS4DPF6SXTBP4Q360JZVWPNT)
after the operator noticed stories vanishing on every deploy.
Confirmed on ana-docker: container had a 40KB skaldsong-ui.db sitting
in /app/data/, while /opt/docker/conf/skaldsong/db/ on the host was
empty. Rescued the live DB to the bind-mount target before recreate.
Fix: rename env vars to match what the app reads. Bind targets stay
at /app/state/{db,runs} (parent-dir mount for SQLite WAL+SHM).
Two corrections surfaced by the first end-to-end deploy that didn't
land in the pre-flight align:
- SPA static assets are at /app/spa, not /app/web/dist (Dockerfile
COPYs the SvelteKit build output flat into /app/spa, not into
/app/spa/dist). Mismatch caused /health to 500 with
"RuntimeError: File at path /app/web/dist/index.html does not
exist."
- SKALDSONG_HOST_CORS_ORIGINS must be a JSON array literal in .env.
Pydantic-settings parses complex-typed env vars via json.loads();
bare URL string fails first-boot with SettingsError.
Container now reports Up (healthy) on ana-docker; /health 200.
skaldsong-dev surfaced three contract corrections before the first
deploy:
- WORLDTREE_TOKEN (outbound HTTP Bearer) was missing — separate code
path from SKALDSONG_BIFROST_JWT_KEY (inbound HS256 verify) but
same secret value.
- WORLDTREE_BASE_URL replaces SKALDSONG_WORLDTREE_API_URL (the
former is what the app actually reads).
- SKALDSONG_HOST_WIZARD_AGENT_ID was missing entirely — must pin to
skaldsong:wizard-v2 to inherit the existing Worldtree agent slot;
blank would burn another slot of the 50-per-key Heimdall quota.
Registry-pull pattern matching Worldtree: CI on vh/skaldsong builds and
pushes gitea.phasefinal.com/vh/skaldsong:<sha>, this playbook pulls +
recreates. SHA-pin only per current preference; no :latest moving-tag
advance yet (revisit once /health exercises Worldtree + Kokoro reach).
Host port 8300 (host) → 8000 (container). Persistent state under
/opt/docker/conf/skaldsong/{db,runs}.
Bifrost endpoint URL 10.250.50.70:8300 will need a paired
BIFROST_CLIENT_ALLOWED_HOSTS update on corviduo-dev Worldtree at first
deploy.
Phase 3.1 closes the cross-process gap the Phase 3 smoke surfaced —
streaming events (msg_start/thinking/delta/complete/curated) flow
from agent-runner → chamber via valkey pub/sub rather than the
SQLite bridge (too high-volume + ephemeral for the DB).
New service: `althing-valkey` (stock `valkey/valkey:8-alpine`).
Internal-only — no exposed port, no volume. chamber + agent-runner
reach via docker DNS at `valkey:6379` on the compose default
network. healthcheck via `valkey-cli ping` (5s interval). chamber
+ agent-runner gain `depends_on: valkey: service_healthy` so the
bridge is up before either side starts publishing or subscribing.
Forseti unchanged — never publishes Phase 3 events.
Operational properties (per forseti's deployment notes):
- Mixed-state safe at every step. Missing valkey.url config key
→ chamber + runner stay on v3.0 / Phase 2 equivalent paths.
- Backward path is single config-key delete + restart.
- streaming_enabled: true (set on agent-runner 2026-05-17) is
unaffected by this change.
README's services table + playbook header + verify section all
extended to reflect the four-service shape. Forseti's contract
at vh/althing:docs/contracts/phase3_1_valkey_bridge.contract.md
carries the wire-protocol spec.
Phase 2 daemon added to the althing-chamber stack per forseti's request
(vh/althing@5cd088a..ad1d025). Polls floor_grants WHERE consumed_at IS
NULL AND agents.driver='worldtree', claims via atomic UPDATE, calls
Worldtree's conversation API, posts the response back through the bus
as a broadcast.
Shape matches the existing forseti daemon:
- Same ${ALTHING_IMAGE} (the binary is already in [project.scripts]
as of ad1d025)
- command: ["althing-agent-runner"]
- Same shared SQLite bind-mount at /app/data
- No port, no healthcheck (CLI doesn't expose one; same liveness
story as forseti)
Safe to enable preemptively per forseti — when no driver=worldtree
handles are declared in config, the runner sleeps at
poll_interval_seconds. Multi-instance safe via the atomic claim
primitive (no flock needed).
Compose top comment, README "Services in this stack" table, playbook
header + verify steps all extended to reflect the three-service
shape. Will land on ana-docker on vh/althing's next push (compose
deployed via the elway playbook's upload step; image already carries
the binary).
Two-service compose (chamber + forseti sidecar daemon) sharing a single
SQLite store via bind-mount under /opt/docker/conf/althing-chamber/data.
eventbus.bridge_from_db is the cross-process glue — forseti's commits
reach chamber's SSE subscribers via the bridge.
Pattern matches task-board's build-on-host deploy:
- elway playbook clones vh/althing into /opt/docker/build/
- docker build -t althing-chamber:local . (no registry)
- playbook uploads compose + seeds .env one-time, brings both
services up, polls /health
- Gitea Actions workflow lives in vh/althing; reference copy here.
Internal tooling — host port 7881 (chamber's default of 7878 collides
with task-board). LAN-direct, no Traefik. Container always listens on
8000 internally.
Scaffold will fail to bring the chamber container up healthy until
galdrabok-side commits land:
- Dockerfile at vh/althing repo root (two-stage: uv-bookworm-slim
build → python:3.12-slim runtime, locked per open_questions §2
of the v1 contract).
- GET /health endpoint on the chamber app (200, no DB read).
- ALTHING_BIND / ALTHING_PORT env-var support in
core.cli.chamber_serve / core.chamber.cli (env > config.yaml >
defaults precedence).
Coordinated via althing thread 01KRMAK7RD7TP6C8DF4KXV31RT.
Stack was retired and replaced by the vllm stack (originally vllm-qwen3,
renamed 2026-05-13). Its README still framed it as a current solution
while ana-ml2's README + vllm's README both documented the retirement.
stacks/vllm/README.md "Migrating off Infinity" step 3 explicitly said
"Delete stacks/infinity/ from this workspace" — actioning that now.
No backwards-compat shims (PRACTICES §4): contract of a deleted system
has no historical value the next contributor needs; the replacement
path is documented in stacks/vllm/README.md.
Surfaced by /tend-docs audit 2026-05-14.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Playbook handles models, datasets, and spaces (via --var repo_type=...)
since 3025d49 — the "-model" suffix was misleading. Renaming to match
actual scope.
Updates the single in-repo reference (changelog comment in
stacks/llama-swap/conf/config.yaml). config.yaml was scp'd to ana-ml2;
no docker compose restart needed (comment-only).
AtlaAI's Selene-1-Mini judge model for evaluation/scoring tasks.
Llama 3.1 8B base, mradermacher imatrix-quantized Q6_K (~6.5GB,
quality-leaning quant). Apache-2.0. Per Atla cookbook these defaults
hit 84% on RAGTruth hallucination eval.
New 'JUDGE / EVAL MODELS' section between the dense chat models and
the embedding models — separate category from chat/reasoning since
the run-params shape is different (deterministic-leaning: temp 0.01,
top-p 1.0, no repeat penalty).
q8_0 KV cache to fit 32K ctx cleanly on the 3090 with headroom.
Pre-pulled into the shared HF cache via the new
playbooks/pull-hf-model.yaml playbook (canonical replacement for
ad-hoc huggingface_hub.snapshot_download calls; see CHANGELOG).
Smoke-tested 2026-05-13: GET /v1/models lists selene-1-mini-8b,
POST /v1/chat/completions returns expected output cleanly.
Three changes prepping infra for asset_engine's orchestration feature
(SSH-driven bring-up / bring-down of irv-ml1 inference services with
per-device VRAM gating, contract in vh/asset-engine commit 5a36f8c):
1. asset-engine compose + .env.example + playbook gain a read-only
bind-mount for /app/runtime/ssh — the dedicated ed25519 keypair
(generated on ana-docker, not in the repo) plus a pinned known_hosts
for irv-ml1's host fingerprint. Env vars SSH_KEY_PATH and
SSH_KNOWN_HOSTS are exposed for the app to consume.
2. docs/asset-engine/services.yaml gains a `lifecycle: { stack, vram_gb,
gpu_device_id }` block on each of 12 orchestratable irv-ml1 services
(kokoro, chatterbox, index-tts, qwen3-tts, cosyvoice, fish-s2,
kyutai-tts, vibevoice, voxtral, parakeet, stable-audio-open, ace-step).
VRAM numbers are estimates from model footprint at fp16 — tune from
real nvidia-smi measurements once the gate is live. comfyui and
kokoro-captioned are deliberately excluded (variable-VRAM and
shared-container respectively).
3. servers/irv-ml1/README.md docker-stacks table now lists all 13
inference stacks (was only dockge + agents + comfyui) with port +
GPU pinning columns.
Pubkey deployed to ~lkraven/.ssh/authorized_keys on irv-ml1;
end-to-end SSH from ana-docker → irv-ml1 verified with strict
host-key checking.
Adds VOR_WORLDTREE_KEY + VOR_WORLDTREE_BASE + VOR_WORLDTREE_MODEL to vor's
compose environment with sane defaults. Empty key falls back to the
in-process MockWorldtree (the /mockup/ surface returns canned fixtures);
a real key issued by architect routes LLM calls at the demo Saga instance.
Key itself lives in ana-docker:/opt/docker/compose/vor/.env (not in the
repo).
Internal tooling — accessed at http://10.250.50.70:8200, not through
Traefik. Removes the unused traefik labels (router rule, TLS, crowdsec
middleware, loadbalancer port) and the traefik-net network membership;
homepage.href now points at host:port for direct discovery, matching
task-board's pattern. Playbook verify drops the traefik-net membership
check.
vh