Commit Graph

252 Commits

Author SHA1 Message Date
vh 3b54519d60 memory: correct Fish-cloning finding — Fish clones competently (ECAPA 0.79); 'not British' was the undefined bug, not Fish/transcript 2026-06-01 15:26:58 -07:00
vh 7f9dc2dcf1 memory: snapshot — CSM teardown, Chatterbox v2 + Fish v6 reworks, undefined-bug root cause, dia2 restart; archive 8 (2026-05-27) entries 2026-06-01 15:07:55 -07:00
vh 38d9e3b7aa catalog(fish-s2): drop Imogen_Contralto (pitch-shift variants abandoned)
Pitch-shifted deepening (rubberband, formant on/off) sounded bad at every
depth (tuba / over-gravelly), so abandoned. Removed Imogen_Contralto from the
dropdown + deleted the staged variants (fish + chatterbox). Plain unmodified
Imogen remains. version 5->6.
2026-06-01 14:40:09 -07:00
vh 43e9616dd1 catalog(fish-s2): add Imogen_Contralto to voice dropdown
Imogen pitch-shifted -3 semitones (rubberband, formants follow) -> deeper
contralto register. Staged + verified cloning; added to reference_id options
(33 voices). version 4->5.
2026-06-01 13:34:29 -07:00
vh f52558dc0d catalog(fish-s2): add 3 British-female VCTK voices (Imogen/Eleanor/Beatrice)
Staged consenting VCTK Southern-England female speakers (p225/p228/p229, CC BY
4.0) as subtle-British-accent clone voices — repurposed from the on-host kyutai
tts-voices cache. Named neutrally; NOT modeled on or representing any public
figure. Added to the reference_id dropdown (32 voices total). version 3->4.
2026-06-01 13:24:50 -07:00
vh 284ec5b4c4 catalog(fish-s2): voice dropdown via reference_id (29 staged voices)
Staged 28 single-speaker dia voices + glados into /worktank/fish-s2/references/
(internal research use). Discovered the path-form references shape 500s on this
build; reference_id (by name) is the working voice path (verified live). So:
reference_id -> select 'Voice' with the 29 staged names (default Emily, female);
references demoted to advanced inline-base64 custom clones with the path->500
caveat documented. vram 10->24 (measured ~25GB in use); version 2->3.
2026-06-01 13:15:15 -07:00
vh f3c260e010 catalog(fish-s2): expose full control surface (was text + references only)
Fish-S2 rendered as an essentially blank form — only text + a references JSON
blob — despite being the fleet's richest-control engine. Expose the real
fish-speech ServeTTSRequest levers: temperature/top_p/repetition_penalty,
latency (normal|balanced), seed, format (wav/mp3/opus), + advanced cloning
(references/reference_id) and chunking (max_new_tokens/chunk_length/normalize/
use_memory_cache). Defaults from upstream schema.py, verified live 2026-06-01
(no /openapi.json; Kui server). Sections basic/sampling/advanced.

Also corrected: seedable false->true (/tts has seed); cold_start 8s->240s
(measured compile warmup); vram 4->10GB (~9GB BF16 weights); dropped the
misleading '~150ms TTFB' for honest ~realtime throughput. version 1->2.
2026-06-01 01:41:19 -07:00
vh 93cedd6951 catalog(chatterbox): route to /tts, expose emotion levers + sane defaults
Chatterbox was producing poor output because the catalog pointed at the thin
OpenAI /v1/audio/speech endpoint, which exposes none of Resemble's emotion/
pacing knobs — and the devnen server's shipped default exaggeration is 1.3
(tuned for its theatrical demo presets), which over-acts.

Re-point to the wrapper's richer /tts and expose the real control surface
(exaggeration, cfg_weight, temperature, speed_factor, seed, voice_mode),
mirroring the sibling dia stack (same devnen author). Defaults sourced live
2026-06-01: exaggeration + cfg_weight = 0.5 (Resemble README 'works well for
most prompts'), temperature 0.8 / speed 1.0 / seed 0 (server generation_
defaults). The shipped 1.3 exaggeration is deliberately NOT adopted.

Voices: expose the 28 built-in predefined voices via /get_predefined_voices
(default Emily.wav, the server default_voice_id) + clone via /get_reference_
files — replacing the wrong 'OpenAI aliases only' claim. Corrected seedable:
false -> true (/tts has seed) and image_tag_mutable -> true (:latest). Bumped
service version 1 -> 2 (breaking field-shape change); status down -> ready
(live + healthy). catalog_version unchanged (no new field types).
2026-06-01 01:18:59 -07:00
vh c54ab13d3c catalog: remove Sesame CSM stack — open csm-1b verified not deploy-worthy
Tear down the parked CSM stack (status: down, never successfully built).
Bring-up attempts failed at the image build: upstream
phildougherty/sesame_csm_openai pins no huggingface_hub version, which now
resolves to 1.17.0 where the `huggingface-cli` the Dockerfile relies on has
been removed (replaced by `hf`). Building would require vendoring + patching
the upstream Dockerfile.

Deep-research verdict (primary + community sourced) confirmed it isn't worth
that: the acclaimed Maya/Miles demo runs a fine-tuned, larger CSM variant
Sesame never open-sourced; the open csm-1b is the un-fine-tuned 1B base
(only the smallest of 1B/3B/8B shipped, no newer checkpoint as of mid-2026).
Ships no usable voices, can't generate text, English-only, can't stream
real-time out of the box; absent from current TTS leaderboards and dominated
by Kokoro/Dia2/Fish-S2/IndexTTS for narration.

Removes: stacks/csm/, playbooks/deploy-csm.yaml, the csm catalog service +
reproducibility_audit entries. Host state (compose dir, /worktank/csm) torn
down on irv-ml1; no container/image existed.
2026-06-01 00:58:56 -07:00
vh b74da99634 memory: snapshot — Dia2 two-instance deploy, /tts rework, zonos adapter
Capture this session's durable state: Dia2 live as dia2-2b/dia2-1b (legacy
dia retired), dia2 catalog on /tts with full controls + stable-voice default,
zonos REST adapter (built, stack down), schema regen, all pushed. New
foot-guns: dia2 image-build quirks, predefined-voices-not-in-/tts, voice_mode
clone 'undefined' 404. Archived 6 older entries (4 decisions, 2 foot-guns).
2026-05-31 22:16:40 -07:00
vh 5c4784339c catalog(dia2): default to a working stable voice (fix undefined-ref 404)
Reported: dia2-1b 404 'Reference audio file not found: undefined' when
accepting defaults — voice_mode=clone with no clone_reference_filename made
the UI submit the literal 'undefined'. Fix: voice_mode now defaults to clone
AND clone_reference_filename defaults to Abigail.wav (a staged voice), so the
out-of-box request is a valid, stable voice. Reproduced the 404 and verified
the Abigail.wav path returns 200. Folded into v2 (not yet consumed downstream).
2026-05-31 15:27:56 -07:00
vh 55602b7251 catalog(dia2): expose full /tts control surface + stable predefined voices
Repoint both dia2 entries from /v1/audio/speech to the wrapper's richer /tts
endpoint (CustomTTSRequest), exposing the levers that fix the random-voice
problem: voice_mode, clone_reference_filename, cfg_scale, temperature, top_p,
cfg_filter_top_k, speed_factor, seed, split_text, chunk_size, transcript,
max_tokens. All defaults are the wrapper's Pydantic blessed values (cfg 3.0 /
temp 1.3 / top_p 0.95 / top_k 35 / speed_factor 0.94 / chunk 300). Fields
grouped (basic/sampling/advanced). dia2 -> version 2 (field-shape change).

Voice stability: Dia2 samples a random speaker per call unless anchored. The
43 curated voices baked at /app/voices aren't reachable from /tts's clone path
(reference_audio dir only), so they're staged into reference_audio; the
clone_reference_filename picker now sources /get_reference_files. voice_mode=
clone + a reference filename pins voice/gender. Verified /tts clone end-to-end
(HTTP 200, Ogg/Opus 24 kHz). README documents the staging + two-instance shape.
2026-05-31 15:18:53 -07:00
vh e97d80cb8e catalog: re-source today's engine defaults from canonical upstreams
Replace the invented sample inputs / topk with canonically-sourced values
(per CATALOG-CONTRACT source precedence + cite-your-source rule), inline-cited:
  * dia2-2b/dia2-1b input -> '[S1] Hello Dia2!'  (nari-labs/dia2 README example)
  * zonos input -> 'Zonos uses eSpeak for text to phoneme conversion!'
      (upstream gradio_interface.py blessed Textbox default)
  * csm input -> 'Hello from Sesame.'  (sesame/csm-1b model card example)
  * csm topk -> default REMOVED (the card recommends no top-k default; the
    wrapper's internal default applies when omitted).
2026-05-31 14:52:16 -07:00
vh 373a0046b1 catalog: fill sensible field defaults for today's engines
Pre-fill the bare input textareas with format-demonstrating samples
(dia2-2b/dia2-1b: [S1]/[S2] dialogue + a nonverbal; csm: conversational;
zonos: expressive multilingual) and give csm.topk a standard default (50).
Addresses asset-engine-dev's best-practice-defaults request so the picker
forms pre-fill usefully instead of empty. No version bumps — these entries
ship fresh in this batch (no prior consumer-pinned shape).
2026-05-31 14:46:12 -07:00
vh db15638d8e catalog: replace dia with dia2-1b/dia2-2b; move zonos-api 8201->8203
Replace the single dia entry (legacy Dia 1.6B, retired) with two fixed-model
Dia2 entries (dia2-2b :8200, dia2-1b :8202), status ready (both exercised),
image local/dia:v2. Matching reproducibility_audit rows. catalog_version
unchanged (add/remove services = no vocab change).

Also fix a port collision I introduced earlier: the zonos-api adapter and
csm both claimed 8201 — move zonos-api to 8203 (catalog endpoint + voices
source_url, zonos .env.example, README).

NOTE FOR CONSUMERS: removing the dia id is a breaking catalog change for
asset_engine (it vendored dia in v0.1.4) — re-vendor + drop the dia tile,
add the two dia2 tiles.
2026-05-31 14:41:11 -07:00
vh 3139e81e8a feat(dia): two-instance Dia2 deployment (dia2-1b + dia2-2b)
The devnen wrapper is single-model and ignores the OpenAI model field, so
offering both Dia2 models to asset-engine as real per-request choices means
one fixed-model instance per model. Rework the dia stack to run two services
from a dia2-capable image:
  * dia2-2b (:8200, best quality), dia2-1b (:8202, streaming) — both GPU 0
  * each pins its model via a mounted /opt/docker/conf/dia2-*/config.yaml
Retire the legacy Dia 1.6B service.

New dia2-image/Dockerfile builds local/dia:v2 = upstream devnen wrapper +
the dia2 package (copied into site-packages; its pyproject build backend
yields an empty UNKNOWN wheel under the base's old setuptools) + the three
missing runtime deps (transformers/sphn/whisper-timestamped); torch 2.12 /
numpy 2.2 in the base already satisfy Dia2. Both instances verified
end-to-end (HTTP 200, Ogg/Opus 24 kHz).
2026-05-31 14:41:11 -07:00
vh 02b26f45a6 catalog: add zonos service entry; fix dia voice picker options
zonos: new tts entry routing to the REST adapter on 8201, JSON-envelope
response with reproducibility.seed_field=seed (seedable+deterministic),
lifecycle block, section groups. Matching reproducibility_audit row.

dia: voice select had a default (S1) but no options/source_url -> empty
picker; add options [S1, S2, dialogue] (per asset-engine-dev), bump dia
to version 2. Closed select drops clone-by-filename free-text; flagged
inline for a future source_url-backed picker. catalog_version unchanged
(add-service + field-options = no vocab change).
2026-05-31 13:59:10 -07:00
vh b7b2130784 catalog-schema: add CatalogLifecycle + reproducibility.seed_field
Resolves the stale-schema gap asset-engine-dev flagged (the published
schema rejected the lifecycle field 12/14 live services already use) and
adds reproducibility.seed_field so a seedable engine can declare which
response key carries the seed used. Authoritative regen path remains
dump_schema.py against asset_engine/catalog.py; reconcile there.
2026-05-31 13:59:10 -07:00
vh 81efa8da96 feat(zonos): OpenAI-ish REST adapter for asset-engine routing
Upstream Zonos ships only Gradio + Python SDK — no REST surface — so
asset-engine (which routes a clean JSON POST to /v1/audio/speech) can't
target it directly. Add a thin FastAPI adapter (stacks/zonos/adapter/):
POST /v1/audio/speech in front of the Zonos SDK, built FROM local/zonos
to reuse torch/CUDA/SDK. Returns a JSON envelope {audio, audio_format,
seed} — the seed rides back so asset-engine regenerate/fork can pin it
(Zonos is the fleet's first genuinely seedable TTS). compose gains a
zonos-api service on 8201; .env.example gains the port + voices dir.
2026-05-31 13:59:10 -07:00
vh 71df6f7474 fix(zonos): add missing gradio launch command
Upstream Zyphra/Zonos ships no CMD in its Dockerfile (it launches the
app from its own compose), so our container ran the NVIDIA entrypoint,
printed the CUDA banner, exited 0, and restart-looped — nothing ever
bound 7860/8199. Add command: python3 gradio_interface.py to match
upstream, plus an explicit GRADIO_SHARE=False. Built + deployed to
irv-ml1; 8199 now serves HTTP 200 and the transformer model loads.
2026-05-31 13:34:33 -07:00
vh 8eac39b6ce catalog: add dia + csm service entries (status: down) + repro audits
dia (:8200) — OpenAI-compat /v1/audio/speech, seedable (not byte-exact),
Apache-2.0 weights. Clean catalog fit; flip to ready after first exercise.

csm (:8201) — OpenAI-compat, but NO seed + temperature-sampled =
non-reproducible (contract's fix-before-adding case), catalogued by
operator direction with a reproducibility caveat + gated-license warning;
belongs at experimental once running.

Fields read from each wrapper's API docs (2026-05-31), to confirm against
live OpenAPI/Pydantic at deploy. No catalog_version bump (add-service =
no bump). NOTE: services.schema.json is stale (pre-existing — 13 errors;
live catalog uses lifecycle, schema predates it); regen via dump_schema.py.
2026-05-31 11:18:46 -07:00
vh a4b8c2a9f4 stacks: add csm (Sesame Conversational Speech Model) to the TTS bench
Sesame CSM-1B via phildougherty/sesame_csm_openai — OpenAI-compat
/v1/audio/speech, context-aware conversational speech (voice-agent
layer, not a plain reader). Port 8201 on irv-ml1. Gated model:
requires CSM_HF_TOKEN (license acceptance) — placeholder in .env.example,
real token only in host .env.
2026-05-31 10:59:22 -07:00
vh 666f7f31e0 stacks: add dia + zonos to the TTS bench
dia: Nari Labs dialogue TTS (Dia 1.6B / Dia2-1B / Dia2-2B) via
devnen/Dia-TTS-Server — OpenAI-compat, fills the multi-speaker
dialogue-scene slot for skaldsong. Port 8200 on irv-ml1.

zonos: Zyphra Zonos-v0.1 (Apache-2.0, 44kHz, emotion sliders) via the
official Gradio interface. Audition surface only — no OpenAI-compat
endpoint yet (needs the FastAPI fork to become skaldsong-pluggable).
Port 8199 on irv-ml1.

Both follow the chatterbox/fish-s2 convention: local image built from a
pinned wrapper SHA via buildx git-context, .env-driven port/GPU, python
healthcheck, homepage labels.
2026-05-31 10:53:27 -07:00
vh 53157b193d playbooks: fix drop-in filename collision in esh NFS boot-ordering
override.conf already existed on esh-docker-vm (dockerd ExecStart), so
the creates: guard skipped the step and the After=remote-fs.target
directive was never written. Use a distinct filename
(10-after-remote-fs.conf) that systemd merges alongside override.conf,
and guard on content rather than file existence so a stale file can't
silently skip.
2026-05-30 14:55:31 -07:00
vh c0458d974d playbooks: fix esh-docker-vm NFS boot ordering (paperless exit-255 on reboot)
Adds _netdev,nofail to the 4 10.0.50.50 NFS lines in /etc/fstab and a
docker.service drop-in (After=remote-fs.target) so Docker waits for the
NFS mounts before starting NFS-bind containers. Root cause of paperless
Exited(255) on every reboot. Idempotent; backs up fstab; no reboot.
2026-05-30 14:45:56 -07:00
vh c3aae2e435 skaldsong: declare VibeVoice TTS env vars for issue #73 dual-engine reader 2026-05-27 20:41:41 -07:00
vh 4a4a6fc803 llama-swap: artemis-31b-v1i — add --reasoning on + deepseek format
Worldtree-dev's --reasoning-format gemma suggestion isn't supported in
the deployed llama.cpp build (accepts only none|deepseek|deepseek-legacy).
Falling back to deepseek, which also populates reasoning_content — the
field Worldtree's GemmaProvider fallback path checks. Verified via test
inference: 608 reasoning_content deltas + 199 content deltas + 0 raw
<|channel> marker leaks.
2026-05-27 00:06:54 -07:00
vh 7f918271ac llama-swap: add artemis-31b-v1i (BeaverAI Gemma 4 fine-tune, Q6_K) 2026-05-25 22:42:25 -07:00
vh 8db180885f skaldsong: drop PYTHONASYNCIODEBUG=1 — SSE-wedge diagnostic expired 2026-05-25 00:43:17 -07:00
vh 532356fcbe corviduo-dev: log Worldtree v0.25.2 config-as-bind-mount landing (#205) 2026-05-24 20:28:46 -07:00
vh 5528769211 dockge: parameterize stacks-root path + deploy on corviduo-dev
Made the host-stacks bind-mount path configurable via
DOCKGE_HOST_STACKS_ROOT (default /opt/docker, unchanged for the
existing five hosts). Override on corviduo-dev to /home/vh/docker
because that host's /opt/ is owned by deploy:deploy (Worldtree team)
and vh lacks passwordless sudo for the fleet-standard path — same
reasoning as the beszel + dozzle agent placement earlier today.

Deployed to corviduo-dev. Reachable at http://10.250.50.152:5001
(first probe 200 — Docker's port-mapping route through iptables
worked without firewall changes, unlike beszel's network_mode: host).

Scoped to PFI-managed stacks only (/home/vh/docker/compose/) — does
NOT see /opt/worldtree*/ deployments. Keeps the management boundary
clean: dockge can restart/recreate PFI's beszel+dozzle+itself but
not the Worldtree-team-owned containers.
2026-05-23 21:23:44 -07:00
vh 1cd79e8274 corviduo-dev: deploy beszel + dozzle monitoring agents
Operator-approved fleet monitoring extension. Both agents up + healthy:
- beszel-agent on host port 45876 (KEY-mode, hub at ana-docker:8090
  SSH-polls inbound; seeded with hub's ed25519 pubkey).
- dozzle-agent on host port 7007 (mTLS auto-generated; hub at
  ana-docker:8088 connects inbound).

Compose lives at /home/vh/docker/compose/{beszel,dozzle-agent}/ rather
than the fleet-standard /opt/docker/compose/{...}/ because corviduo-dev's
/opt/ is owned by deploy:deploy (Worldtree team) and vh lacks
passwordless sudo to create the fleet path. Functionally identical;
documented in servers/corviduo-dev/README.md so future infra-ops
sessions find them.

Created an empty traefik-net external docker network on corviduo-dev
as a side effect of dozzle-agent's compose (which declares it external).
Future PFI services landing here can reuse it.

Dozzle hub on ana-docker had 10.250.50.152:7007 appended to
DOZZLE_REMOTE_AGENT (a host-side change to a non-tracked .env, not
canonical). Beszel hub still needs corviduo-dev added via the UI's
"Add System" action — one-time operator step, flagged in the corviduo
README.
2026-05-23 18:13:59 -07:00
vh 92554221be skaldsong: enable PYTHONASYNCIODEBUG to surface orphaned-task warnings
Temporary diagnostic for the class of bug story 83ff386d47c6 hit
2026-05-23: POST /generation/start returned 202, then total silence
— no log, no DB state update, py-spy showed event loop idle with no
GenerationRunner frame anywhere. Strongly suggests a created_task()
result not held → GC'd → silent destroy.

PYTHONASYNCIODEBUG=1 emits "Task was destroyed but it is pending"
and "Task exception was never retrieved" warnings to stderr; that
should distinguish lost-task from cancelled-task on the next attempt.

Per skaldsong-dev's note, remove once they wire proper task-exception
capture upstream.

Diagnosis thread: althing 01KSBGKQBXA756JWW1KD4MPGXM
2026-05-23 16:04:55 -07:00
vh 307b01e8a1 snapshot: capture skaldsong env-var-name footgun (CD wiping state)
Same lesson family as the /app/web/dist mismatch — encoding
container-internal contract (paths OR env var names) in compose
needs to be verified against the Dockerfile + app, not against
design-doc shorthand. Wrong env var names silently no-op; app
falls back to Dockerfile defaults which orthogonally miss the
bind mount, and state goes to ephemeral layer until next recreate.
2026-05-20 21:57:51 -07:00
vh 52e98fa2d0 skaldsong: fix CD-wipes-state bug (env var names didn't match app)
The compose set SKALDSONG_DB_PATH + SKALDSONG_RUNS_DIR, but skaldsong's
app reads SKALDSONG_HOST_SQLITE_PATH + SKALDSONG_HOST_RUNS_ROOT (per
its Dockerfile ENV defaults). Our values were orthogonal — the app
fell back to Dockerfile defaults pointing at /app/data/... which is
NOT bind-mounted, so every --force-recreate wiped the SQLite DB +
runs/ tree along with the ephemeral container layer.

Surfaced by skaldsong-dev (althing thread 01KS4DPF6SXTBP4Q360JZVWPNT)
after the operator noticed stories vanishing on every deploy.

Confirmed on ana-docker: container had a 40KB skaldsong-ui.db sitting
in /app/data/, while /opt/docker/conf/skaldsong/db/ on the host was
empty. Rescued the live DB to the bind-mount target before recreate.

Fix: rename env vars to match what the app reads. Bind targets stay
at /app/state/{db,runs} (parent-dir mount for SQLite WAL+SHM).
2026-05-20 21:55:48 -07:00
vh 1a0059f589 snapshot: roll persistent-memory.md from 2026-05-17 → 2026-05-20
Captures the May 18-20 session arc:

- Skaldsong CD shipped on ana-docker:8300 via registry-pull pattern
  (differs from althing/asset-engine which build-on-host); three
  first-deploy footguns surfaced + canonical-patched.
- Worldtree CD disk-hygiene PR #184 (vh/Worldtree) — watermark gate,
  eager post-deploy SHA prune, set -e hardening in SSH blocks.
  Approved by worldtree-dev, pending merge.
- mead-hall Bifrost v0.3 wire-layer + data-flow end-to-end smoke
  green (task #32 closed) — resolves the "stalled mid-Worldtree"
  state in the prior snapshot.
- Volva systemd install — three-stage debug (PATH, ReadWritePaths,
  handle pinning); impersonation footgun caught via PID/journal
  cross-reference.
- Per-project user keys minted against personal Worldtree (:8081)
  for skaldsong, althing, mead-hall + skaldsong-prod. Personal
  becomes multi-consumer dev iteration instance; demo stays for
  isolation.

File now at 434 lines (over 300 soft cap); next /snapshot run will
surface archival candidates.
2026-05-20 21:51:36 -07:00
vh c6864996f0 skaldsong playbook: verify container is Up, not 'healthy' (was racing start_period)
The verify step ran 0.09s after `docker compose up -d --force-recreate`,
well before the container's 30s start_period elapsed — so 'healthy'
isn't yet in docker ps's Status string and the grep failed. False
negative; container was operationally up (verify 1/2 already confirmed
/health 200).

Greping for '^Up' instead catches the case verify 2/2 actually wants
to catch (container exited entirely) without racing the healthcheck.
2026-05-18 23:47:51 -07:00
vh 4b8dc9f7e1 skaldsong: fix first-deploy footguns (SPA path + cors_origins shape)
Two corrections surfaced by the first end-to-end deploy that didn't
land in the pre-flight align:

- SPA static assets are at /app/spa, not /app/web/dist (Dockerfile
  COPYs the SvelteKit build output flat into /app/spa, not into
  /app/spa/dist). Mismatch caused /health to 500 with
  "RuntimeError: File at path /app/web/dist/index.html does not
  exist."

- SKALDSONG_HOST_CORS_ORIGINS must be a JSON array literal in .env.
  Pydantic-settings parses complex-typed env vars via json.loads();
  bare URL string fails first-boot with SettingsError.

Container now reports Up (healthy) on ana-docker; /health 200.
2026-05-18 23:35:13 -07:00
vh 8a4276d6b5 skaldsong: align env-var contract with app reality (pre-first-deploy)
skaldsong-dev surfaced three contract corrections before the first
deploy:

- WORLDTREE_TOKEN (outbound HTTP Bearer) was missing — separate code
  path from SKALDSONG_BIFROST_JWT_KEY (inbound HS256 verify) but
  same secret value.
- WORLDTREE_BASE_URL replaces SKALDSONG_WORLDTREE_API_URL (the
  former is what the app actually reads).
- SKALDSONG_HOST_WIZARD_AGENT_ID was missing entirely — must pin to
  skaldsong:wizard-v2 to inherit the existing Worldtree agent slot;
  blank would burn another slot of the 50-per-key Heimdall quota.
2026-05-18 23:10:37 -07:00
vh 55e2e836a8 skaldsong: scaffold compose stack + deploy playbook for ana-docker
Registry-pull pattern matching Worldtree: CI on vh/skaldsong builds and
pushes gitea.phasefinal.com/vh/skaldsong:<sha>, this playbook pulls +
recreates. SHA-pin only per current preference; no :latest moving-tag
advance yet (revisit once /health exercises Worldtree + Kokoro reach).

Host port 8300 (host) → 8000 (container). Persistent state under
/opt/docker/conf/skaldsong/{db,runs}.

Bifrost endpoint URL 10.250.50.70:8300 will need a paired
BIFROST_CLIENT_ALLOWED_HOSTS update on corviduo-dev Worldtree at first
deploy.
2026-05-18 22:01:29 -07:00
vh 8f50d702d1 homepage: add Notes group to Main tab (4-column row) 2026-05-17 22:46:41 -07:00
vh 9e9bdf9810 deploy-stack: add -n to ssh calls to prevent stdin slurp in loops
Without -n, ssh inherits the surrounding loop's stdin and consumes
the heredoc that feeds $changed / $deleted, silently truncating the
diff output to the first file only.
2026-05-17 22:46:41 -07:00
vh 4b986f0b23 snapshot: roll persistent-memory.md from 2026-05-08 → 2026-05-17
Captures 9 days of multi-stack work:
  - asset-engine catalog (lifecycle blocks for orchestrator,
    defaults audit, voice probe of voxtral/qwen3-tts/kyutai-tts)
  - asset-engine stack scaffold + LAN-direct deploy
  - corviduo-dev inventory (Worldtree-team dev VM)
  - Worldtree outage recovery + :latest tag drift fix
  - vor key injection
  - vllm-qwen3 → vllm rename + vllm-reward (Skywork-Reward-V2 AWQ)
  - pull-hf-repo.yaml (model+dataset+space puller)
  - selene-1-mini-8b judge model in llama-swap
  - /tend-docs first pass (delete infinity, archive design-brief,
    fix pfi-pve VM list, drop stale postgres-rotate section)
  - althing-chamber stack (chamber + forseti + agent-runner + valkey
    across Phase 1/2/3.1)
  - Worldtree admin workflow shift (infra-ops permanent admin key
    replaces docker-as-root + bootstrap admin)
  - mead-hall Bifrost consumer registration
  - skaldsong CI bootstrap (3 captured gotchas)
  - Phase 3 + Phase 3.1 streaming activation

Sections grew: Tools and conventions (+sister-repo rows, admin auth
note, docker-as-root pattern), Recent decisions (+12), Tried and
abandoned (+5). Current state / in-flight rewritten to today's
posture (Worldtree handshake gap, hygiene-debt rotation list, prior
open items still active).

294 lines total; under the 300 soft cap. No archival this run.
2026-05-17 14:25:56 -07:00
vh 5704fbb08f althing-chamber: add valkey for Phase 3.1 cross-process streaming
Phase 3.1 closes the cross-process gap the Phase 3 smoke surfaced —
streaming events (msg_start/thinking/delta/complete/curated) flow
from agent-runner → chamber via valkey pub/sub rather than the
SQLite bridge (too high-volume + ephemeral for the DB).

New service: `althing-valkey` (stock `valkey/valkey:8-alpine`).
Internal-only — no exposed port, no volume. chamber + agent-runner
reach via docker DNS at `valkey:6379` on the compose default
network. healthcheck via `valkey-cli ping` (5s interval). chamber
+ agent-runner gain `depends_on: valkey: service_healthy` so the
bridge is up before either side starts publishing or subscribing.

Forseti unchanged — never publishes Phase 3 events.

Operational properties (per forseti's deployment notes):
  - Mixed-state safe at every step. Missing valkey.url config key
    → chamber + runner stay on v3.0 / Phase 2 equivalent paths.
  - Backward path is single config-key delete + restart.
  - streaming_enabled: true (set on agent-runner 2026-05-17) is
    unaffected by this change.

README's services table + playbook header + verify section all
extended to reflect the four-service shape. Forseti's contract
at vh/althing:docs/contracts/phase3_1_valkey_bridge.contract.md
carries the wire-protocol spec.
2026-05-17 08:36:46 -07:00
vh 59899e4e0f althing-chamber: add althing-agent-runner as third compose service
Phase 2 daemon added to the althing-chamber stack per forseti's request
(vh/althing@5cd088a..ad1d025). Polls floor_grants WHERE consumed_at IS
NULL AND agents.driver='worldtree', claims via atomic UPDATE, calls
Worldtree's conversation API, posts the response back through the bus
as a broadcast.

Shape matches the existing forseti daemon:
  - Same ${ALTHING_IMAGE} (the binary is already in [project.scripts]
    as of ad1d025)
  - command: ["althing-agent-runner"]
  - Same shared SQLite bind-mount at /app/data
  - No port, no healthcheck (CLI doesn't expose one; same liveness
    story as forseti)

Safe to enable preemptively per forseti — when no driver=worldtree
handles are declared in config, the runner sleeps at
poll_interval_seconds. Multi-instance safe via the atomic claim
primitive (no flock needed).

Compose top comment, README "Services in this stack" table, playbook
header + verify steps all extended to reflect the three-service
shape. Will land on ana-docker on vh/althing's next push (compose
deployed via the elway playbook's upload step; image already carries
the binary).
2026-05-16 20:01:57 -07:00
vh 91d5417b0b althing-chamber: scaffold deploy stack on ana-docker
Two-service compose (chamber + forseti sidecar daemon) sharing a single
SQLite store via bind-mount under /opt/docker/conf/althing-chamber/data.
eventbus.bridge_from_db is the cross-process glue — forseti's commits
reach chamber's SSE subscribers via the bridge.

Pattern matches task-board's build-on-host deploy:
  - elway playbook clones vh/althing into /opt/docker/build/
  - docker build -t althing-chamber:local . (no registry)
  - playbook uploads compose + seeds .env one-time, brings both
    services up, polls /health
  - Gitea Actions workflow lives in vh/althing; reference copy here.

Internal tooling — host port 7881 (chamber's default of 7878 collides
with task-board). LAN-direct, no Traefik. Container always listens on
8000 internally.

Scaffold will fail to bring the chamber container up healthy until
galdrabok-side commits land:
  - Dockerfile at vh/althing repo root (two-stage: uv-bookworm-slim
    build → python:3.12-slim runtime, locked per open_questions §2
    of the v1 contract).
  - GET /health endpoint on the chamber app (200, no DB read).
  - ALTHING_BIND / ALTHING_PORT env-var support in
    core.cli.chamber_serve / core.chamber.cli (env > config.yaml >
    defaults precedence).

Coordinated via althing thread 01KRMAK7RD7TP6C8DF4KXV31RT.
2026-05-14 15:54:19 -07:00
vh c27761f608 servers/pfi-pve: full VM/CT enumeration from live qm list
Previously a partial list "from vzdump logs + servers/ dirs". Replaced
with the full output of `qm list` on pfi-pve (2026-05-14), cross-
referenced against `servers/`. VMs that have a server dir are
annotated with the path; ones that don't are flagged "not yet in
`servers/` inventory" so the gap is discoverable.

Resolved:
- VM 100 = pbs-ana (was "VM (TBD)" in prior version)
- VM 106 = corviduo-dev (added 2026-05-12; was missing from the list)

Surfaced (new findings, not yet inventoried):
- VM 101 — PFI-ANA-DC (Active Directory domain controller)
- VM 103 — PFI-SlaveBot (purpose unclear from name alone)
- VM 104 — PFI-Mongo (MongoDB host, separate from pfi-postgres)

The three new-to-inventory VMs deserve `servers/<name>/` directories
with READMEs, but that's a follow-up — creating new inventory entries
is out of tend-docs scope.
2026-05-14 15:38:34 -07:00
vh 1f37857acc servers/pfi-postgres: drop stale "Known weak password (rotate)" section
Section claimed gitea + paperless-ng Postgres passwords were "currently
in use" with trivial values and that rotation was pending. Per
STATUS.md: "Rotate exposed secrets — done 2026-04-23. All six rotated:
vaultwarden/gitea/paperless-ng Postgres passwords (hardcoded
compose.yaml literals moved to gitignored .env files in the process)..."

Rotation happened; literals are no longer in compose.yaml; passwords are
no longer trivial. Removing the section rather than amending — once
fixed, there's no value in carrying a "we have weak passwords"
section that lies about the current state.

Surfaced by /tend-docs audit 2026-05-14.
2026-05-14 15:34:02 -07:00
vh 4e0059f388 servers/pfi-pve: pfi-postgres PGDATA is on local VM disk (not NFS)
VM 105's annotation said "PGDATA on NFS from ana-nas" — postgres
migrated off NFS to local VM disk on 2026-04-23 per STATUS.md ("DB data
on local disk, not NFS. pfi-postgres migrated 2026-04-23"). Updated to
reflect current reality with the migration date as the rot detector.

Still-TBD note for VM 33: `pbs-ana` is listed as "VM (TBD)" — was
deployed and has its own server dir, but its VM ID isn't recorded
here. `qm list | grep pbs` on pfi-pve would resolve, but my SSH to
pfi-pve as lkraven is currently password-required so I can't pull
this myself. Flag for next pfi-pve console session.

Surfaced by /tend-docs audit 2026-05-14.
2026-05-14 15:33:45 -07:00
vh 54d1e7137e docs: archive asset-engine design-brief.md
Moved docs/asset-engine/design-brief.md → docs/archive/asset-engine/design-brief.md
with a 12-line archival header pointing at the live implementation
artifacts (vh/asset-engine source, stacks/asset-engine/ deploy,
CATALOG-CONTRACT.md, services.yaml).

The brief explicitly framed itself as a pre-implementation handoff
("Hand this to a design agent before any pixels"). Implementation
shipped 2026-05-12; the brief's role is past. Kept for the design
rationale it carries (why Asset is first-class, why v1 is synchronous,
v2/v3 seam reasoning) — future contributors benefit from finding it
when wondering "why is it this way."

Surfaced by /tend-docs audit 2026-05-14.
2026-05-14 15:33:15 -07:00