From b74da996341dbbe21231fb9a3a9cb1e39354a3c2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 31 May 2026 22:16:40 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20Dia2=20two-ins?= =?UTF-8?q?tance=20deploy,=20/tts=20rework,=20zonos=20adapter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture this session's durable state: Dia2 live as dia2-2b/dia2-1b (legacy dia retired), dia2 catalog on /tts with full controls + stable-voice default, zonos REST adapter (built, stack down), schema regen, all pushed. New foot-guns: dia2 image-build quirks, predefined-voices-not-in-/tts, voice_mode clone 'undefined' 404. Archived 6 older entries (4 decisions, 2 foot-guns). --- archival-memory.md | 385 ++++++++++++++++++++++++++++ persistent-memory.md | 587 ++++++++++++++++++------------------------- 2 files changed, 632 insertions(+), 340 deletions(-) create mode 100644 archival-memory.md diff --git a/archival-memory.md b/archival-memory.md new file mode 100644 index 0000000..8489074 --- /dev/null +++ b/archival-memory.md @@ -0,0 +1,385 @@ +# Archival memory — eshpfi-management + +_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._ + +## Recent decisions (archived) + +- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152, + CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF + client hosts: PFI hosts + provides emergency-ops backstop; + Worldtree team owns OS config + deploys + backup decisions. + _Archived 2026-05-27._ + +- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by + health-gated `:latest` advance in vh/worldtree's deploy workflow + (architect commit `8ef3801`): only tag `:latest` AFTER the new + container's `/health` probe passes. Build-on-host stacks here + don't have this problem because the playbook always builds the + SHA-tagged image from a `git reset --hard ` checkout. + _Archived 2026-05-27._ + +- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at + `http://10.250.50.70:8200`. Initially included Traefik labels for + public hostname; user pulled them out (internal tool, no public + TLS surface needed). Pattern: internal tools default LAN-direct; + Traefik wiring only when external/TLS required. + _Archived 2026-05-27._ + +- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack, + vram_gb, gpu_device_id }` per irv-ml1 service for the + orchestrator feature. SSH keypair scaffolded at + `ana-docker:/opt/docker/conf/asset-engine/ssh/` for + asset-engine container → irv-ml1 orchestration via dedicated + ed25519 key. + _Archived 2026-05-27._ + +- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch + playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`. + Replaces ad-hoc `huggingface_hub.snapshot_download` calls. + _Archived 2026-05-27._ + +- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as + judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI + reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE / + EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`. + _Archived 2026-05-27._ + +- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added + `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier). + Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30 + utilization; 30% headroom). All use `--runner pooling`; classification + drives via model's `architectures: [LlamaForSequenceClassification]` + in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1). + _Archived 2026-05-27._ + +- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/` + removed (retired by vllm). Archived + `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/` + with archival header. Fixed `pfi-pve` VM list to full + `qm list` enumeration. Dropped stale weak-password section from + `pfi-postgres` (rotation done 2026-04-23). + _Archived 2026-05-27._ + +- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti. + Internal LAN-only at port 7881 (chamber default 7878 collides with + task-board). Two-service compose, shared SQLite bind-mount, + build-on-host pattern via vh/althing's gitea-workflow. Forseti is the + canonical dev for this stack (galdrabok is on a different project). + _Archived 2026-05-31._ + +- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner` + as third compose service (worldtree-driver agent dispatcher). + All three althing services use the same image; `command:` selects + entrypoint. Safe to enable preemptively (sleeps when no + driver=worldtree handles declared). + _Archived 2026-05-31._ + +- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8 + alpine as a sibling compose service in `stacks/althing-chamber/`, + redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` / + `msg_start` / `msg_complete` event kinds. DB bridge keeps + `msg_curated` + `floor_grant` (structured / canonical). Two-channel + architecture, no overlap. chamber + agent-runner `depends_on: + valkey: service_healthy`. + _Archived 2026-05-31._ + +- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops + gets its own permanent admin-tier key (`61419c92`, stored at + `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`). + Future admin ops route through this key, not the bootstrap admin + via docker-as-root. + _Archived 2026-05-31._ + +- `[2026-05-17]` Worldtree env-var addition checklist: anytime + introducing `os.environ.get("FOO")` in worldtree code, update BOTH + `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the + same PR. Same Z_AI_API_KEY-shape footgun bit + `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the + passthrough line in `08f02b2`. + _Archived 2026-05-31._ + +- `[2026-05-18]` Volva systemd install complete after three-stage + debug. Final unit at `/etc/systemd/system/volva.service` runs as + `User=lkraven` with `ProtectHome=read-only` + + `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` + carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/ + .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in + env.sh. + _Archived 2026-05-31._ + +- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate + (env-tunable threshold + window, fail-loud on still-low post-prune) + + eager post-deploy prune (only after `:latest` advance succeeds, + uses `docker image prune -a --filter "until=24h"` which respects + in-use semantic — protects pinned + personal images automatically). + Combined: demo VM holds ~24h of deploy history instead of unbounded + accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` + + `613dac2` + `bd91df5`). + _Archived 2026-05-31._ + +- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator + options — container + Gitea registry + pull-restart, matching + Worldtree's pattern. Target host ana-docker (NOT nh3-dev where + skaldsong-dev runs for iteration). SHA-pin only for now; health-gated + `:latest` advance is a follow-up once `/health` exercises Worldtree + + Kokoro reachability. + _Archived 2026-05-31._ + +- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo + Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong` + as the nh3-dev hand-launch key — shared Heimdall agent slot + (`skaldsong:wizard-v2`), different `key_id`s for independent + rotation. Demo Worldtree stays for isolation; personal becomes + the multi-consumer dev iteration instance. + _Archived 2026-05-31._ + +- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green. + Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`). + Wire layer (handshake + binding + dispatch) + data-flow + (per-dispatch JWT claims → `ctx.session_id` populated → real + session-scoped data) + agent-loop (LLM reads + quotes back) all + proven. Resolves the "stalled mid-Worldtree" state from the + 2026-05-17 snapshot. + _Archived 2026-05-31._ + +- `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` + + `repetition_penalty 1.0 → 1.15` on default+fast profiles. + Heretic-abliterated qwen3.6 was locking into degenerate attractors + at the model's thinking-mode floor (0.6). Pattern: + abliterated/uncensored Qwen variants need higher temp + non-trivial + rep-penalty than base, NOT the model-card's documented floors. + _Archived 2026-05-31._ + +- `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as + bind-mount + root-then-drop entrypoint shim (gosu). Operators + get persistent per-instance config without container-rebuild. + Same bind-mount pattern hit twice subsequently in v0.27.0 (selene) + and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so + every new required key surfaces as a crash-loop on existing + deployments. The v0.29.12 canonical example files close this loop. + _Archived 2026-05-31._ + +- `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET + visibility. `available_agents()` helper was over-applied to + `GET /agents/`, masking ALL Tier 3 agents regardless of row + state. Bug only visible as "agent not found" via GET; storage was + fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added + fail-fast hardening for the startup pre-resolve fragility class. + _Archived 2026-05-31._ + +- `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback. + v0.30.6's GET-then-define-or-patch path crash-looped against + pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom → + define 409 conflict). v0.30.7 catches the 409 and falls through + to PATCH (which silently 2xx'd on the pre-fix Worldtree). + _Archived 2026-05-31._ + +## Tried and abandoned (archived) + +- `[2026-04-30]` task-board workflow with + `container: image: debian:bookworm-slim` — fails: + `actions/checkout@v4` needs `node` at runtime, slim image lacks it. + Switched to `node:20-bookworm-slim` (has node + apt) or runner-label + default. (Pattern revisited 2026-05-17 for skaldsong-dev: container + override needs `nodejs` apt-installed unless it IS the default.) + _Archived 2026-05-27._ + +- `[2026-04-30]` Dropping the `container:` directive before runner + re-registration with docker-schema labels — runner silently falls + back to **host mode** (jobs run inside the alpine `act_runner` + container itself, no apt). The `:host` suffix in startup logs + (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the + giveaway. Fix: register with `pfi-fleet:docker://` schema + labels. + _Archived 2026-05-27._ + +- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing + — doesn't take. The `.runner` registration cache pins labels at + first registration; env-var updates are read each start but the + stored token + UUID are tied to the original label set on the gitea + side. Fix: stop runner, delete `.runner`, generate new admin + registration token, redeploy. + _Archived 2026-05-27._ + +- `[2026-04-30]` `git reset --hard origin/` in + `deploy-task-board.yaml` (and the in-repo nevermore playbook before + fix) — invalid syntax: `origin/` prefix only works for branch refs. + SHAs need `git reset --hard ` directly. Resolved with + `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, + then bare `"{{ ref }}^{commit}"` fallback. + _Archived 2026-05-27._ + +- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after + task-board wiring — it was actually only repo-scope on + `vh/task-board`. vor's first CI run failed with empty SSH key + (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user + scope at `gitea.phasefinal.com/user/settings/actions/secrets`. + _Archived 2026-05-27._ + +- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — + leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a + follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` + still leaked the first 4 chars. Lesson: when probing secret-bearing + files, use field-by-field SELECTIVE inspection + (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any + password line, `grep -c` (existence) or `test -n "$(...)"` + (non-empty), never `cat` or value-printing. + _Archived 2026-05-27._ + +- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT + timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero + matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. + **Gitea logs in PDT, Traefik logs in UTC** — same host, different + timezones. Always normalize timezones (UTC) when correlating logs + across services on the same box. Cost: ~30 min in the wrong + direction. + _Archived 2026-05-27._ + +- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / + `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on + `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern + **response writes**, not request body reads. `unexpected EOF` from + Go's HTTP server means the client closed mid-body-upload; not a + knob Gitea exposes server-side. + _Archived 2026-05-27._ + +- `[2026-05-12]` Defaulting asset-engine to Traefik-routed + (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on + first scaffold — user pulled it back to LAN-direct. Internal + tools default LAN-direct; only add Traefik when an external/TLS + surface is actually needed. + _Archived 2026-05-31._ + +- `[2026-05-12]` Routing althing thread replies through `galdrabok` + when the actual dev handle is `forseti` — bus rejected + `to=forseti` initially because thread participants list was + `[galdrabok, infra]`. Solved by starting a new thread with + `forseti` as the direct recipient. Lesson: when the bus + auto-resolves a sender handle that doesn't match the actual dev + role, start a fresh thread rather than fighting the participant list. + _Archived 2026-05-31._ + +- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat + naming) — vLLM-Omni serving Voxtral does NOT translate aliases. + Native presets are `_` shape (`neutral_female`, + `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the + exact wrapper-deployed preset names before setting a catalog + default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9 + Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer + wrapper has NO voice-listing endpoint at all; voices are + filesystem paths under the `kyutai/tts-voices` HF repo). + _Archived 2026-05-31._ + +- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 — + flag was deprecated. Use `--runner pooling`; the model's + `architectures: [LlamaForSequenceClassification]` in config.json + drives the classification head. Surfaced as `vllm: error: + unrecognized arguments: --task classify` in container logs. + _Archived 2026-05-31._ + +- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env + var into a worldtree container — `compose.yaml`'s `&worldtree-env` + anchor must explicitly declare the passthrough or the value silently + doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND + `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is + env empty?" diagnosis each time. Worldtree-side fix in + `vh/worldtree@08f02b2`. + _Archived 2026-05-31._ + +- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli + sandbox without explicit `-e WORLDTREE_IMAGE=` re-pins the + container to `:latest`, even when a newer SHA-tagged image is on + disk. Symptom: container "recreated" but actually reverted to a + stale image. Pass `-e WORLDTREE_IMAGE=...:` to the docker run + invocation. Worldtree-dev's `8ef3801` health-gated `:latest` + advance is the long-term fix. + _Archived 2026-05-31._ + +- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples + — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`; + uncommenting lands the literal `$HOME/...` string. Volva-dev's + `f4dda73` swapped to `/home//...` placeholders. + _Archived 2026-05-31._ + +- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only` + without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/ + althing.db`) and codex's session state (`~/.codex/`) both need to + write. Container started but every poll failed with "db path not + writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing + /home/lkraven/.codex` (preserves the hardening intent, only carves + out the specific dirs). + _Archived 2026-05-31._ + +- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd + "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works + under systemd — `EnvironmentFile=` parser aborts on the first + unparseable line (command substitution), and `VOLVA_ALTHING_CLI` + declared below silently never lands. Symptom: `Environment=` + property empty, daemon error "althing-cli not found at + 'althing-cli'". Fix: replace command-substitution with literal + path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream. + _Archived 2026-05-31._ + +- `[2026-05-19]` Naive `docker rmi worldtree: --force` + for CD SHA cleanup — would untag pinned/personal worldtree images + since all three deployments share corviduo-dev. Use + `docker image prune -a --filter "until=Xh"` instead — respects + in-use semantic (Docker won't remove an image referenced by any + container on the host), so pinned/personal protected automatically. + _Archived 2026-05-31._ + +- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step + failed with 401 unauthorized. ana-docker had no `docker login` + for `gitea.phasefinal.com`. My playbook prereq note ("docker login + has been done at least once") was an unverified assumption. One-time + manual login persists in `~/.docker/config.json`; architectural fix + (workflow-side `ssh ana-docker 'docker login ...'` step using + `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2. + _Archived 2026-05-31._ + +- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300` + as a bare URL — pydantic-settings parses complex env vars via + `json.loads()`; first-boot crashloop with `SettingsError: error + parsing value for field "cors_origins"`. Must be JSON array literal: + `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`. + _Archived 2026-05-31._ + +- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist` + in compose — mismatched Dockerfile reality. The Dockerfile COPYs + SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`). + Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`") + rather than verifying against the actual Dockerfile they shipped. + Lesson: when encoding container-internal paths in compose, verify + against the Dockerfile, not the design-doc. + _Archived 2026-05-31._ + +- `[2026-05-19]` Playbook verify step `docker ps | grep healthy` + racing the container's `start_period` (30s in compose's + healthcheck). Verify ran 0.09s after `compose up -d --force-recreate` + — well before docker's healthcheck could flip the status from + `(health: starting)` to `(healthy)`. False-negative; container was + operationally up (the earlier `/health` poll verify already + confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the + liveness check; docker's `(healthy)` is just a delayed echo. + _Archived 2026-05-31._ + +- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in + compose env block — names skaldsong's app doesn't read. App reads + `SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per + Dockerfile ENV defaults). Wrong names = silently no-op; app fell + back to Dockerfile defaults pointing at `/app/data/...` which the + compose's bind mount did NOT cover (target was `/app/state/...`). + Result: every `--force-recreate` wiped the SQLite DB. Caught by + skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in + `52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app, + not against design-doc shorthand. + _Archived 2026-05-31._ + +- `[2026-05-25]` First selene-block patch put the block under + `saga_allowed_models:` instead of top-level `models:` — used + `text.replace("models:\n", ...)` which substring-matched the + `saga_allowed_models:\n` line first. Caused YAML parse error. Fix: + anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern: + substring replace on YAML top-level keys WILL match suffix-containing + keys. + _Archived 2026-05-31._ diff --git a/persistent-memory.md b/persistent-memory.md index 9b7a9df..069b660 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-05-20_ +_Last updated: 2026-05-31_ ## Repo purpose @@ -71,378 +71,285 @@ Sister repos (separate gitea repos, deployed by playbooks here): without sudo. Documented with security warning in `servers/corviduo-dev/README.md`. docker-group membership is effectively root via bind-mount; treat as a sudo-equivalent grant. + **Foot-gun: when running `docker compose` inside this sandbox, + any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`) + resolves against the sandbox CWD, but Docker daemon interprets the + resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path` + to the docker run invocation for any relative-default config dir.** ## Current state / in-flight -_As of 2026-05-20:_ +_As of 2026-05-31:_ -- **Worldtree CD disk-hygiene PR #184 approved, ready for merge.** - Three commits on `infra/disk-watermark-gate-cd` in vh/Worldtree: - watermark gate (pre-pull recovery), eager post-deploy SHA prune - (post-`:latest`-advance), `set -e` hardening in SSH blocks. Pending - worldtree-dev's merge button. Follow-up extension to - `deploy-personal.yml` + `deploy-pinned.yml` + Beszel disk-usage - metric all queued, not bundled. +- **Dia2 LIVE as two instances on irv-ml1; zonos + csm still down.** + `dia2-2b` (:8200, Dia2-2B) + `dia2-1b` (:8202, Dia2-1B) running + + healthy on GPU 0 from `local/dia:v2`; catalog `status: ready` (both + exercised). Legacy Dia 1.6B retired. **zonos**: REST adapter built + (`stacks/zonos/adapter/`, `local/zonos-api`) but the stack is DOWN — + brought down to free 3090 VRAM for Dia2; catalog entry `status: down`, + adapter port 8203. **csm**: still PARKED (`status: down`, gated — needs + `CSM_HF_TOKEN` before first start; non-reproducible → `experimental` + when up). ComfyUI `:8188` still a catalog-deferred UI **link** (open + hand-off to asset-engine-dev, no form-field API). -- **Skaldsong live on ana-docker:8300 (registry-pull pattern), pointed - at personal Worldtree (:8081).** Three first-deploy footguns - surfaced + canonical-patched: SPA path mismatch (`/app/web/dist` → - `/app/spa`), CORS env shape (pydantic-settings expects JSON array), - verify-step race vs healthcheck `start_period` (grep `^Up` not - `healthy`). nh3-dev hand-launched skaldsong on :8080 also still - running for dev iteration — same user_id=skaldsong, different - key_id, both authorized. +- **Dia2 voices staged — re-stage if host rebuilt.** The 43 curated + voices baked at `/app/voices` aren't reachable from `/tts`'s clone path, + so they're copied into `/worktank/dia/reference_audio` (restic-included). + This is a **one-time manual host step** (`docker exec dia2-2b sh -c 'cp + -n /app/voices/* /app/reference_audio/'`), documented in + `stacks/dia/README.md`, NOT auto-run on recreate. Catalog voice default + is `voice_mode: clone` + `clone_reference_filename: Abigail.wav`. Open + offer (not done): make staging auto-run via container entrypoint. -- **Worldtree personal (:8081) is the iteration-target instance.** - skaldsong + mead-hall + althing-chamber all rotated to personal - via the per-project user keys minted 2026-05-19. Demo (:8080) - stays as the canonical prod surface; pinned (:8082) intentionally - lags. Personal's `BIFROST_CLIENT_ALLOWED_HOSTS` now mirrors demo's: - `10.100.10.50:5173,10.100.10.50:8080,10.250.50.70:8300`. +- **asset-engine catalog fully synced + pushed.** `docs/asset-engine/ + services.yaml` (canonical) and `vh/asset-engine`'s vendored + `data/services.yaml` (re-synced via `sync_catalog.py`, commit + `58dadfe`) both on origin; catalog v1, 18 services; schema regenerated. + asset-engine deploy-on-push triggered. asset-engine-dev notified on + althing thread `01KSZM5MNQYP2DZ25Y2F2DXDBY` of the breaking dia→dia2 + swap (drop dia tile; add dia2-2b/dia2-1b/zonos), the `/tts` field + shapes, and a UI-hardening ask (don't serialize empty optional selects + as the string `"undefined"`; enforce clone_reference_filename + required-when-clone). -- **Disclosed-keys hygiene queue** — these warrant rotation at - convenience (mostly in-bus disclosure during this session): - - `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev — operator - should shred after verify (file mode 600, lkraven-owned) - - mead-hall's prior Worldtree bearer - (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — now superseded - by personal-instance key `a360822d` per the 2026-05-19 rotation - - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from - the 2026-05-12 corviduo-dev outage - - chamber config.yaml's `forseti.api_key` + `agent_runner.api_key` - — superseded by personal-instance key `50d85460` 2026-05-19 - - Gitea runner registration token (`a1135753...`) — rotate via - Gitea admin UI's runner-token reset +- **ttyd fleet driver seat operational on nh3-dev** (browser/iPad/Vision + Pro into the zellij `Claude` session): RO `:7681` / drive `:7682` via + a Caddy front, network-gated (no auth). Lives in `~/.config` on + nh3-dev (out of this repo). Full wiring + gotchas → auto-memory + `reference_ttyd_fleet_seat`. + +- **Worldtree all instances healthy on v0.29.13** (last-known, not + re-checked this session). Demo (:8080), personal (:8081), pinned + (:8082). v0.29.x = saga→echo refactor + selene-judgment validator + + canonical `*.yaml.example` configs. + +- **Skaldsong on v0.32.2** (ana-docker:8300). Kokoro TTS streaming via + SSE working end-to-end; default voice `af_heart`. Last-known. + +- **artemis-31b-v1i live on llama-swap + worldtree personal.** BeaverAI + Gemma 4 31B Q6_K (~28.6GB, 128K ctx), reasoning via + `--reasoning-format deepseek`. + +- **All commits pushed (operator-authorized 2026-05-31).** The prior + pending-push backlog + this session's TTS/catalog work are all on + origin (eshpfi-management through `5c47843`; asset-engine through + `58dadfe`). No local-only backlog. (Standing rule still holds: never + push without an explicit greenlight — this push was granted.) + +- **VibeVoice catalogued for skaldsong #73 contract** (TTS alongside + Kokoro). Endpoint `http://10.100.79.3:8194`, OpenAI-compat. NO + per-word timings (diffusion); read-along is a v2 forced-alignment + upgrade (Parakeet irv-ml1:8765 or Whisper). + +- **Disclosed-keys hygiene queue** — rotate at convenience: + `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev (shred after verify); + mead-hall's prior Worldtree bearer (superseded by `a360822d`); + Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY` (2026-05-12 outage); chamber + `forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea + runner registration token (`a1135753...`). - **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD` - (leaked twice); clean up legacy `news-digest` detritus on ana-docker - (`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`, - image `local/news-digest:v5`); watch nh3-nas `/volume1` (was 65%; - recheck retention or expand before ~80%); the `docker push 60s - client-side ceiling` mystery remains uninstrumented. + (leaked twice); clean up legacy `news-digest` detritus on ana-docker; + watch nh3-nas `/volume1` (was 65%; recheck before ~80%); the + `docker push 60s client-side ceiling` mystery remains uninstrumented. ## Recent decisions -- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate - (env-tunable threshold + window, fail-loud on still-low post-prune) - + eager post-deploy prune (only after `:latest` advance succeeds, - uses `docker image prune -a --filter "until=24h"` which respects - in-use semantic — protects pinned + personal images automatically). - Combined: demo VM holds ~24h of deploy history instead of unbounded - accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` + - `613dac2` + `bd91df5`). +- `[2026-05-31]` Dia2 deployed as **two fixed-model instances** + (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring + legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b** + (breaking for asset-engine). Rationale: the devnen wrapper is + single-model and IGNORES the OpenAI `model` field (verified on its live + OpenAPI), so the only way to offer both Dia2 models as real per-request + asset-engine choices is one fixed endpoint per model. `3139e81` + (deploy), `db15638` (catalog swap). -- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator - options — container + Gitea registry + pull-restart, matching - Worldtree's pattern. Target host ana-docker (NOT nh3-dev where - skaldsong-dev runs for iteration). SHA-pin only for now; health-gated - `:latest` advance is a follow-up once `/health` exercises Worldtree - + Kokoro reachability. +- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer + **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control + surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone); + all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic + blessed values. Voice default is `voice_mode: clone` + + `clone_reference_filename: Abigail.wav` so a stable (non-random-gender) + voice is the out-of-box behavior. `55602b7`, `5c47843`. -- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo - Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong` - as the nh3-dev hand-launch key — shared Heimdall agent slot - (`skaldsong:wizard-v2`), different `key_id`s for independent - rotation. Demo Worldtree stays for isolation; personal becomes - the multi-consumer dev iteration instance. +- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`, + `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front + of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format, + seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off + 8201 — collided with csm). Built; NOT deployed (stack down for VRAM). + Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`. -- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green. - Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`). - Wire layer (handshake + binding + dispatch) + data-flow - (per-dispatch JWT claims → `ctx.session_id` populated → real - session-scoped data) + agent-loop (LLM reads + quotes back) all - proven. Resolves the "stalled mid-Worldtree" state from the - 2026-05-17 snapshot. +- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` + + `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema + hand-off; catalog now validates clean. (asset_engine `catalog.py` + Pydantic already supported both — schema file was just behind.) -- `[2026-05-18]` Volva systemd install complete after three-stage - debug. Final unit at `/etc/systemd/system/volva.service` runs as - `User=lkraven` with `ProtectHome=read-only` + - `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` - carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/ - .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in - env.sh. +- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}` + (`666f7f3` dia+zonos, `a4b8c2a` csm): dia = Nari dialogue TTS via + devnen/Dia-TTS-Server (OpenAI-compat, :8200), zonos = Zyphra Gradio + audition surface (:8199, no API), csm = Sesame conversational + (gated model, :8201). dia + csm catalogued in + `docs/asset-engine/services.yaml` `status: down` + repro audits + (`8eac39b`); all three deployed PARKED to irv-ml1 (no GPU spin). + Zonos + ComfyUI = catalog-deferred (no form-field API) → asset-engine + UI **links**, handed to asset-engine-dev. The bench already had Fish + S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro / + VibeVoice / Qwen3-TTS / Kyutai, so the research-agent's top-2 picks + were already deployed. -- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8 - alpine as a sibling compose service in `stacks/althing-chamber/`, - redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` / - `msg_start` / `msg_complete` event kinds. DB bridge keeps - `msg_curated` + `floor_grant` (structured / canonical). Two-channel - architecture, no overlap. chamber + agent-runner `depends_on: - valkey: service_healthy`. +- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the + agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind + Caddy, network-gated). Brokkr-smithy-dev request; no Tailscale (WG at + firewall). Out of this repo — full architecture + the + HTTP2/OSC52/Safari-auth gotchas in auto-memory + `reference_ttyd_fleet_seat`. -- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops - gets its own permanent admin-tier key (`61419c92`, stored at - `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`). - Future admin ops route through this key, not the bootstrap admin - via docker-as-root. +- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix: + `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1` + drop-in filename-collision fix) adds `_netdev,nofail` to the four + 10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in — + resolves paperless `Exited(255)` on every reboot. traefik also gained + `restart: unless-stopped` (was Docker-default `no` → never + auto-started after a reboot; "homepage broken" was collateral via the + traefik route). Full incident → auto-memory + `incident_esh_docker_nfs_boot_race`. -- `[2026-05-17]` Worldtree env-var addition checklist: anytime - introducing `os.environ.get("FOO")` in worldtree code, update BOTH - `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the - same PR. Same Z_AI_API_KEY-shape footgun bit - `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the - passthrough line in `08f02b2`. +- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor + + config-validator hardening. Net image: + - v0.29.10 — `create_provider` checks model_catalog `family:` + before regex auto-detect (artemis routing fix) + - v0.29.11 — collect-then-raise echo startup validators (UX fix + for cascading one-at-a-time crash cycle that bit demo + personal + during the v0.29.0 → v0.29.9 deploy) + - v0.29.12 — ships `providers.yaml.example` + `defaults.yaml.example` + canonical configs (REQUIRED/OPTIONAL annotated, guaranteed-pass- + validator). Replaces reverse-engineer-from-validator-errors as + the operator's path. + - v0.29.13 — `reasoning_content` extraction in base provider + + catalog `family:` lookup fix (artemis reasoning route through + worldtree GemmaProvider). + Operator-asked, worldtree-dev-shipped, all auto-deployed via CI. -- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner` - as third compose service (worldtree-driver agent dispatcher). - All three althing services use the same image; `command:` selects - entrypoint. Safe to enable preemptively (sleeps when no - driver=worldtree handles declared). +- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree + personal providers.yaml. BeaverAI Gemma 4 31B Q6_K (~28.6GB) + pulled via `pull-hf-repo.yaml`, llama-swap config mirrors + gemma4-31b-dense settings at 128K ctx, `--reasoning-format deepseek` + (gemma format not supported in deployed llama.cpp). Worldtree + catalog entry has `family: gemma` so GemmaProvider routes reasoning + tokens correctly. Pinned locally — push pending operator greenlight. -- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti. - Internal LAN-only at port 7881 (chamber default 7878 collides with - task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the - canonical dev for this stack (galdrabok is on a different project). +- `[2026-05-27]` Skaldsong streaming TTS v0.32.0 → v0.32.1 → v0.32.2. + v0.32.0: chunked-batch via SSE on `/api/stories/{id}/audio?voice=X`, + one Kokoro POST per paragraph. v0.32.1: defensive catch-all in + event_stream after silent JSONDecodeError (made the real + exception surface in an SSE error frame). v0.32.2: NDJSON parsing + — Kokoro `/dev/captioned_speech` returns multi-line JSON for + large inputs (one object per chunk); was using `resp.json()` which + only parsed the first object. Verified end-to-end clean. -- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added - `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier). - Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30 - utilization; 30% headroom). All use `--runner pooling`; classification - drives via model's `architectures: [LlamaForSequenceClassification]` - in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1). - -- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch - playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`. - Replaces ad-hoc `huggingface_hub.snapshot_download` calls. - -- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as - judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI - reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE / - EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`. - -- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/` - removed (retired by vllm). Archived - `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/` - with archival header. Fixed `pfi-pve` VM list to full - `qm list` enumeration. Dropped stale weak-password section from - `pfi-postgres` (rotation done 2026-04-23). - -- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152, - CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF - client hosts: PFI hosts + provides emergency-ops backstop; - Worldtree team owns OS config + deploys + backup decisions. - -- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by - health-gated `:latest` advance in vh/worldtree's deploy workflow - (architect commit `8ef3801`): only tag `:latest` AFTER the new - container's `/health` probe passes. Build-on-host stacks here - don't have this problem because the playbook always builds the - SHA-tagged image from a `git reset --hard ` checkout. - -- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at - `http://10.250.50.70:8200`. Initially included Traefik labels for - public hostname; user pulled them out (internal tool, no public - TLS surface needed). Pattern: internal tools default LAN-direct; - Traefik wiring only when external/TLS required. - -- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack, - vram_gb, gpu_device_id }` per irv-ml1 service for the - orchestrator feature. SSH keypair scaffolded at - `ana-docker:/opt/docker/conf/asset-engine/ssh/` for - asset-engine container → irv-ml1 orchestration via dedicated - ed25519 key. +_22 older entries archived to archival-memory.md._ ## Tried and abandoned -- `[2026-05-19]` Naive `docker rmi worldtree: --force` - for CD SHA cleanup — would untag pinned/personal worldtree images - since all three deployments share corviduo-dev. Use - `docker image prune -a --filter "until=Xh"` instead — respects - in-use semantic (Docker won't remove an image referenced by any - container on the host), so pinned/personal protected automatically. +- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream + packaging quirks: (1) `pip install -e nari-labs/dia2` fails — its + pyproject build backend lacks the PEP 660 `build_editable` hook; (2) + plain `pip install` builds an empty `UNKNOWN-0.0.0` wheel because the + base image's setuptools (59.6) is older than dia2's required `>=70`, so + it ignores the `[project]` metadata; (3) `--no-deps` (which the wrapper's + own auto-installer uses) leaves `transformers`/`sphn`/`whisper-timestamped` + missing. Fix that worked (`local/dia:v2`, `stacks/dia/dia2-image/`): copy + the pure-python `dia2/` package straight into site-packages + install + ONLY those 3 deps; torch 2.12 / numpy 2.2 in the base already satisfy + Dia2, so no risky CUDA/torch churn. -- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step - failed with 401 unauthorized. ana-docker had no `docker login` - for `gitea.phasefinal.com`. My playbook prereq note ("docker login - has been done at least once") was an unverified assumption. One-time - manual login persists in `~/.docker/config.json`; architectural fix - (workflow-side `ssh ana-docker 'docker login ...'` step using - `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2. +- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are + NOT reachable from the `/tts` clone path — it resolves + `clone_reference_filename` against the reference_audio dir ONLY + (`engine.py`). The OpenAI `/v1/audio/speech` `voice` param DOES + auto-resolve them by name (separate code path), which masked the gap in + early testing. Fix: stage `/app/voices/*` into + `/worktank/dia/reference_audio` so `/tts` clone can use them. (So a + "verify on the endpoint the catalog actually targets" lesson — testing + the OpenAI endpoint nearly shipped a catalog that 404s on `/tts`.) -- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300` - as a bare URL — pydantic-settings parses complex env vars via - `json.loads()`; first-boot crashloop with `SettingsError: error - parsing value for field "cors_origins"`. Must be JSON array literal: - `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`. +- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename` + → asset-engine serializes it as the literal string `"undefined"` → + `/tts` 404 `Reference audio file not found: undefined`. Reproduced + directly. Worked around in the catalog (default the field to a real + voice); proper fix is asset-engine-side (don't send `"undefined"` for + empty optionals; enforce conditional-required) — flagged to + asset-engine-dev. -- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist` - in compose — mismatched Dockerfile reality. The Dockerfile COPYs - SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`). - Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`") - rather than verifying against the actual Dockerfile they shipped. - Lesson: when encoding container-internal paths in compose, verify - against the Dockerfile, not the design-doc. +- `[2026-05-31]` asset-engine `docs/asset-engine/services.schema.json` + is a DERIVED artifact (regen from the Pydantic model via + `dump_schema.py`) and had DRIFTED: it rejected the `lifecycle` field + that 12/14 live services already use (13 pre-existing validation + errors against the committed catalog). + **[2026-05-31] RESOLVED:** schema regenerated with `CatalogLifecycle` + + `reproducibility.seed_field` (`b7b2130`); catalog now validates clean. + Standing lesson: the schema is generated from asset_engine's + `catalog.py` — hand-editing `services.yaml` shape without regenerating + re-introduces drift. -- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in - compose env block — names skaldsong's app doesn't read. App reads - `SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per - Dockerfile ENV defaults). Wrong names = silently no-op; app fell - back to Dockerfile defaults pointing at `/app/data/...` which the - compose's bind mount did NOT cover (mount target was - `/app/state/...`). Result: every `--force-recreate` wiped the - SQLite DB along with the ephemeral container layer. Caught by - skaldsong-dev after operator noticed stories vanishing on each - CD push (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in - `52e98fa` — rename env vars, bind targets unchanged. Same lesson - as the `/app/web/dist` footgun: verify env var NAMES against the - Dockerfile/app, not against design-doc shorthand. +- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze + a container worker in UNKILLABLE D-state when the NAS stalled — + `docker restart` fails with "tried to kill container, but did not + receive an exit event"; only a host reboot clears it. Separately, + `fstab defaults` (no `_netdev`) made NFS-bind containers `Exited(255)` + on reboot. Diagnosis + recovery steps → auto-memory + `incident_esh_docker_nfs_boot_race`. -- `[2026-05-19]` Playbook verify step `docker ps | grep healthy` - racing the container's `start_period` (30s in compose's - healthcheck). Verify ran 0.09s after `compose up -d --force-recreate` - — well before docker's healthcheck could flip the status from - `(health: starting)` to `(healthy)`. False-negative; container was - operationally up (the earlier `/health` poll verify already - confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the - liveness check; docker's `(healthy)` is just a delayed echo. +- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (no flag disables it) which + kills ttyd's terminal WebSocket (blank screen); Safari/WebKit never + sends HTTP basic-auth on WS upgrades (→ ttyd "User code denied"). Both + solved for the fleet seat (Caddy forces HTTP/1.1; auth dropped to + network-gating) — detail in auto-memory `reference_ttyd_fleet_seat`. -- `[2026-05-19]` Volva daemon impersonating volva-dev (the human) - for hours of debugging because cwd-based handle resolution. - `session_handles.json` maps `/home/lkraven/development/volva` → - `volva-dev`; the daemon's systemd unit sets - `WorkingDirectory=/home/lkraven/development/volva` and env.sh - didn't pin `ALTHING_HANDLE`, so the daemon polled volva-dev's - inbox and replied AS volva-dev. Catch: cross-reference journal - PIDs against `message_received` events to verify which voice - generated which reply. Fix: pin `ALTHING_HANDLE=volva` in env.sh - (worldtree-dev's `e39be87` made this the default in - env.sh.template). Real follow-on risk: hallucinated structured - feedback from impersonating-daemon can bootstrap real (and - correct) downstream commits — the technical artifact survives - but the conversational attribution rots. +- `[2026-05-27]` `docker compose up -d` from within the `docker:cli` + sandbox: `${VAR:-./config}` defaults resolve `./config` to the + sandbox CWD (e.g. `/wt/config`). Docker daemon interprets the + resulting path against the HOST filesystem, not the sandbox's. + Auto-creates an empty dir on the host → container saw an empty + bind target → entrypoint shim seeded image-baked defaults + (lost all the host-side providers.yaml patches I'd applied earlier + in the day). Root cause for ratatoskr-dev's Tier 3 sessions 500. + Fix: pass `-e WORLDTREE_CONFIG_DIR=/opt/worldtree-personal/config` + to the docker run invocation. Captured into the docker-as-root + pattern note in Tools and conventions. -- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples - — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`; - uncommenting lands the literal `$HOME/...` string. Volva-dev's - `f4dda73` swapped to `/home//...` placeholders. +- `[2026-05-27]` `:latest`-pinned compose + private gitea registry + + sandboxed pull = recreate on ancient cached `:latest`. Local + `:latest` tag never advanced because deploy.yml pulls by SHA + (not by tag); sandbox can't `docker pull` (no creds for private + registry); `compose up -d --pull never` used the 2-week-old + cached `:latest`. Symptom: personal worldtree's matrix container + crash-looped because the ancient image didn't understand the new + config schema. Fix: retag SHA → `:latest` on host first + (`docker tag /: /:latest`), + then recreate with `--pull never`. Better long-term: pin SHA + explicitly in `.env`, advance in CI. -- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only` - without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/ - althing.db`) and codex's session state (`~/.codex/`) both need to - write. Container started but every poll failed with "db path not - writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing - /home/lkraven/.codex` (preserves the hardening intent, only carves - out the specific dirs). +- `[2026-05-27]` Container recreate during in-flight skaldsong gen + kills the runner. Pattern: my llama-swap restart at 23:57Z + a + v0.31.2 `compose up -d` at 02:47Z both took out active dramatist + passes. Confirmed by skaldsong-dev INV-006 ("partial state only + flips on host restart"). With deploys landing every ~10min during + active dev and stories taking >5min, runner-killed-by-recreate is + structural, not incidental. Roadmap (skaldsong-dev): pre-shutdown + signal handler (cleanup row state to failed/aborted), per-scene + resume-from-checkpoint, `/api/admin/quiesce` endpoint. None shipped + yet; operator's call which path. -- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd - "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works - under systemd — `EnvironmentFile=` parser aborts on the first - unparseable line (command substitution), and `VOLVA_ALTHING_CLI` - declared below silently never lands. Symptom: `Environment=` - property empty, daemon error "althing-cli not found at - 'althing-cli'". Fix: replace command-substitution with literal - path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream. +- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — + not supported in the deployed llama.cpp build. Accepts only + `none | deepseek | deepseek-legacy`. `deepseek` populates the + `reasoning_content` SSE delta field which Worldtree GemmaProvider's + fallback path checks. Was load-bearing for getting reasoning + tokens through to ratatoskr's wire. -- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 — - flag was deprecated. Use `--runner pooling`; the model's - `architectures: [LlamaForSequenceClassification]` in config.json - drives the classification head. Surfaced as `vllm: error: - unrecognized arguments: --task classify` in container logs. +- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs + the curl, killing the request early. Bit me twice while probing + SSE-shape samples. Use file-write + separate read. -- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env - var into a worldtree container — `compose.yaml`'s `&worldtree-env` - anchor must explicitly declare the passthrough or the value silently - doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND - `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is - env empty?" diagnosis each time. Worldtree-side fix in - `vh/worldtree@08f02b2`. - -- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli - sandbox without explicit `-e WORLDTREE_IMAGE=` re-pins the - container to `:latest`, even when a newer SHA-tagged image is on - disk. Symptom: container "recreated" but actually reverted to a - stale image. Pass `-e WORLDTREE_IMAGE=...:` to the docker run - invocation. Worldtree-dev's `8ef3801` health-gated `:latest` - advance is the long-term fix. - -- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat - naming) — vLLM-Omni serving Voxtral does NOT translate aliases. - Native presets are `_` shape (`neutral_female`, - `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the - exact wrapper-deployed preset names before setting a catalog - default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9 - Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer - wrapper has NO voice-listing endpoint at all; voices are - filesystem paths under the `kyutai/tts-voices` HF repo). - -- `[2026-05-12]` Defaulting asset-engine to Traefik-routed - (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on - first scaffold — user pulled it back to LAN-direct. Internal - tools default LAN-direct; only add Traefik when an external/TLS - surface is actually needed. - -- `[2026-05-12]` Routing althing thread replies through `galdrabok` - when the actual dev handle is `forseti` — bus rejected - `to=forseti` initially because thread participants list was - `[galdrabok, infra]`. Solved by starting a new thread with - `forseti` as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role, - start a fresh thread rather than fighting the participant list. - -- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT - timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero - matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. - **Gitea logs in PDT, Traefik logs in UTC** — same host, different - timezones. Always normalize timezones (UTC) when correlating logs - across services on the same box. Cost: ~30 min in the wrong - direction. - -- `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump - for "Traefik is dropping connections on big docker pushes" without - first verifying which component was actually in the failure path. - Traefik turned out to be innocent (`499 / 60012ms` = client closed, - Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs - response writes (wrong direction), and the real ceiling was a ~60s - client-side timer no compose change can reach. **Lesson: validate - the diagnostic premise — which component is actually in the failure - path? — before refining the proposed fix.** - -- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / - `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on - `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern - **response writes**, not request body reads. `unexpected EOF` from - Go's HTTP server means the client closed mid-body-upload; not a - knob Gitea exposes server-side. - -- `[2026-04-30]` task-board workflow with - `container: image: debian:bookworm-slim` — fails: - `actions/checkout@v4` needs `node` at runtime, slim image lacks it. - Switched to `node:20-bookworm-slim` (has node + apt) or runner-label - default. (Pattern revisited 2026-05-17 for skaldsong-dev: container - override needs `nodejs` apt-installed unless it IS the default.) - -- `[2026-04-30]` Dropping the `container:` directive before runner - re-registration with docker-schema labels — runner silently falls - back to **host mode** (jobs run inside the alpine `act_runner` - container itself, no apt). The `:host` suffix in startup logs - (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the - giveaway. Fix: register with `pfi-fleet:docker://` schema - labels. - -- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing - — doesn't take. The `.runner` registration cache pins labels at - first registration; env-var updates are read each start but the - stored token + UUID are tied to the original label set on the gitea - side. Fix: stop runner, delete `.runner`, generate new admin - registration token, redeploy. - -- `[2026-04-30]` `git reset --hard origin/` in - `deploy-task-board.yaml` (and the in-repo nevermore playbook before - fix) — invalid syntax: `origin/` prefix only works for branch refs. - SHAs need `git reset --hard ` directly. Resolved with - `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, - then bare `"{{ ref }}^{commit}"` fallback. - -- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after - task-board wiring — it was actually only repo-scope on - `vh/task-board`. vor's first CI run failed with empty SSH key - (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user - scope at `gitea.phasefinal.com/user/settings/actions/secrets`. - -- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — - leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a - follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` - still leaked the first 4 chars. Lesson: when probing secret-bearing - files, use field-by-field SELECTIVE inspection - (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any - password line, `grep -c` (existence) or `test -n "$(...)"` - (non-empty), never `cat` or value-printing. +_24 older entries archived to archival-memory.md._