vh
1dba05d8c8
Captures: phi4-mini FP8 on ana-ml2 vLLM as nevermore summarizer (granite retired, canonical Phi-4 template after apply→revert); infra-ops NOPASSWD-sudo identity across PFI boxes; R15/R16 brokkr-smithy stand-up + ollama 0.30.4 + A6000 UUID-pin; brokkr user + substrate migration + audition service; homepage incident; observability roadmap. Foot-guns: headgun chat-template lesson, UUID GPU-pin, system-unit reaping, pyworld/ gfortran, homepage D-state. Archived 11 settled 2026-06-01 TTS-arc entries (Recent 5 + Tried 6) to archival-memory.md; pruned 145 lines of DONE 2026-06-03 in-flight bulk. File back under the 300-line cap (299).
632 lines
32 KiB
Markdown
632 lines
32 KiB
Markdown
# Archival memory — eshpfi-management
|
|
|
|
_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._
|
|
|
|
## Recent decisions (archived)
|
|
|
|
- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
|
|
CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
|
|
client hosts: PFI hosts + provides emergency-ops backstop;
|
|
Worldtree team owns OS config + deploys + backup decisions.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
|
|
health-gated `:latest` advance in vh/worldtree's deploy workflow
|
|
(architect commit `8ef3801`): only tag `:latest` AFTER the new
|
|
container's `/health` probe passes. Build-on-host stacks here
|
|
don't have this problem because the playbook always builds the
|
|
SHA-tagged image from a `git reset --hard <ref>` checkout.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
|
|
`http://10.250.50.70:8200`. Initially included Traefik labels for
|
|
public hostname; user pulled them out (internal tool, no public
|
|
TLS surface needed). Pattern: internal tools default LAN-direct;
|
|
Traefik wiring only when external/TLS required.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
|
|
vram_gb, gpu_device_id }` per irv-ml1 service for the
|
|
orchestrator feature. SSH keypair scaffolded at
|
|
`ana-docker:/opt/docker/conf/asset-engine/ssh/` for
|
|
asset-engine container → irv-ml1 orchestration via dedicated
|
|
ed25519 key.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
|
|
playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
|
|
Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
|
|
judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
|
|
reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
|
|
EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
|
|
`vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
|
|
Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
|
|
utilization; 30% headroom). All use `--runner pooling`; classification
|
|
drives via model's `architectures: [LlamaForSequenceClassification]`
|
|
in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
|
|
removed (retired by vllm). Archived
|
|
`docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
|
|
with archival header. Fixed `pfi-pve` VM list to full
|
|
`qm list` enumeration. Dropped stale weak-password section from
|
|
`pfi-postgres` (rotation done 2026-04-23).
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
|
|
Internal LAN-only at port 7881 (chamber default 7878 collides with
|
|
task-board). Two-service compose, shared SQLite bind-mount,
|
|
build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
|
|
canonical dev for this stack (galdrabok is on a different project).
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
|
|
as third compose service (worldtree-driver agent dispatcher).
|
|
All three althing services use the same image; `command:` selects
|
|
entrypoint. Safe to enable preemptively (sleeps when no
|
|
driver=worldtree handles declared).
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
|
|
alpine as a sibling compose service in `stacks/althing-chamber/`,
|
|
redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
|
|
`msg_start` / `msg_complete` event kinds. DB bridge keeps
|
|
`msg_curated` + `floor_grant` (structured / canonical). Two-channel
|
|
architecture, no overlap. chamber + agent-runner `depends_on:
|
|
valkey: service_healthy`.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
|
|
gets its own permanent admin-tier key (`61419c92`, stored at
|
|
`ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
|
|
Future admin ops route through this key, not the bootstrap admin
|
|
via docker-as-root.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-17]` Worldtree env-var addition checklist: anytime
|
|
introducing `os.environ.get("FOO")` in worldtree code, update BOTH
|
|
`.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
|
|
same PR. Same Z_AI_API_KEY-shape footgun bit
|
|
`BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
|
|
passthrough line in `08f02b2`.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-18]` Volva systemd install complete after three-stage
|
|
debug. Final unit at `/etc/systemd/system/volva.service` runs as
|
|
`User=lkraven` with `ProtectHome=read-only` +
|
|
`ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
|
|
carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
|
|
.local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
|
|
env.sh.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
|
|
(env-tunable threshold + window, fail-loud on still-low post-prune)
|
|
+ eager post-deploy prune (only after `:latest` advance succeeds,
|
|
uses `docker image prune -a --filter "until=24h"` which respects
|
|
in-use semantic — protects pinned + personal images automatically).
|
|
Combined: demo VM holds ~24h of deploy history instead of unbounded
|
|
accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
|
|
`613dac2` + `bd91df5`).
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
|
|
options — container + Gitea registry + pull-restart, matching
|
|
Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
|
|
skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
|
|
`:latest` advance is a follow-up once `/health` exercises Worldtree
|
|
+ Kokoro reachability.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
|
|
Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
|
|
as the nh3-dev hand-launch key — shared Heimdall agent slot
|
|
(`skaldsong:wizard-v2`), different `key_id`s for independent
|
|
rotation. Demo Worldtree stays for isolation; personal becomes
|
|
the multi-consumer dev iteration instance.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
|
|
Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
|
|
Wire layer (handshake + binding + dispatch) + data-flow
|
|
(per-dispatch JWT claims → `ctx.session_id` populated → real
|
|
session-scoped data) + agent-loop (LLM reads + quotes back) all
|
|
proven. Resolves the "stalled mid-Worldtree" state from the
|
|
2026-05-17 snapshot.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` +
|
|
`repetition_penalty 1.0 → 1.15` on default+fast profiles.
|
|
Heretic-abliterated qwen3.6 was locking into degenerate attractors
|
|
at the model's thinking-mode floor (0.6). Pattern:
|
|
abliterated/uncensored Qwen variants need higher temp + non-trivial
|
|
rep-penalty than base, NOT the model-card's documented floors.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as
|
|
bind-mount + root-then-drop entrypoint shim (gosu). Operators
|
|
get persistent per-instance config without container-rebuild.
|
|
Same bind-mount pattern hit twice subsequently in v0.27.0 (selene)
|
|
and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so
|
|
every new required key surfaces as a crash-loop on existing
|
|
deployments. The v0.29.12 canonical example files close this loop.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET
|
|
visibility. `available_agents()` helper was over-applied to
|
|
`GET /agents/<id>`, masking ALL Tier 3 agents regardless of row
|
|
state. Bug only visible as "agent not found" via GET; storage was
|
|
fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added
|
|
fail-fast hardening for the startup pre-resolve fragility class.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback.
|
|
v0.30.6's GET-then-define-or-patch path crash-looped against
|
|
pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom →
|
|
define 409 conflict). v0.30.7 catches the 409 and falls through
|
|
to PATCH (which silently 2xx'd on the pre-fix Worldtree).
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
|
|
config-validator hardening (v0.29.10 create_provider family-before-regex;
|
|
v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships
|
|
providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13
|
|
reasoning_content extraction + catalog family lookup). Operator-asked,
|
|
worldtree-dev-shipped, CI-deployed.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal.
|
|
BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek`
|
|
(gemma format unsupported in deployed llama.cpp). Worldtree catalog
|
|
`family: gemma` so GemmaProvider routes reasoning tokens.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE
|
|
(one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON
|
|
parsing for Kokoro /dev/captioned_speech multi-line responses.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
|
|
(`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
|
|
legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
|
|
(breaking for asset-engine). Rationale: the devnen wrapper is
|
|
single-model and IGNORES the OpenAI `model` field (verified on its live
|
|
OpenAPI), so the only way to offer both Dia2 models as real per-request
|
|
asset-engine choices is one fixed endpoint per model. `3139e81`
|
|
(deploy), `db15638` (catalog swap).
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
|
|
**`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
|
|
surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
|
|
all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
|
|
blessed values. Voice default is `voice_mode: clone` +
|
|
`clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
|
|
voice is the out-of-box behavior. `55602b7`, `5c47843`.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
|
|
`local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
|
|
of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
|
|
seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
|
|
8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
|
|
Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
|
|
`reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
|
|
hand-off; catalog now validates clean. (asset_engine `catalog.py`
|
|
Pydantic already supported both — schema file was just behind.)
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
|
|
(`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish
|
|
S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
|
|
VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.)
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
|
|
agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
|
|
Caddy, network-gated). Out of this repo — full architecture + the
|
|
HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
|
|
`playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
|
|
drop-in filename-collision fix) adds `_netdev,nofail` to the four
|
|
10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
|
|
resolves paperless `Exited(255)` on reboot. traefik also gained
|
|
`restart: unless-stopped`. Full incident → auto-memory
|
|
`incident_esh_docker_nfs_boot_race`.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting
|
|
the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79
|
|
cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default;
|
|
transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves
|
|
identity (affects pronunciation, not timbre). Root cause of "Imogen sounds
|
|
nothing like British" = the `"undefined"` select bug feeding Fish its default
|
|
voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no
|
|
Fish-side fix needed. (Methodology lessons → Tried and abandoned.)
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from
|
|
catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host
|
|
(`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed
|
|
Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced;
|
|
the open 1B base ships no usable voices, can't generate text, English-only,
|
|
no real-time streaming, absent from leaderboards, dominated by
|
|
Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and
|
|
abandoned). asset-engine re-synced (`306aae9`).
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the
|
|
devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing
|
|
none of the emotion knobs → poor output). Default `exaggeration: 0.5` per
|
|
Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3
|
|
which over-acts — deliberately NOT adopted. 28 predefined voices via
|
|
`/get_predefined_voices` (default Emily.wav). Corrected seedable false→true.
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a
|
|
blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface
|
|
+ a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no
|
|
/voices API). `reference_id`-by-name works ONLY when each voice is staged as
|
|
a `references/<name>/` dir with `<name>.wav` + `<name>.lab` — was silently a
|
|
no-op until that layout fix on 2026-06-01 (see the two-root-causes entry
|
|
above); the inline-path-as-filesystem-path `references` shape 500s (see Tried
|
|
and abandoned). Corrected vram
|
|
4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting
|
|
British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229).
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending
|
|
asset-engine-dev honoring it in the form template. Clone-voice selects
|
|
(chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend
|
|
widget (blending reference clones is nonsensical). I ship the catalog flag +
|
|
schema field once they read it. Tracked: althing msg
|
|
`01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.
|
|
_Archived 2026-06-04._
|
|
|
|
## Tried and abandoned (archived)
|
|
|
|
- `[2026-04-30]` task-board workflow with
|
|
`container: image: debian:bookworm-slim` — fails:
|
|
`actions/checkout@v4` needs `node` at runtime, slim image lacks it.
|
|
Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
|
|
default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
|
|
override needs `nodejs` apt-installed unless it IS the default.)
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-04-30]` Dropping the `container:` directive before runner
|
|
re-registration with docker-schema labels — runner silently falls
|
|
back to **host mode** (jobs run inside the alpine `act_runner`
|
|
container itself, no apt). The `:host` suffix in startup logs
|
|
(`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
|
|
giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
|
|
labels.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
|
|
— doesn't take. The `.runner` registration cache pins labels at
|
|
first registration; env-var updates are read each start but the
|
|
stored token + UUID are tied to the original label set on the gitea
|
|
side. Fix: stop runner, delete `.runner`, generate new admin
|
|
registration token, redeploy.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-04-30]` `git reset --hard origin/<sha>` in
|
|
`deploy-task-board.yaml` (and the in-repo nevermore playbook before
|
|
fix) — invalid syntax: `origin/` prefix only works for branch refs.
|
|
SHAs need `git reset --hard <sha>` directly. Resolved with
|
|
`git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
|
|
then bare `"{{ ref }}^{commit}"` fallback.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
|
|
task-board wiring — it was actually only repo-scope on
|
|
`vh/task-board`. vor's first CI run failed with empty SSH key
|
|
(`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
|
|
scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
|
|
leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
|
|
follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
|
|
still leaked the first 4 chars. Lesson: when probing secret-bearing
|
|
files, use field-by-field SELECTIVE inspection
|
|
(`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
|
|
password line, `grep -c` (existence) or `test -n "$(...)"`
|
|
(non-empty), never `cat` or value-printing.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
|
|
timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
|
|
matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
|
|
**Gitea logs in PDT, Traefik logs in UTC** — same host, different
|
|
timezones. Always normalize timezones (UTC) when correlating logs
|
|
across services on the same box. Cost: ~30 min in the wrong
|
|
direction.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
|
|
`PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
|
|
`/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
|
|
**response writes**, not request body reads. `unexpected EOF` from
|
|
Go's HTTP server means the client closed mid-body-upload; not a
|
|
knob Gitea exposes server-side.
|
|
_Archived 2026-05-27._
|
|
|
|
- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
|
|
(`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
|
|
first scaffold — user pulled it back to LAN-direct. Internal
|
|
tools default LAN-direct; only add Traefik when an external/TLS
|
|
surface is actually needed.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-12]` Routing althing thread replies through `galdrabok`
|
|
when the actual dev handle is `forseti` — bus rejected
|
|
`to=forseti` initially because thread participants list was
|
|
`[galdrabok, infra]`. Solved by starting a new thread with
|
|
`forseti` as the direct recipient. Lesson: when the bus
|
|
auto-resolves a sender handle that doesn't match the actual dev
|
|
role, start a fresh thread rather than fighting the participant list.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
|
|
naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
|
|
Native presets are `<register>_<gender>` shape (`neutral_female`,
|
|
`casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
|
|
exact wrapper-deployed preset names before setting a catalog
|
|
default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
|
|
Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
|
|
wrapper has NO voice-listing endpoint at all; voices are
|
|
filesystem paths under the `kyutai/tts-voices` HF repo).
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
|
|
flag was deprecated. Use `--runner pooling`; the model's
|
|
`architectures: [LlamaForSequenceClassification]` in config.json
|
|
drives the classification head. Surfaced as `vllm: error:
|
|
unrecognized arguments: --task classify` in container logs.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
|
|
var into a worldtree container — `compose.yaml`'s `&worldtree-env`
|
|
anchor must explicitly declare the passthrough or the value silently
|
|
doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
|
|
`BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
|
|
env empty?" diagnosis each time. Worldtree-side fix in
|
|
`vh/worldtree@08f02b2`.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
|
|
sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
|
|
container to `:latest`, even when a newer SHA-tagged image is on
|
|
disk. Symptom: container "recreated" but actually reverted to a
|
|
stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
|
|
invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
|
|
advance is the long-term fix.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
|
|
— systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
|
|
uncommenting lands the literal `$HOME/...` string. Volva-dev's
|
|
`f4dda73` swapped to `/home/<svc-user>/...` placeholders.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
|
|
without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
|
|
althing.db`) and codex's session state (`~/.codex/`) both need to
|
|
write. Container started but every poll failed with "db path not
|
|
writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
|
|
/home/lkraven/.codex` (preserves the hardening intent, only carves
|
|
out the specific dirs).
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
|
|
"$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
|
|
under systemd — `EnvironmentFile=` parser aborts on the first
|
|
unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
|
|
declared below silently never lands. Symptom: `Environment=`
|
|
property empty, daemon error "althing-cli not found at
|
|
'althing-cli'". Fix: replace command-substitution with literal
|
|
path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
|
|
for CD SHA cleanup — would untag pinned/personal worldtree images
|
|
since all three deployments share corviduo-dev. Use
|
|
`docker image prune -a --filter "until=Xh"` instead — respects
|
|
in-use semantic (Docker won't remove an image referenced by any
|
|
container on the host), so pinned/personal protected automatically.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
|
|
failed with 401 unauthorized. ana-docker had no `docker login`
|
|
for `gitea.phasefinal.com`. My playbook prereq note ("docker login
|
|
has been done at least once") was an unverified assumption. One-time
|
|
manual login persists in `~/.docker/config.json`; architectural fix
|
|
(workflow-side `ssh ana-docker 'docker login ...'` step using
|
|
`REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
|
|
as a bare URL — pydantic-settings parses complex env vars via
|
|
`json.loads()`; first-boot crashloop with `SettingsError: error
|
|
parsing value for field "cors_origins"`. Must be JSON array literal:
|
|
`SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
|
|
in compose — mismatched Dockerfile reality. The Dockerfile COPYs
|
|
SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
|
|
Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
|
|
rather than verifying against the actual Dockerfile they shipped.
|
|
Lesson: when encoding container-internal paths in compose, verify
|
|
against the Dockerfile, not the design-doc.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
|
|
racing the container's `start_period` (30s in compose's
|
|
healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
|
|
— well before docker's healthcheck could flip the status from
|
|
`(health: starting)` to `(healthy)`. False-negative; container was
|
|
operationally up (the earlier `/health` poll verify already
|
|
confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
|
|
liveness check; docker's `(healthy)` is just a delayed echo.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in
|
|
compose env block — names skaldsong's app doesn't read. App reads
|
|
`SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per
|
|
Dockerfile ENV defaults). Wrong names = silently no-op; app fell
|
|
back to Dockerfile defaults pointing at `/app/data/...` which the
|
|
compose's bind mount did NOT cover (target was `/app/state/...`).
|
|
Result: every `--force-recreate` wiped the SQLite DB. Caught by
|
|
skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in
|
|
`52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app,
|
|
not against design-doc shorthand.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-25]` First selene-block patch put the block under
|
|
`saga_allowed_models:` instead of top-level `models:` — used
|
|
`text.replace("models:\n", ...)` which substring-matched the
|
|
`saga_allowed_models:\n` line first. Caused YAML parse error. Fix:
|
|
anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern:
|
|
substring replace on YAML top-level keys WILL match suffix-containing
|
|
keys.
|
|
_Archived 2026-05-31._
|
|
|
|
- `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox:
|
|
`${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the
|
|
Docker daemon interprets the path against the HOST filesystem → auto-creates
|
|
an empty dir → entrypoint reseeded image-baked defaults (lost host-side
|
|
providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`.
|
|
Folded into the docker-as-root convention note.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed
|
|
pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag
|
|
never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then
|
|
`--pull never`. Better: pin SHA in `.env`, advance in CI.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the
|
|
runner. With deploys every ~10min and stories >5min, structural not
|
|
incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene
|
|
resume-from-checkpoint, /api/admin/quiesce. None shipped.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in
|
|
the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek`
|
|
populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl,
|
|
killing the request early. Use file-write + separate read.
|
|
_Archived 2026-06-01._
|
|
|
|
- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
|
|
packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660
|
|
`build_editable` hook; (2) plain `pip install` builds an empty
|
|
`UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3)
|
|
`--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix
|
|
(`local/dia:v2`): copy the pure-python `dia2/` package into site-packages +
|
|
install ONLY those 3 deps; base torch/numpy already satisfy Dia2.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT
|
|
reachable from the `/tts` clone path — it resolves `clone_reference_filename`
|
|
against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice`
|
|
param auto-resolves them (separate code path), which masked the gap. Fix:
|
|
stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on
|
|
the endpoint the catalog ACTUALLY targets.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
|
|
→ asset-engine serializes it as the literal string `"undefined"` → `/tts`
|
|
404. First observed on dia2; worked around in the catalog (default the field
|
|
to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend
|
|
widget reading Shoelace `.value` before hydration (see Current state); the
|
|
real fix is asset-engine-side and is escalated.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the
|
|
Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the
|
|
`lifecycle` field 12/14 services use. RESOLVED: regenerated with
|
|
`CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson:
|
|
hand-editing `services.yaml` shape without regenerating re-introduces drift.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket
|
|
→ blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades.
|
|
Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating)
|
|
— detail in auto-memory `reference_ttyd_fleet_seat`.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a
|
|
container worker in UNKILLABLE D-state when the NAS stalled — only a host
|
|
reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind
|
|
containers `Exited(255)` on reboot. → auto-memory
|
|
`incident_esh_docker_nfs_boot_race`.
|
|
_Archived 2026-06-03._
|
|
|
|
- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the
|
|
growing token prefix each chunk + emit the delta, assuming the causal flow
|
|
gives a stable prefix. It does NOT (full-context attention, see above);
|
|
maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at
|
|
the start every call. Don't go this way.
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
|
|
Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
|
|
where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies
|
|
at `huggingface-cli login`/`download`. Passing the HF token as a build-arg
|
|
got the token in (the model downloads at BUILD time, not runtime — our
|
|
compose only wired it as runtime env, the original failure) but then hit the
|
|
rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth
|
|
it for a weak model → CSM removed.
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`)
|
|
→ HTTP 500 on this build, despite the stack README showing that shape. The
|
|
working clone paths are `reference_id` (by name → resolves the DIRECTORY
|
|
`references/<name>/` containing `<name>.wav` + `<name>.lab`; the flat
|
|
`references/<name>.wav` is ignored — see the two-root-causes entry above) or
|
|
inline base64 `references`. The catalog uses `reference_id`.
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY**
|
|
(disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt`
|
|
for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778)
|
|
/ empty (0.738) transcripts all clone Imogen about equally — the transcript
|
|
affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds
|
|
like. The real culprit for "not British" was the `"undefined"` select bug, not
|
|
the transcript. (A correct transcript still marginally helps pronunciation —
|
|
cheap to stage, not load-bearing.)
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen
|
|
to contralto/mezzo) — Fish ignores small reference shifts and overshoots
|
|
large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba"
|
|
(formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad.
|
|
Abandoned at every depth; all variants deleted. Finer independent
|
|
pitch/formant control needs praat (not installed). Future clones = operator's.
|
|
_Archived 2026-06-04._
|
|
|
|
- `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its
|
|
dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen
|
|
clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish
|
|
is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the
|
|
same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to
|
|
default). Use ECAPA, not resemblyzer, for speaker-verification.
|
|
_Archived 2026-06-04._
|