memory: snapshot — Dia2 two-instance deploy, /tts rework, zonos adapter
Capture this session's durable state: Dia2 live as dia2-2b/dia2-1b (legacy dia retired), dia2 catalog on /tts with full controls + stable-voice default, zonos REST adapter (built, stack down), schema regen, all pushed. New foot-guns: dia2 image-build quirks, predefined-voices-not-in-/tts, voice_mode clone 'undefined' 404. Archived 6 older entries (4 decisions, 2 foot-guns).
This commit is contained in:
@@ -0,0 +1,385 @@
|
|||||||
|
# Archival memory — eshpfi-management
|
||||||
|
|
||||||
|
_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._
|
||||||
|
|
||||||
|
## Recent decisions (archived)
|
||||||
|
|
||||||
|
- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
|
||||||
|
CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
|
||||||
|
client hosts: PFI hosts + provides emergency-ops backstop;
|
||||||
|
Worldtree team owns OS config + deploys + backup decisions.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
|
||||||
|
health-gated `:latest` advance in vh/worldtree's deploy workflow
|
||||||
|
(architect commit `8ef3801`): only tag `:latest` AFTER the new
|
||||||
|
container's `/health` probe passes. Build-on-host stacks here
|
||||||
|
don't have this problem because the playbook always builds the
|
||||||
|
SHA-tagged image from a `git reset --hard <ref>` checkout.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
|
||||||
|
`http://10.250.50.70:8200`. Initially included Traefik labels for
|
||||||
|
public hostname; user pulled them out (internal tool, no public
|
||||||
|
TLS surface needed). Pattern: internal tools default LAN-direct;
|
||||||
|
Traefik wiring only when external/TLS required.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
|
||||||
|
vram_gb, gpu_device_id }` per irv-ml1 service for the
|
||||||
|
orchestrator feature. SSH keypair scaffolded at
|
||||||
|
`ana-docker:/opt/docker/conf/asset-engine/ssh/` for
|
||||||
|
asset-engine container → irv-ml1 orchestration via dedicated
|
||||||
|
ed25519 key.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
|
||||||
|
playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
|
||||||
|
Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
|
||||||
|
judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
|
||||||
|
reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
|
||||||
|
EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
|
||||||
|
`vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
|
||||||
|
Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
|
||||||
|
utilization; 30% headroom). All use `--runner pooling`; classification
|
||||||
|
drives via model's `architectures: [LlamaForSequenceClassification]`
|
||||||
|
in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
|
||||||
|
removed (retired by vllm). Archived
|
||||||
|
`docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
|
||||||
|
with archival header. Fixed `pfi-pve` VM list to full
|
||||||
|
`qm list` enumeration. Dropped stale weak-password section from
|
||||||
|
`pfi-postgres` (rotation done 2026-04-23).
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
|
||||||
|
Internal LAN-only at port 7881 (chamber default 7878 collides with
|
||||||
|
task-board). Two-service compose, shared SQLite bind-mount,
|
||||||
|
build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
|
||||||
|
canonical dev for this stack (galdrabok is on a different project).
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
|
||||||
|
as third compose service (worldtree-driver agent dispatcher).
|
||||||
|
All three althing services use the same image; `command:` selects
|
||||||
|
entrypoint. Safe to enable preemptively (sleeps when no
|
||||||
|
driver=worldtree handles declared).
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
|
||||||
|
alpine as a sibling compose service in `stacks/althing-chamber/`,
|
||||||
|
redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
|
||||||
|
`msg_start` / `msg_complete` event kinds. DB bridge keeps
|
||||||
|
`msg_curated` + `floor_grant` (structured / canonical). Two-channel
|
||||||
|
architecture, no overlap. chamber + agent-runner `depends_on:
|
||||||
|
valkey: service_healthy`.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
|
||||||
|
gets its own permanent admin-tier key (`61419c92`, stored at
|
||||||
|
`ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
|
||||||
|
Future admin ops route through this key, not the bootstrap admin
|
||||||
|
via docker-as-root.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-17]` Worldtree env-var addition checklist: anytime
|
||||||
|
introducing `os.environ.get("FOO")` in worldtree code, update BOTH
|
||||||
|
`.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
|
||||||
|
same PR. Same Z_AI_API_KEY-shape footgun bit
|
||||||
|
`BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
|
||||||
|
passthrough line in `08f02b2`.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-18]` Volva systemd install complete after three-stage
|
||||||
|
debug. Final unit at `/etc/systemd/system/volva.service` runs as
|
||||||
|
`User=lkraven` with `ProtectHome=read-only` +
|
||||||
|
`ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
|
||||||
|
carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
|
||||||
|
.local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
|
||||||
|
env.sh.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
|
||||||
|
(env-tunable threshold + window, fail-loud on still-low post-prune)
|
||||||
|
+ eager post-deploy prune (only after `:latest` advance succeeds,
|
||||||
|
uses `docker image prune -a --filter "until=24h"` which respects
|
||||||
|
in-use semantic — protects pinned + personal images automatically).
|
||||||
|
Combined: demo VM holds ~24h of deploy history instead of unbounded
|
||||||
|
accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
|
||||||
|
`613dac2` + `bd91df5`).
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
|
||||||
|
options — container + Gitea registry + pull-restart, matching
|
||||||
|
Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
|
||||||
|
skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
|
||||||
|
`:latest` advance is a follow-up once `/health` exercises Worldtree
|
||||||
|
+ Kokoro reachability.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
|
||||||
|
Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
|
||||||
|
as the nh3-dev hand-launch key — shared Heimdall agent slot
|
||||||
|
(`skaldsong:wizard-v2`), different `key_id`s for independent
|
||||||
|
rotation. Demo Worldtree stays for isolation; personal becomes
|
||||||
|
the multi-consumer dev iteration instance.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
|
||||||
|
Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
|
||||||
|
Wire layer (handshake + binding + dispatch) + data-flow
|
||||||
|
(per-dispatch JWT claims → `ctx.session_id` populated → real
|
||||||
|
session-scoped data) + agent-loop (LLM reads + quotes back) all
|
||||||
|
proven. Resolves the "stalled mid-Worldtree" state from the
|
||||||
|
2026-05-17 snapshot.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` +
|
||||||
|
`repetition_penalty 1.0 → 1.15` on default+fast profiles.
|
||||||
|
Heretic-abliterated qwen3.6 was locking into degenerate attractors
|
||||||
|
at the model's thinking-mode floor (0.6). Pattern:
|
||||||
|
abliterated/uncensored Qwen variants need higher temp + non-trivial
|
||||||
|
rep-penalty than base, NOT the model-card's documented floors.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as
|
||||||
|
bind-mount + root-then-drop entrypoint shim (gosu). Operators
|
||||||
|
get persistent per-instance config without container-rebuild.
|
||||||
|
Same bind-mount pattern hit twice subsequently in v0.27.0 (selene)
|
||||||
|
and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so
|
||||||
|
every new required key surfaces as a crash-loop on existing
|
||||||
|
deployments. The v0.29.12 canonical example files close this loop.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET
|
||||||
|
visibility. `available_agents()` helper was over-applied to
|
||||||
|
`GET /agents/<id>`, masking ALL Tier 3 agents regardless of row
|
||||||
|
state. Bug only visible as "agent not found" via GET; storage was
|
||||||
|
fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added
|
||||||
|
fail-fast hardening for the startup pre-resolve fragility class.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback.
|
||||||
|
v0.30.6's GET-then-define-or-patch path crash-looped against
|
||||||
|
pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom →
|
||||||
|
define 409 conflict). v0.30.7 catches the 409 and falls through
|
||||||
|
to PATCH (which silently 2xx'd on the pre-fix Worldtree).
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
## Tried and abandoned (archived)
|
||||||
|
|
||||||
|
- `[2026-04-30]` task-board workflow with
|
||||||
|
`container: image: debian:bookworm-slim` — fails:
|
||||||
|
`actions/checkout@v4` needs `node` at runtime, slim image lacks it.
|
||||||
|
Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
|
||||||
|
default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
|
||||||
|
override needs `nodejs` apt-installed unless it IS the default.)
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-04-30]` Dropping the `container:` directive before runner
|
||||||
|
re-registration with docker-schema labels — runner silently falls
|
||||||
|
back to **host mode** (jobs run inside the alpine `act_runner`
|
||||||
|
container itself, no apt). The `:host` suffix in startup logs
|
||||||
|
(`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
|
||||||
|
giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
|
||||||
|
labels.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
|
||||||
|
— doesn't take. The `.runner` registration cache pins labels at
|
||||||
|
first registration; env-var updates are read each start but the
|
||||||
|
stored token + UUID are tied to the original label set on the gitea
|
||||||
|
side. Fix: stop runner, delete `.runner`, generate new admin
|
||||||
|
registration token, redeploy.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-04-30]` `git reset --hard origin/<sha>` in
|
||||||
|
`deploy-task-board.yaml` (and the in-repo nevermore playbook before
|
||||||
|
fix) — invalid syntax: `origin/` prefix only works for branch refs.
|
||||||
|
SHAs need `git reset --hard <sha>` directly. Resolved with
|
||||||
|
`git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
|
||||||
|
then bare `"{{ ref }}^{commit}"` fallback.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
|
||||||
|
task-board wiring — it was actually only repo-scope on
|
||||||
|
`vh/task-board`. vor's first CI run failed with empty SSH key
|
||||||
|
(`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
|
||||||
|
scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
|
||||||
|
leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
|
||||||
|
follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
|
||||||
|
still leaked the first 4 chars. Lesson: when probing secret-bearing
|
||||||
|
files, use field-by-field SELECTIVE inspection
|
||||||
|
(`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
|
||||||
|
password line, `grep -c` (existence) or `test -n "$(...)"`
|
||||||
|
(non-empty), never `cat` or value-printing.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
|
||||||
|
timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
|
||||||
|
matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
|
||||||
|
**Gitea logs in PDT, Traefik logs in UTC** — same host, different
|
||||||
|
timezones. Always normalize timezones (UTC) when correlating logs
|
||||||
|
across services on the same box. Cost: ~30 min in the wrong
|
||||||
|
direction.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
|
||||||
|
`PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
|
||||||
|
`/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
|
||||||
|
**response writes**, not request body reads. `unexpected EOF` from
|
||||||
|
Go's HTTP server means the client closed mid-body-upload; not a
|
||||||
|
knob Gitea exposes server-side.
|
||||||
|
_Archived 2026-05-27._
|
||||||
|
|
||||||
|
- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
|
||||||
|
(`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
|
||||||
|
first scaffold — user pulled it back to LAN-direct. Internal
|
||||||
|
tools default LAN-direct; only add Traefik when an external/TLS
|
||||||
|
surface is actually needed.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-12]` Routing althing thread replies through `galdrabok`
|
||||||
|
when the actual dev handle is `forseti` — bus rejected
|
||||||
|
`to=forseti` initially because thread participants list was
|
||||||
|
`[galdrabok, infra]`. Solved by starting a new thread with
|
||||||
|
`forseti` as the direct recipient. Lesson: when the bus
|
||||||
|
auto-resolves a sender handle that doesn't match the actual dev
|
||||||
|
role, start a fresh thread rather than fighting the participant list.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
|
||||||
|
naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
|
||||||
|
Native presets are `<register>_<gender>` shape (`neutral_female`,
|
||||||
|
`casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
|
||||||
|
exact wrapper-deployed preset names before setting a catalog
|
||||||
|
default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
|
||||||
|
Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
|
||||||
|
wrapper has NO voice-listing endpoint at all; voices are
|
||||||
|
filesystem paths under the `kyutai/tts-voices` HF repo).
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
|
||||||
|
flag was deprecated. Use `--runner pooling`; the model's
|
||||||
|
`architectures: [LlamaForSequenceClassification]` in config.json
|
||||||
|
drives the classification head. Surfaced as `vllm: error:
|
||||||
|
unrecognized arguments: --task classify` in container logs.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
|
||||||
|
var into a worldtree container — `compose.yaml`'s `&worldtree-env`
|
||||||
|
anchor must explicitly declare the passthrough or the value silently
|
||||||
|
doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
|
||||||
|
`BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
|
||||||
|
env empty?" diagnosis each time. Worldtree-side fix in
|
||||||
|
`vh/worldtree@08f02b2`.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
|
||||||
|
sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
|
||||||
|
container to `:latest`, even when a newer SHA-tagged image is on
|
||||||
|
disk. Symptom: container "recreated" but actually reverted to a
|
||||||
|
stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
|
||||||
|
invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
|
||||||
|
advance is the long-term fix.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
|
||||||
|
— systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
|
||||||
|
uncommenting lands the literal `$HOME/...` string. Volva-dev's
|
||||||
|
`f4dda73` swapped to `/home/<svc-user>/...` placeholders.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
|
||||||
|
without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
|
||||||
|
althing.db`) and codex's session state (`~/.codex/`) both need to
|
||||||
|
write. Container started but every poll failed with "db path not
|
||||||
|
writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
|
||||||
|
/home/lkraven/.codex` (preserves the hardening intent, only carves
|
||||||
|
out the specific dirs).
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
|
||||||
|
"$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
|
||||||
|
under systemd — `EnvironmentFile=` parser aborts on the first
|
||||||
|
unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
|
||||||
|
declared below silently never lands. Symptom: `Environment=`
|
||||||
|
property empty, daemon error "althing-cli not found at
|
||||||
|
'althing-cli'". Fix: replace command-substitution with literal
|
||||||
|
path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
|
||||||
|
for CD SHA cleanup — would untag pinned/personal worldtree images
|
||||||
|
since all three deployments share corviduo-dev. Use
|
||||||
|
`docker image prune -a --filter "until=Xh"` instead — respects
|
||||||
|
in-use semantic (Docker won't remove an image referenced by any
|
||||||
|
container on the host), so pinned/personal protected automatically.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
|
||||||
|
failed with 401 unauthorized. ana-docker had no `docker login`
|
||||||
|
for `gitea.phasefinal.com`. My playbook prereq note ("docker login
|
||||||
|
has been done at least once") was an unverified assumption. One-time
|
||||||
|
manual login persists in `~/.docker/config.json`; architectural fix
|
||||||
|
(workflow-side `ssh ana-docker 'docker login ...'` step using
|
||||||
|
`REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
|
||||||
|
as a bare URL — pydantic-settings parses complex env vars via
|
||||||
|
`json.loads()`; first-boot crashloop with `SettingsError: error
|
||||||
|
parsing value for field "cors_origins"`. Must be JSON array literal:
|
||||||
|
`SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
|
||||||
|
in compose — mismatched Dockerfile reality. The Dockerfile COPYs
|
||||||
|
SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
|
||||||
|
Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
|
||||||
|
rather than verifying against the actual Dockerfile they shipped.
|
||||||
|
Lesson: when encoding container-internal paths in compose, verify
|
||||||
|
against the Dockerfile, not the design-doc.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
|
||||||
|
racing the container's `start_period` (30s in compose's
|
||||||
|
healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
|
||||||
|
— well before docker's healthcheck could flip the status from
|
||||||
|
`(health: starting)` to `(healthy)`. False-negative; container was
|
||||||
|
operationally up (the earlier `/health` poll verify already
|
||||||
|
confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
|
||||||
|
liveness check; docker's `(healthy)` is just a delayed echo.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in
|
||||||
|
compose env block — names skaldsong's app doesn't read. App reads
|
||||||
|
`SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per
|
||||||
|
Dockerfile ENV defaults). Wrong names = silently no-op; app fell
|
||||||
|
back to Dockerfile defaults pointing at `/app/data/...` which the
|
||||||
|
compose's bind mount did NOT cover (target was `/app/state/...`).
|
||||||
|
Result: every `--force-recreate` wiped the SQLite DB. Caught by
|
||||||
|
skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in
|
||||||
|
`52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app,
|
||||||
|
not against design-doc shorthand.
|
||||||
|
_Archived 2026-05-31._
|
||||||
|
|
||||||
|
- `[2026-05-25]` First selene-block patch put the block under
|
||||||
|
`saga_allowed_models:` instead of top-level `models:` — used
|
||||||
|
`text.replace("models:\n", ...)` which substring-matched the
|
||||||
|
`saga_allowed_models:\n` line first. Caused YAML parse error. Fix:
|
||||||
|
anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern:
|
||||||
|
substring replace on YAML top-level keys WILL match suffix-containing
|
||||||
|
keys.
|
||||||
|
_Archived 2026-05-31._
|
||||||
+247
-340
@@ -1,6 +1,6 @@
|
|||||||
# Persistent memory — eshpfi-management
|
# Persistent memory — eshpfi-management
|
||||||
|
|
||||||
_Last updated: 2026-05-20_
|
_Last updated: 2026-05-31_
|
||||||
|
|
||||||
## Repo purpose
|
## Repo purpose
|
||||||
|
|
||||||
@@ -71,378 +71,285 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
|||||||
without sudo. Documented with security warning in
|
without sudo. Documented with security warning in
|
||||||
`servers/corviduo-dev/README.md`. docker-group membership is
|
`servers/corviduo-dev/README.md`. docker-group membership is
|
||||||
effectively root via bind-mount; treat as a sudo-equivalent grant.
|
effectively root via bind-mount; treat as a sudo-equivalent grant.
|
||||||
|
**Foot-gun: when running `docker compose` inside this sandbox,
|
||||||
|
any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
|
||||||
|
resolves against the sandbox CWD, but Docker daemon interprets the
|
||||||
|
resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
|
||||||
|
to the docker run invocation for any relative-default config dir.**
|
||||||
|
|
||||||
## Current state / in-flight
|
## Current state / in-flight
|
||||||
|
|
||||||
_As of 2026-05-20:_
|
_As of 2026-05-31:_
|
||||||
|
|
||||||
- **Worldtree CD disk-hygiene PR #184 approved, ready for merge.**
|
- **Dia2 LIVE as two instances on irv-ml1; zonos + csm still down.**
|
||||||
Three commits on `infra/disk-watermark-gate-cd` in vh/Worldtree:
|
`dia2-2b` (:8200, Dia2-2B) + `dia2-1b` (:8202, Dia2-1B) running +
|
||||||
watermark gate (pre-pull recovery), eager post-deploy SHA prune
|
healthy on GPU 0 from `local/dia:v2`; catalog `status: ready` (both
|
||||||
(post-`:latest`-advance), `set -e` hardening in SSH blocks. Pending
|
exercised). Legacy Dia 1.6B retired. **zonos**: REST adapter built
|
||||||
worldtree-dev's merge button. Follow-up extension to
|
(`stacks/zonos/adapter/`, `local/zonos-api`) but the stack is DOWN —
|
||||||
`deploy-personal.yml` + `deploy-pinned.yml` + Beszel disk-usage
|
brought down to free 3090 VRAM for Dia2; catalog entry `status: down`,
|
||||||
metric all queued, not bundled.
|
adapter port 8203. **csm**: still PARKED (`status: down`, gated — needs
|
||||||
|
`CSM_HF_TOKEN` before first start; non-reproducible → `experimental`
|
||||||
|
when up). ComfyUI `:8188` still a catalog-deferred UI **link** (open
|
||||||
|
hand-off to asset-engine-dev, no form-field API).
|
||||||
|
|
||||||
- **Skaldsong live on ana-docker:8300 (registry-pull pattern), pointed
|
- **Dia2 voices staged — re-stage if host rebuilt.** The 43 curated
|
||||||
at personal Worldtree (:8081).** Three first-deploy footguns
|
voices baked at `/app/voices` aren't reachable from `/tts`'s clone path,
|
||||||
surfaced + canonical-patched: SPA path mismatch (`/app/web/dist` →
|
so they're copied into `/worktank/dia/reference_audio` (restic-included).
|
||||||
`/app/spa`), CORS env shape (pydantic-settings expects JSON array),
|
This is a **one-time manual host step** (`docker exec dia2-2b sh -c 'cp
|
||||||
verify-step race vs healthcheck `start_period` (grep `^Up` not
|
-n /app/voices/* /app/reference_audio/'`), documented in
|
||||||
`healthy`). nh3-dev hand-launched skaldsong on :8080 also still
|
`stacks/dia/README.md`, NOT auto-run on recreate. Catalog voice default
|
||||||
running for dev iteration — same user_id=skaldsong, different
|
is `voice_mode: clone` + `clone_reference_filename: Abigail.wav`. Open
|
||||||
key_id, both authorized.
|
offer (not done): make staging auto-run via container entrypoint.
|
||||||
|
|
||||||
- **Worldtree personal (:8081) is the iteration-target instance.**
|
- **asset-engine catalog fully synced + pushed.** `docs/asset-engine/
|
||||||
skaldsong + mead-hall + althing-chamber all rotated to personal
|
services.yaml` (canonical) and `vh/asset-engine`'s vendored
|
||||||
via the per-project user keys minted 2026-05-19. Demo (:8080)
|
`data/services.yaml` (re-synced via `sync_catalog.py`, commit
|
||||||
stays as the canonical prod surface; pinned (:8082) intentionally
|
`58dadfe`) both on origin; catalog v1, 18 services; schema regenerated.
|
||||||
lags. Personal's `BIFROST_CLIENT_ALLOWED_HOSTS` now mirrors demo's:
|
asset-engine deploy-on-push triggered. asset-engine-dev notified on
|
||||||
`10.100.10.50:5173,10.100.10.50:8080,10.250.50.70:8300`.
|
althing thread `01KSZM5MNQYP2DZ25Y2F2DXDBY` of the breaking dia→dia2
|
||||||
|
swap (drop dia tile; add dia2-2b/dia2-1b/zonos), the `/tts` field
|
||||||
|
shapes, and a UI-hardening ask (don't serialize empty optional selects
|
||||||
|
as the string `"undefined"`; enforce clone_reference_filename
|
||||||
|
required-when-clone).
|
||||||
|
|
||||||
- **Disclosed-keys hygiene queue** — these warrant rotation at
|
- **ttyd fleet driver seat operational on nh3-dev** (browser/iPad/Vision
|
||||||
convenience (mostly in-bus disclosure during this session):
|
Pro into the zellij `Claude` session): RO `:7681` / drive `:7682` via
|
||||||
- `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev — operator
|
a Caddy front, network-gated (no auth). Lives in `~/.config` on
|
||||||
should shred after verify (file mode 600, lkraven-owned)
|
nh3-dev (out of this repo). Full wiring + gotchas → auto-memory
|
||||||
- mead-hall's prior Worldtree bearer
|
`reference_ttyd_fleet_seat`.
|
||||||
(`wt_live_80e1570620ef2aba998dc63954cce3a6`) — now superseded
|
|
||||||
by personal-instance key `a360822d` per the 2026-05-19 rotation
|
- **Worldtree all instances healthy on v0.29.13** (last-known, not
|
||||||
- Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from
|
re-checked this session). Demo (:8080), personal (:8081), pinned
|
||||||
the 2026-05-12 corviduo-dev outage
|
(:8082). v0.29.x = saga→echo refactor + selene-judgment validator +
|
||||||
- chamber config.yaml's `forseti.api_key` + `agent_runner.api_key`
|
canonical `*.yaml.example` configs.
|
||||||
— superseded by personal-instance key `50d85460` 2026-05-19
|
|
||||||
- Gitea runner registration token (`a1135753...`) — rotate via
|
- **Skaldsong on v0.32.2** (ana-docker:8300). Kokoro TTS streaming via
|
||||||
Gitea admin UI's runner-token reset
|
SSE working end-to-end; default voice `af_heart`. Last-known.
|
||||||
|
|
||||||
|
- **artemis-31b-v1i live on llama-swap + worldtree personal.** BeaverAI
|
||||||
|
Gemma 4 31B Q6_K (~28.6GB, 128K ctx), reasoning via
|
||||||
|
`--reasoning-format deepseek`.
|
||||||
|
|
||||||
|
- **All commits pushed (operator-authorized 2026-05-31).** The prior
|
||||||
|
pending-push backlog + this session's TTS/catalog work are all on
|
||||||
|
origin (eshpfi-management through `5c47843`; asset-engine through
|
||||||
|
`58dadfe`). No local-only backlog. (Standing rule still holds: never
|
||||||
|
push without an explicit greenlight — this push was granted.)
|
||||||
|
|
||||||
|
- **VibeVoice catalogued for skaldsong #73 contract** (TTS alongside
|
||||||
|
Kokoro). Endpoint `http://10.100.79.3:8194`, OpenAI-compat. NO
|
||||||
|
per-word timings (diffusion); read-along is a v2 forced-alignment
|
||||||
|
upgrade (Parakeet irv-ml1:8765 or Whisper).
|
||||||
|
|
||||||
|
- **Disclosed-keys hygiene queue** — rotate at convenience:
|
||||||
|
`/tmp/wt-personal-skaldsong-prod.key` on nh3-dev (shred after verify);
|
||||||
|
mead-hall's prior Worldtree bearer (superseded by `a360822d`);
|
||||||
|
Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY` (2026-05-12 outage); chamber
|
||||||
|
`forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea
|
||||||
|
runner registration token (`a1135753...`).
|
||||||
|
|
||||||
- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD`
|
- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD`
|
||||||
(leaked twice); clean up legacy `news-digest` detritus on ana-docker
|
(leaked twice); clean up legacy `news-digest` detritus on ana-docker;
|
||||||
(`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`,
|
watch nh3-nas `/volume1` (was 65%; recheck before ~80%); the
|
||||||
image `local/news-digest:v5`); watch nh3-nas `/volume1` (was 65%;
|
`docker push 60s client-side ceiling` mystery remains uninstrumented.
|
||||||
recheck retention or expand before ~80%); the `docker push 60s
|
|
||||||
client-side ceiling` mystery remains uninstrumented.
|
|
||||||
|
|
||||||
## Recent decisions
|
## Recent decisions
|
||||||
|
|
||||||
- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
|
- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
|
||||||
(env-tunable threshold + window, fail-loud on still-low post-prune)
|
(`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
|
||||||
+ eager post-deploy prune (only after `:latest` advance succeeds,
|
legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
|
||||||
uses `docker image prune -a --filter "until=24h"` which respects
|
(breaking for asset-engine). Rationale: the devnen wrapper is
|
||||||
in-use semantic — protects pinned + personal images automatically).
|
single-model and IGNORES the OpenAI `model` field (verified on its live
|
||||||
Combined: demo VM holds ~24h of deploy history instead of unbounded
|
OpenAPI), so the only way to offer both Dia2 models as real per-request
|
||||||
accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
|
asset-engine choices is one fixed endpoint per model. `3139e81`
|
||||||
`613dac2` + `bd91df5`).
|
(deploy), `db15638` (catalog swap).
|
||||||
|
|
||||||
- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
|
- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
|
||||||
options — container + Gitea registry + pull-restart, matching
|
**`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
|
||||||
Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
|
surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
|
||||||
skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
|
all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
|
||||||
`:latest` advance is a follow-up once `/health` exercises Worldtree
|
blessed values. Voice default is `voice_mode: clone` +
|
||||||
+ Kokoro reachability.
|
`clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
|
||||||
|
voice is the out-of-box behavior. `55602b7`, `5c47843`.
|
||||||
|
|
||||||
- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
|
- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
|
||||||
Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
|
`local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
|
||||||
as the nh3-dev hand-launch key — shared Heimdall agent slot
|
of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
|
||||||
(`skaldsong:wizard-v2`), different `key_id`s for independent
|
seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
|
||||||
rotation. Demo Worldtree stays for isolation; personal becomes
|
8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
|
||||||
the multi-consumer dev iteration instance.
|
Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.
|
||||||
|
|
||||||
- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
|
- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
|
||||||
Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
|
`reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
|
||||||
Wire layer (handshake + binding + dispatch) + data-flow
|
hand-off; catalog now validates clean. (asset_engine `catalog.py`
|
||||||
(per-dispatch JWT claims → `ctx.session_id` populated → real
|
Pydantic already supported both — schema file was just behind.)
|
||||||
session-scoped data) + agent-loop (LLM reads + quotes back) all
|
|
||||||
proven. Resolves the "stalled mid-Worldtree" state from the
|
|
||||||
2026-05-17 snapshot.
|
|
||||||
|
|
||||||
- `[2026-05-18]` Volva systemd install complete after three-stage
|
- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
|
||||||
debug. Final unit at `/etc/systemd/system/volva.service` runs as
|
(`666f7f3` dia+zonos, `a4b8c2a` csm): dia = Nari dialogue TTS via
|
||||||
`User=lkraven` with `ProtectHome=read-only` +
|
devnen/Dia-TTS-Server (OpenAI-compat, :8200), zonos = Zyphra Gradio
|
||||||
`ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
|
audition surface (:8199, no API), csm = Sesame conversational
|
||||||
carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
|
(gated model, :8201). dia + csm catalogued in
|
||||||
.local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
|
`docs/asset-engine/services.yaml` `status: down` + repro audits
|
||||||
env.sh.
|
(`8eac39b`); all three deployed PARKED to irv-ml1 (no GPU spin).
|
||||||
|
Zonos + ComfyUI = catalog-deferred (no form-field API) → asset-engine
|
||||||
|
UI **links**, handed to asset-engine-dev. The bench already had Fish
|
||||||
|
S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
|
||||||
|
VibeVoice / Qwen3-TTS / Kyutai, so the research-agent's top-2 picks
|
||||||
|
were already deployed.
|
||||||
|
|
||||||
- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
|
- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
|
||||||
alpine as a sibling compose service in `stacks/althing-chamber/`,
|
agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
|
||||||
redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
|
Caddy, network-gated). Brokkr-smithy-dev request; no Tailscale (WG at
|
||||||
`msg_start` / `msg_complete` event kinds. DB bridge keeps
|
firewall). Out of this repo — full architecture + the
|
||||||
`msg_curated` + `floor_grant` (structured / canonical). Two-channel
|
HTTP2/OSC52/Safari-auth gotchas in auto-memory
|
||||||
architecture, no overlap. chamber + agent-runner `depends_on:
|
`reference_ttyd_fleet_seat`.
|
||||||
valkey: service_healthy`.
|
|
||||||
|
|
||||||
- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
|
- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
|
||||||
gets its own permanent admin-tier key (`61419c92`, stored at
|
`playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
|
||||||
`ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
|
drop-in filename-collision fix) adds `_netdev,nofail` to the four
|
||||||
Future admin ops route through this key, not the bootstrap admin
|
10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
|
||||||
via docker-as-root.
|
resolves paperless `Exited(255)` on every reboot. traefik also gained
|
||||||
|
`restart: unless-stopped` (was Docker-default `no` → never
|
||||||
|
auto-started after a reboot; "homepage broken" was collateral via the
|
||||||
|
traefik route). Full incident → auto-memory
|
||||||
|
`incident_esh_docker_nfs_boot_race`.
|
||||||
|
|
||||||
- `[2026-05-17]` Worldtree env-var addition checklist: anytime
|
- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
|
||||||
introducing `os.environ.get("FOO")` in worldtree code, update BOTH
|
config-validator hardening. Net image:
|
||||||
`.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
|
- v0.29.10 — `create_provider` checks model_catalog `family:`
|
||||||
same PR. Same Z_AI_API_KEY-shape footgun bit
|
before regex auto-detect (artemis routing fix)
|
||||||
`BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
|
- v0.29.11 — collect-then-raise echo startup validators (UX fix
|
||||||
passthrough line in `08f02b2`.
|
for cascading one-at-a-time crash cycle that bit demo + personal
|
||||||
|
during the v0.29.0 → v0.29.9 deploy)
|
||||||
|
- v0.29.12 — ships `providers.yaml.example` + `defaults.yaml.example`
|
||||||
|
canonical configs (REQUIRED/OPTIONAL annotated, guaranteed-pass-
|
||||||
|
validator). Replaces reverse-engineer-from-validator-errors as
|
||||||
|
the operator's path.
|
||||||
|
- v0.29.13 — `reasoning_content` extraction in base provider +
|
||||||
|
catalog `family:` lookup fix (artemis reasoning route through
|
||||||
|
worldtree GemmaProvider).
|
||||||
|
Operator-asked, worldtree-dev-shipped, all auto-deployed via CI.
|
||||||
|
|
||||||
- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
|
- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree
|
||||||
as third compose service (worldtree-driver agent dispatcher).
|
personal providers.yaml. BeaverAI Gemma 4 31B Q6_K (~28.6GB)
|
||||||
All three althing services use the same image; `command:` selects
|
pulled via `pull-hf-repo.yaml`, llama-swap config mirrors
|
||||||
entrypoint. Safe to enable preemptively (sleeps when no
|
gemma4-31b-dense settings at 128K ctx, `--reasoning-format deepseek`
|
||||||
driver=worldtree handles declared).
|
(gemma format not supported in deployed llama.cpp). Worldtree
|
||||||
|
catalog entry has `family: gemma` so GemmaProvider routes reasoning
|
||||||
|
tokens correctly. Pinned locally — push pending operator greenlight.
|
||||||
|
|
||||||
- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
|
- `[2026-05-27]` Skaldsong streaming TTS v0.32.0 → v0.32.1 → v0.32.2.
|
||||||
Internal LAN-only at port 7881 (chamber default 7878 collides with
|
v0.32.0: chunked-batch via SSE on `/api/stories/{id}/audio?voice=X`,
|
||||||
task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
|
one Kokoro POST per paragraph. v0.32.1: defensive catch-all in
|
||||||
canonical dev for this stack (galdrabok is on a different project).
|
event_stream after silent JSONDecodeError (made the real
|
||||||
|
exception surface in an SSE error frame). v0.32.2: NDJSON parsing
|
||||||
|
— Kokoro `/dev/captioned_speech` returns multi-line JSON for
|
||||||
|
large inputs (one object per chunk); was using `resp.json()` which
|
||||||
|
only parsed the first object. Verified end-to-end clean.
|
||||||
|
|
||||||
- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
|
_22 older entries archived to archival-memory.md._
|
||||||
`vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
|
|
||||||
Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
|
|
||||||
utilization; 30% headroom). All use `--runner pooling`; classification
|
|
||||||
drives via model's `architectures: [LlamaForSequenceClassification]`
|
|
||||||
in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
|
|
||||||
|
|
||||||
- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
|
|
||||||
playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
|
|
||||||
Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
|
|
||||||
|
|
||||||
- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
|
|
||||||
judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
|
|
||||||
reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
|
|
||||||
EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
|
|
||||||
|
|
||||||
- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
|
|
||||||
removed (retired by vllm). Archived
|
|
||||||
`docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
|
|
||||||
with archival header. Fixed `pfi-pve` VM list to full
|
|
||||||
`qm list` enumeration. Dropped stale weak-password section from
|
|
||||||
`pfi-postgres` (rotation done 2026-04-23).
|
|
||||||
|
|
||||||
- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
|
|
||||||
CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
|
|
||||||
client hosts: PFI hosts + provides emergency-ops backstop;
|
|
||||||
Worldtree team owns OS config + deploys + backup decisions.
|
|
||||||
|
|
||||||
- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
|
|
||||||
health-gated `:latest` advance in vh/worldtree's deploy workflow
|
|
||||||
(architect commit `8ef3801`): only tag `:latest` AFTER the new
|
|
||||||
container's `/health` probe passes. Build-on-host stacks here
|
|
||||||
don't have this problem because the playbook always builds the
|
|
||||||
SHA-tagged image from a `git reset --hard <ref>` checkout.
|
|
||||||
|
|
||||||
- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
|
|
||||||
`http://10.250.50.70:8200`. Initially included Traefik labels for
|
|
||||||
public hostname; user pulled them out (internal tool, no public
|
|
||||||
TLS surface needed). Pattern: internal tools default LAN-direct;
|
|
||||||
Traefik wiring only when external/TLS required.
|
|
||||||
|
|
||||||
- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
|
|
||||||
vram_gb, gpu_device_id }` per irv-ml1 service for the
|
|
||||||
orchestrator feature. SSH keypair scaffolded at
|
|
||||||
`ana-docker:/opt/docker/conf/asset-engine/ssh/` for
|
|
||||||
asset-engine container → irv-ml1 orchestration via dedicated
|
|
||||||
ed25519 key.
|
|
||||||
|
|
||||||
## Tried and abandoned
|
## Tried and abandoned
|
||||||
|
|
||||||
- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
|
- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
|
||||||
for CD SHA cleanup — would untag pinned/personal worldtree images
|
packaging quirks: (1) `pip install -e nari-labs/dia2` fails — its
|
||||||
since all three deployments share corviduo-dev. Use
|
pyproject build backend lacks the PEP 660 `build_editable` hook; (2)
|
||||||
`docker image prune -a --filter "until=Xh"` instead — respects
|
plain `pip install` builds an empty `UNKNOWN-0.0.0` wheel because the
|
||||||
in-use semantic (Docker won't remove an image referenced by any
|
base image's setuptools (59.6) is older than dia2's required `>=70`, so
|
||||||
container on the host), so pinned/personal protected automatically.
|
it ignores the `[project]` metadata; (3) `--no-deps` (which the wrapper's
|
||||||
|
own auto-installer uses) leaves `transformers`/`sphn`/`whisper-timestamped`
|
||||||
|
missing. Fix that worked (`local/dia:v2`, `stacks/dia/dia2-image/`): copy
|
||||||
|
the pure-python `dia2/` package straight into site-packages + install
|
||||||
|
ONLY those 3 deps; torch 2.12 / numpy 2.2 in the base already satisfy
|
||||||
|
Dia2, so no risky CUDA/torch churn.
|
||||||
|
|
||||||
- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
|
- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are
|
||||||
failed with 401 unauthorized. ana-docker had no `docker login`
|
NOT reachable from the `/tts` clone path — it resolves
|
||||||
for `gitea.phasefinal.com`. My playbook prereq note ("docker login
|
`clone_reference_filename` against the reference_audio dir ONLY
|
||||||
has been done at least once") was an unverified assumption. One-time
|
(`engine.py`). The OpenAI `/v1/audio/speech` `voice` param DOES
|
||||||
manual login persists in `~/.docker/config.json`; architectural fix
|
auto-resolve them by name (separate code path), which masked the gap in
|
||||||
(workflow-side `ssh ana-docker 'docker login ...'` step using
|
early testing. Fix: stage `/app/voices/*` into
|
||||||
`REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
|
`/worktank/dia/reference_audio` so `/tts` clone can use them. (So a
|
||||||
|
"verify on the endpoint the catalog actually targets" lesson — testing
|
||||||
|
the OpenAI endpoint nearly shipped a catalog that 404s on `/tts`.)
|
||||||
|
|
||||||
- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
|
- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
|
||||||
as a bare URL — pydantic-settings parses complex env vars via
|
→ asset-engine serializes it as the literal string `"undefined"` →
|
||||||
`json.loads()`; first-boot crashloop with `SettingsError: error
|
`/tts` 404 `Reference audio file not found: undefined`. Reproduced
|
||||||
parsing value for field "cors_origins"`. Must be JSON array literal:
|
directly. Worked around in the catalog (default the field to a real
|
||||||
`SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
|
voice); proper fix is asset-engine-side (don't send `"undefined"` for
|
||||||
|
empty optionals; enforce conditional-required) — flagged to
|
||||||
|
asset-engine-dev.
|
||||||
|
|
||||||
- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
|
- `[2026-05-31]` asset-engine `docs/asset-engine/services.schema.json`
|
||||||
in compose — mismatched Dockerfile reality. The Dockerfile COPYs
|
is a DERIVED artifact (regen from the Pydantic model via
|
||||||
SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
|
`dump_schema.py`) and had DRIFTED: it rejected the `lifecycle` field
|
||||||
Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
|
that 12/14 live services already use (13 pre-existing validation
|
||||||
rather than verifying against the actual Dockerfile they shipped.
|
errors against the committed catalog).
|
||||||
Lesson: when encoding container-internal paths in compose, verify
|
**[2026-05-31] RESOLVED:** schema regenerated with `CatalogLifecycle`
|
||||||
against the Dockerfile, not the design-doc.
|
+ `reproducibility.seed_field` (`b7b2130`); catalog now validates clean.
|
||||||
|
Standing lesson: the schema is generated from asset_engine's
|
||||||
|
`catalog.py` — hand-editing `services.yaml` shape without regenerating
|
||||||
|
re-introduces drift.
|
||||||
|
|
||||||
- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in
|
- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze
|
||||||
compose env block — names skaldsong's app doesn't read. App reads
|
a container worker in UNKILLABLE D-state when the NAS stalled —
|
||||||
`SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per
|
`docker restart` fails with "tried to kill container, but did not
|
||||||
Dockerfile ENV defaults). Wrong names = silently no-op; app fell
|
receive an exit event"; only a host reboot clears it. Separately,
|
||||||
back to Dockerfile defaults pointing at `/app/data/...` which the
|
`fstab defaults` (no `_netdev`) made NFS-bind containers `Exited(255)`
|
||||||
compose's bind mount did NOT cover (mount target was
|
on reboot. Diagnosis + recovery steps → auto-memory
|
||||||
`/app/state/...`). Result: every `--force-recreate` wiped the
|
`incident_esh_docker_nfs_boot_race`.
|
||||||
SQLite DB along with the ephemeral container layer. Caught by
|
|
||||||
skaldsong-dev after operator noticed stories vanishing on each
|
|
||||||
CD push (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in
|
|
||||||
`52e98fa` — rename env vars, bind targets unchanged. Same lesson
|
|
||||||
as the `/app/web/dist` footgun: verify env var NAMES against the
|
|
||||||
Dockerfile/app, not against design-doc shorthand.
|
|
||||||
|
|
||||||
- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
|
- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (no flag disables it) which
|
||||||
racing the container's `start_period` (30s in compose's
|
kills ttyd's terminal WebSocket (blank screen); Safari/WebKit never
|
||||||
healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
|
sends HTTP basic-auth on WS upgrades (→ ttyd "User code denied"). Both
|
||||||
— well before docker's healthcheck could flip the status from
|
solved for the fleet seat (Caddy forces HTTP/1.1; auth dropped to
|
||||||
`(health: starting)` to `(healthy)`. False-negative; container was
|
network-gating) — detail in auto-memory `reference_ttyd_fleet_seat`.
|
||||||
operationally up (the earlier `/health` poll verify already
|
|
||||||
confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
|
|
||||||
liveness check; docker's `(healthy)` is just a delayed echo.
|
|
||||||
|
|
||||||
- `[2026-05-19]` Volva daemon impersonating volva-dev (the human)
|
- `[2026-05-27]` `docker compose up -d` from within the `docker:cli`
|
||||||
for hours of debugging because cwd-based handle resolution.
|
sandbox: `${VAR:-./config}` defaults resolve `./config` to the
|
||||||
`session_handles.json` maps `/home/lkraven/development/volva` →
|
sandbox CWD (e.g. `/wt/config`). Docker daemon interprets the
|
||||||
`volva-dev`; the daemon's systemd unit sets
|
resulting path against the HOST filesystem, not the sandbox's.
|
||||||
`WorkingDirectory=/home/lkraven/development/volva` and env.sh
|
Auto-creates an empty dir on the host → container saw an empty
|
||||||
didn't pin `ALTHING_HANDLE`, so the daemon polled volva-dev's
|
bind target → entrypoint shim seeded image-baked defaults
|
||||||
inbox and replied AS volva-dev. Catch: cross-reference journal
|
(lost all the host-side providers.yaml patches I'd applied earlier
|
||||||
PIDs against `message_received` events to verify which voice
|
in the day). Root cause for ratatoskr-dev's Tier 3 sessions 500.
|
||||||
generated which reply. Fix: pin `ALTHING_HANDLE=volva` in env.sh
|
Fix: pass `-e WORLDTREE_CONFIG_DIR=/opt/worldtree-personal/config`
|
||||||
(worldtree-dev's `e39be87` made this the default in
|
to the docker run invocation. Captured into the docker-as-root
|
||||||
env.sh.template). Real follow-on risk: hallucinated structured
|
pattern note in Tools and conventions.
|
||||||
feedback from impersonating-daemon can bootstrap real (and
|
|
||||||
correct) downstream commits — the technical artifact survives
|
|
||||||
but the conversational attribution rots.
|
|
||||||
|
|
||||||
- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
|
- `[2026-05-27]` `:latest`-pinned compose + private gitea registry +
|
||||||
— systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
|
sandboxed pull = recreate on ancient cached `:latest`. Local
|
||||||
uncommenting lands the literal `$HOME/...` string. Volva-dev's
|
`:latest` tag never advanced because deploy.yml pulls by SHA
|
||||||
`f4dda73` swapped to `/home/<svc-user>/...` placeholders.
|
(not by tag); sandbox can't `docker pull` (no creds for private
|
||||||
|
registry); `compose up -d --pull never` used the 2-week-old
|
||||||
|
cached `:latest`. Symptom: personal worldtree's matrix container
|
||||||
|
crash-looped because the ancient image didn't understand the new
|
||||||
|
config schema. Fix: retag SHA → `:latest` on host first
|
||||||
|
(`docker tag <registry>/<repo>:<sha> <registry>/<repo>:latest`),
|
||||||
|
then recreate with `--pull never`. Better long-term: pin SHA
|
||||||
|
explicitly in `.env`, advance in CI.
|
||||||
|
|
||||||
- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
|
- `[2026-05-27]` Container recreate during in-flight skaldsong gen
|
||||||
without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
|
kills the runner. Pattern: my llama-swap restart at 23:57Z + a
|
||||||
althing.db`) and codex's session state (`~/.codex/`) both need to
|
v0.31.2 `compose up -d` at 02:47Z both took out active dramatist
|
||||||
write. Container started but every poll failed with "db path not
|
passes. Confirmed by skaldsong-dev INV-006 ("partial state only
|
||||||
writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
|
flips on host restart"). With deploys landing every ~10min during
|
||||||
/home/lkraven/.codex` (preserves the hardening intent, only carves
|
active dev and stories taking >5min, runner-killed-by-recreate is
|
||||||
out the specific dirs).
|
structural, not incidental. Roadmap (skaldsong-dev): pre-shutdown
|
||||||
|
signal handler (cleanup row state to failed/aborted), per-scene
|
||||||
|
resume-from-checkpoint, `/api/admin/quiesce` endpoint. None shipped
|
||||||
|
yet; operator's call which path.
|
||||||
|
|
||||||
- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
|
- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i —
|
||||||
"$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
|
not supported in the deployed llama.cpp build. Accepts only
|
||||||
under systemd — `EnvironmentFile=` parser aborts on the first
|
`none | deepseek | deepseek-legacy`. `deepseek` populates the
|
||||||
unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
|
`reasoning_content` SSE delta field which Worldtree GemmaProvider's
|
||||||
declared below silently never lands. Symptom: `Environment=`
|
fallback path checks. Was load-bearing for getting reasoning
|
||||||
property empty, daemon error "althing-cli not found at
|
tokens through to ratatoskr's wire.
|
||||||
'althing-cli'". Fix: replace command-substitution with literal
|
|
||||||
path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
|
|
||||||
|
|
||||||
- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
|
- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs
|
||||||
flag was deprecated. Use `--runner pooling`; the model's
|
the curl, killing the request early. Bit me twice while probing
|
||||||
`architectures: [LlamaForSequenceClassification]` in config.json
|
SSE-shape samples. Use file-write + separate read.
|
||||||
drives the classification head. Surfaced as `vllm: error:
|
|
||||||
unrecognized arguments: --task classify` in container logs.
|
|
||||||
|
|
||||||
- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
|
_24 older entries archived to archival-memory.md._
|
||||||
var into a worldtree container — `compose.yaml`'s `&worldtree-env`
|
|
||||||
anchor must explicitly declare the passthrough or the value silently
|
|
||||||
doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
|
|
||||||
`BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
|
|
||||||
env empty?" diagnosis each time. Worldtree-side fix in
|
|
||||||
`vh/worldtree@08f02b2`.
|
|
||||||
|
|
||||||
- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
|
|
||||||
sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
|
|
||||||
container to `:latest`, even when a newer SHA-tagged image is on
|
|
||||||
disk. Symptom: container "recreated" but actually reverted to a
|
|
||||||
stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
|
|
||||||
invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
|
|
||||||
advance is the long-term fix.
|
|
||||||
|
|
||||||
- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
|
|
||||||
naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
|
|
||||||
Native presets are `<register>_<gender>` shape (`neutral_female`,
|
|
||||||
`casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
|
|
||||||
exact wrapper-deployed preset names before setting a catalog
|
|
||||||
default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
|
|
||||||
Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
|
|
||||||
wrapper has NO voice-listing endpoint at all; voices are
|
|
||||||
filesystem paths under the `kyutai/tts-voices` HF repo).
|
|
||||||
|
|
||||||
- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
|
|
||||||
(`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
|
|
||||||
first scaffold — user pulled it back to LAN-direct. Internal
|
|
||||||
tools default LAN-direct; only add Traefik when an external/TLS
|
|
||||||
surface is actually needed.
|
|
||||||
|
|
||||||
- `[2026-05-12]` Routing althing thread replies through `galdrabok`
|
|
||||||
when the actual dev handle is `forseti` — bus rejected
|
|
||||||
`to=forseti` initially because thread participants list was
|
|
||||||
`[galdrabok, infra]`. Solved by starting a new thread with
|
|
||||||
`forseti` as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role,
|
|
||||||
start a fresh thread rather than fighting the participant list.
|
|
||||||
|
|
||||||
- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
|
|
||||||
timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
|
|
||||||
matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
|
|
||||||
**Gitea logs in PDT, Traefik logs in UTC** — same host, different
|
|
||||||
timezones. Always normalize timezones (UTC) when correlating logs
|
|
||||||
across services on the same box. Cost: ~30 min in the wrong
|
|
||||||
direction.
|
|
||||||
|
|
||||||
- `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump
|
|
||||||
for "Traefik is dropping connections on big docker pushes" without
|
|
||||||
first verifying which component was actually in the failure path.
|
|
||||||
Traefik turned out to be innocent (`499 / 60012ms` = client closed,
|
|
||||||
Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs
|
|
||||||
response writes (wrong direction), and the real ceiling was a ~60s
|
|
||||||
client-side timer no compose change can reach. **Lesson: validate
|
|
||||||
the diagnostic premise — which component is actually in the failure
|
|
||||||
path? — before refining the proposed fix.**
|
|
||||||
|
|
||||||
- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
|
|
||||||
`PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
|
|
||||||
`/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
|
|
||||||
**response writes**, not request body reads. `unexpected EOF` from
|
|
||||||
Go's HTTP server means the client closed mid-body-upload; not a
|
|
||||||
knob Gitea exposes server-side.
|
|
||||||
|
|
||||||
- `[2026-04-30]` task-board workflow with
|
|
||||||
`container: image: debian:bookworm-slim` — fails:
|
|
||||||
`actions/checkout@v4` needs `node` at runtime, slim image lacks it.
|
|
||||||
Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
|
|
||||||
default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
|
|
||||||
override needs `nodejs` apt-installed unless it IS the default.)
|
|
||||||
|
|
||||||
- `[2026-04-30]` Dropping the `container:` directive before runner
|
|
||||||
re-registration with docker-schema labels — runner silently falls
|
|
||||||
back to **host mode** (jobs run inside the alpine `act_runner`
|
|
||||||
container itself, no apt). The `:host` suffix in startup logs
|
|
||||||
(`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
|
|
||||||
giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
|
|
||||||
labels.
|
|
||||||
|
|
||||||
- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
|
|
||||||
— doesn't take. The `.runner` registration cache pins labels at
|
|
||||||
first registration; env-var updates are read each start but the
|
|
||||||
stored token + UUID are tied to the original label set on the gitea
|
|
||||||
side. Fix: stop runner, delete `.runner`, generate new admin
|
|
||||||
registration token, redeploy.
|
|
||||||
|
|
||||||
- `[2026-04-30]` `git reset --hard origin/<sha>` in
|
|
||||||
`deploy-task-board.yaml` (and the in-repo nevermore playbook before
|
|
||||||
fix) — invalid syntax: `origin/` prefix only works for branch refs.
|
|
||||||
SHAs need `git reset --hard <sha>` directly. Resolved with
|
|
||||||
`git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
|
|
||||||
then bare `"{{ ref }}^{commit}"` fallback.
|
|
||||||
|
|
||||||
- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
|
|
||||||
task-board wiring — it was actually only repo-scope on
|
|
||||||
`vh/task-board`. vor's first CI run failed with empty SSH key
|
|
||||||
(`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
|
|
||||||
scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
|
|
||||||
|
|
||||||
- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
|
|
||||||
leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
|
|
||||||
follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
|
|
||||||
still leaked the first 4 chars. Lesson: when probing secret-bearing
|
|
||||||
files, use field-by-field SELECTIVE inspection
|
|
||||||
(`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
|
|
||||||
password line, `grep -c` (existence) or `test -n "$(...)"`
|
|
||||||
(non-empty), never `cat` or value-printing.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user