vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot for /clear — booth 3 features shipped, herald v2.1.2, CI-flip PARKED (runner-auth)

Browse Source 
Session (2026-08-05): 3 Booth features live+tagged (verbatim-wrap chip, .md/.txt
doc-viewer, image prev/next arrows); worldtree herald re-nudge bug -> forseti
althing-core v2.1.2; fleet-CI-resilience flip attempted end-to-end and PARKED on
an act_runner->gitea action-fetch auth blocker (infra-ops to research, deferred).
Archived the 2026-07-15/16 recent-decisions batch (8 entries) to keep the index
under the soft cap.
This commit is contained in:
vh
2026-08-05 04:31:59 -07:00
parent c37a425276
commit 6c6d3f2939
3 changed files with 55 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+11 -22
View File
@@ -109,7 +109,7 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-08-03 — **session at a natural close; nothing infra-ops-side blocked.** Two long arcs landed: (1) the mimir-inbox / #377-read-path arc (2026-08-01→02); (2) the **worldtree b168/#384/#385 arc COMPLETE** (2026-08-03 — providers.yaml pre-sync → deploy → DCC+P&P re-ingest 705+667 concepts → #381 restart → operator-approved production dedup sweep; full detail in the 2026-08-03 Recent-decisions entry + `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`). Only live watch = worldtree-dev re-running the personal **b169** deploy (all-clear given). Headlines:_
_As of 2026-08-05 — **session at a natural close; nothing infra-ops-side mid-action.** This session (08-05): shipped **3 Booth features** (verbatim-wrap back-chip+favicon, `.md`/`.txt` doc-viewer, image prev/next arrows — all live on `:8090`, committed + tagged `booth-v0.1.5/0.1.6/0.1.7`); root-caused the **worldtree herald re-nudge bug** → forseti shipped **althing-core v2.1.2** (`d5d33df`, deployed on nh3-dev); attempted the **fleet-CI-resilience flip** and **PARKED** it on a runner-auth blocker (→ Recent decisions, my deferred research task) with gitea **`REQUIRE_SIGNIN_VIEW=false`** now a standing change. The 08-03 b169 watch is CLOSED (worldtree-dev confirmed both instances clean). The prior-arc headlines below are historical context (see the 2026-08-02/03 Recent-decisions entries):_
- **mimir-inbox DEPLOYED + verified on corviduo-dev `10.250.50.152:8091`** (#377 browser-facing half; write+read proven end to end). Live commit `8ece117` (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory `reference_mimir_inbox_deploy`.
- **#377 read path fully working** — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs **#380** (wing-blind index → concepts in wrong collection; fixed b164 + one-shot `--reindex <job_id>`), **#381** (stale Chroma client → **restart `worldtree-personal-worldtree-api-1` after any ingest/re-index** until their fix), **#382** (intermittent Mimir grounding / silent training-substitution; fixed b166 = `_index.md` per wing job-dir + a prompt rule; **verified 3/3** by ratatoskr-dev). DCC re-file SETTLED = **no** (prompt rule grounds it; `_index.md` rides next re-ingest).
- **muninn-gate → muninn-dispatch 0.1.5** (rebuilt off `vh/muninn-gate` `bc04c4c`, image 0.0.14; serves `concept_schema`/`concept_schema_source`). BuildKit gitea secret; recreate w/ `compose up -d` not bare restart.
@@ -122,18 +122,24 @@ _As of 2026-08-03 — **session at a natural close; nothing infra-ops-side block
**Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle, **expanded same day with onyx-58 clips → 52.0s multi-clip ref**); add a voice = drop `<Name>.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched). **ACTIVE WATCH — b169 personal deploy:** worldtree-dev re-running the staging/v1.0.0b169 personal deploy after a pull-fail I diagnosed as a transient shared-containerd concurrent-pull race (NOT disk); all-clear given = just re-run, **do NOT prune** (`6e34a87` is in-use by the running demo instance — pruning would down demo; see Tried-and-abandoned). worldtree-dev verifies health + filed #388 for a deploy concurrency-lock. **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (503 on b146 until a staging promotion — operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry.
**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched). **b169 watch CLOSED** (2026-08-05 — worldtree-dev confirmed both instances clean after the re-run; was a shared-containerd pull-race, NOT disk — see Tried-and-abandoned). **Current standing item — worldtree #389 config delta (2026-08-05, parity-only, NO infra-ops action):** commit `f66559c` (unpushed) changes `agents/mimir/{system.md, config.yaml}`; `agents/` is **IN-IMAGE** (Dockerfile:54 `COPY agents/`), not among the 5 bind-mounted config files, so it ships to demo/personal via worldtree-dev's **CI/CD image build** — infra-ops just stands by, no deploy-wt-config pre-sync (source-verified with worldtree-dev; `matrix.yaml` carries only per-agent display identity, no companion; pinned excluded); b174 pushed 2026-08-05 (`9b7f1fd`), demo deploy running — no action. **+ CARRIED TASK (do early next session):** publish **muninn-dispatch 0.1.6** to the Gitea PyPI registry (worldtree-dev asked; source on Worldtree `origin/main` `packages/muninn-dispatch`, comment-only bump; self-serve via the vh-token per `reference_infra_ops_vh_gitea_token_and_sdk_publish`), then ping worldtree-dev to announce — muninn-dev holds its 0.1.5 pin awaiting it. See the /tmp handoff. **STANDING CONFIG CHANGE:** gitea `REQUIRE_SIGNIN_VIEW=false` on ana-docker (from the CI-flip attempt, kept per operator — internal WG net). **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry.
**Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev).
**eshpfi has UNPUSHED local commits** — `main` is **~21 ahead of `origin/main`** (mostly 2026-08-02→03 memory commits from the worldtree arc + this snapshot; plus the earlier mimir-inbox/muninn-gate/Audio8/#383 work). **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore, don't stage).
**eshpfi push state:** `main` WAS pushed to origin this session (synced through the booth verbatim-wrap + memory commits); a handful of 2026-08-05 commits are unpushed since — booth doc-viewer `315faac`, arrows `c37a425`, triton memory `b5ae936`, + this snapshot. **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns every commit, ignore, don't stage).
**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); **Audio8 → TTS zoo** (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); **worldtree config-fold PENDING** (next routine config sync): add `reference_knowledge.tier3_wings: ["fiction"]` to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose).
**Carried standing (non-blocking):** **INFRA (non-)WATCH (2026-08-05, worldtree-dev — CORRECTED, was over-stated):** the R42 empties were **NOT an infra fault** — Brokkr's full-args capture found mimir **hallucinating a non-empty `facet_filter`** that legitimately matches nothing (agent-side, worldtree's). **NO evidence of an embed/Chroma silent-empty seam fault as of 2026-08-05** (my empty-log grep was the finding, but the cause was agent, not infra). The tripwire now in the instance logs (b175/b176 guards): **"filters … emptied a subject-bearing search"** = agent hallucination (worldtree's, expected occasional noise, NOT infra) · **"silent zero-row"** = a genuine embed/Chroma seam fault (MINE) → dig at the embed/Chroma layer **only if these show in volume.** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); **Audio8 → TTS zoo** (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); **worldtree config-fold PENDING** (next routine config sync): add `reference_knowledge.tier3_wings: ["fiction"]` to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose).
## Recent decisions
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
- `[2026-08-03]` **worldtree b168/#384/#385 arc COMPLETE** — providers.yaml boot-gate pre-sync → b168 deploy → DCC+P&P re-ingest (705+667 concepts, 0 truncations, #385 budget fix validated vs April's 785 control) → #381 restart → operator-approved production dedup sweep (785 April orphans deleted from `main`, 4009→3224). Fiction wing 166→1,372 concepts; consumer verify 0/5→5/5→saturated. Three of MY foot-guns hardened into fleet runbook rules (`mv -t`, `docker exec -u 1000`, shared-containerd pull-race — see Tried-and-abandoned). Full runbooks (deploy-wt-config, Chroma-verify, config-delta pre-sync rule, #381, sweep) → `persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md`
- `[2026-08-02]` **mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice.** mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live `8ece117`); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and **verified 3/3** by ratatoskr-dev; muninn-gate → dispatch 0.1.5; **donut** voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`
@@ -200,28 +206,11 @@ _As of 2026-08-03 — **session at a natural close; nothing infra-ops-side block
- `[2026-07-17]` **Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock.** Config captured (14a0004, unpushed); build = cu128 base + `uv sync` vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → `persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md`
- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. See Current state for the full layout + backups.
- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack).
- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.
- `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md`
- `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md`
- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)
- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.
- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.
- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.
- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).
_167 older entries archived to archival-memory.md._
_175 older entries archived to archival-memory.md._
## Tried and abandoned