diff --git a/archival-memory.md b/archival-memory.md index fad9881..711898a 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -1726,3 +1726,29 @@ _Archived 2026-08-03._ _Archived 2026-08-03._ + +## Recent decisions (archived 2026-08-05 batch) + +- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. + _Archived 2026-08-05._ + +- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.] + _Archived 2026-08-05._ + +- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`. + _Archived 2026-08-05._ + +- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) + _Archived 2026-08-05._ + +- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA → allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`. + _Archived 2026-08-05._ + +- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39→0.38 + max-model-len 192K→150K. LESSON: `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom. + _Archived 2026-08-05._ + +- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library`, restarted. Closed a queued no-rush item. + _Archived 2026-08-05._ + +- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate. + _Archived 2026-08-05._ diff --git a/persistent-memory.d/2026-08-05-ci-flip-parked.md b/persistent-memory.d/2026-08-05-ci-flip-parked.md new file mode 100644 index 0000000..59d90ce --- /dev/null +++ b/persistent-memory.d/2026-08-05-ci-flip-parked.md @@ -0,0 +1,18 @@ +- `[2026-08-05]` **Fleet CI resilience — DEFAULT_ACTIONS_URL=self flip ATTEMPTED end-to-end, PARKED on a runner-auth blocker. Infra-ops to research the runner action-fetch auth, later (operator-directed 2026-08-05, deferred — not now; untracked, no issue).** + +**Goal (worldtree-dev's operator-directed filing, run-9189 evidence):** every Gitea Actions job hard-depends on **github.com** at step zero — `act_runner` resolves bare `uses:` refs (checkout/cache/setup-uv/etc.) against github at job start. A GitHub blip froze a real deploy (run 9189, `connection reset` cloning `actions/checkout`). Fix = mirror the action repos into Gitea + point `DEFAULT_ACTIONS_URL` at self, so github can be down and fleet CI doesn't care. + +**What's DONE + staged (all reversible, still in place):** +- **Fleet `uses:` audit** (scripts in `/tmp/claude-1000/gitea_uses_audit.py`, run via ana-docker localhost API): 71 repos, 23 with workflows, but the raw ~42 action count is **almost all dormant vendored-OSS mirrors** (0 Action runs). The **actually-running CI repos** (Worldtree/arbo/althing/skaldsong/asset-engine/vor/task-board/nevermore/mead-hall/soong-lab) use just **7 action repos**. +- **7 mirrors created + populated + public** under gitea orgs **`actions`** + **`astral-sh`**: checkout, cache, upload-artifact, download-artifact, setup-node, setup-python, astral-sh/setup-uv. All in-use tags verified present (checkout@v4/v6, cache@v4, up/download-artifact@v3, setup-node@v4, setup-python@v5, setup-uv@v3/v5/v7). **Actions DISABLED on all 7** (they're source mirrors; don't want their own CI). Repos are PUBLIC. +- **Gitea = 1.26.1, container `gitea` on ana-docker; runner = `gitea-runner` (act_runner v0.6.0), label `pfi-fleet`, jobs run in a `container:`.** + +**Mirror-creation FOOT-GUN (paid for):** gitea's **migrate-from-github is flaky** — migrations ran 227–531s then 422'd, leaving broken empty repos (only cache synced). And **github throttles ana-docker's colo IP** after a clone burst (same pattern as the original github dependency). **The reliable method: plain `git clone --mirror` on nh3-dev (residential egress) + push to gitea via git-SSH `ssh://git@10.250.50.70:222` (auths as vh from nh3-dev).** That populated the last 3 cleanly. Use that, not the gitea migrate API, to (re)build mirrors. + +**THE BLOCKER (why it's parked):** with `DEFAULT_ACTIONS_URL=self`, the runner correctly resolves `uses: actions/checkout@v4` → `https://gitea.phasefinal.com/actions/checkout` (confirmed in the runner log + the decompressed job log at `/data/gitea/actions_log/vh//*.log.zst` — **zstd, decompress on the ana-docker HOST, not in the gitea container which lacks zstd**). But the fetch **fails on auth**: `authentication required: Invalid username or token. Password authentication is not supported for Git operations.` The runner is **NOT** fetching anonymously — it **sends a credential gitea rejects**. So `REQUIRE_SIGNIN_VIEW=false` did NOT fix it (that would only help an anonymous fetch; anon clone of the public mirror does work now). The real issue is **how act_runner v0.6.0 authenticates its action-fetch to a gitea 1.26 instance** — that's the research task. + +**Current CONFIG STATE (post-revert):** `DEFAULT_ACTIONS_URL` is **REMOVED** from gitea app.ini → **back to github default (CI works normally)**. **`REQUIRE_SIGNIN_VIEW = false` was SET and KEPT** (operator: "require_signin_view false on internal wg net") — now a **standing change** on the internal WG net (anon view of PUBLIC repos only; private repos stay auth-gated). app.ini backups on the box: `/data/gitea/conf/app.ini.bak-*` (signinflip / revert / actions). + +**Smoke method (for when re-attempting):** create a throwaway `vh/actions-smoke` repo with a minimal `runs-on: pfi-fleet` + `container: python:3.11.10-slim-bookworm` + `uses: actions/checkout@v4` + `echo` workflow (adding the workflow file triggers `on: push`); poll `/repos/vh/actions-smoke/actions/tasks`. `ci.yml` has NO `workflow_dispatch` and the run **rerun API 404s** on 1.26 — pushing a commit is the trigger. SUCCESS = the checkout step resolves from the local mirror. + +**NEXT STEP (my deferred task):** research act_runner's action-fetch auth on gitea 1.26 (how it should authenticate; a runner config token, a gitea setting, or a version constraint). worldtree-dev (filer, runs gitea CI daily) offered as an alternative but operator directed **infra-ops** to do it. Everything's staged for a clean re-attempt once the auth path is understood; if dropped, tear down the `actions`/`astral-sh` orgs + 7 mirrors. Related: `[[2026-08-03-worldtree-b168-384-385-arc]]` (the gitea-CI stack context). diff --git a/persistent-memory.md b/persistent-memory.md index 83891c8..49d9297 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -109,7 +109,7 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-08-03 — **session at a natural close; nothing infra-ops-side blocked.** Two long arcs landed: (1) the mimir-inbox / #377-read-path arc (2026-08-01→02); (2) the **worldtree b168/#384/#385 arc COMPLETE** (2026-08-03 — providers.yaml pre-sync → deploy → DCC+P&P re-ingest 705+667 concepts → #381 restart → operator-approved production dedup sweep; full detail in the 2026-08-03 Recent-decisions entry + `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`). Only live watch = worldtree-dev re-running the personal **b169** deploy (all-clear given). Headlines:_ +_As of 2026-08-05 — **session at a natural close; nothing infra-ops-side mid-action.** This session (08-05): shipped **3 Booth features** (verbatim-wrap back-chip+favicon, `.md`/`.txt` doc-viewer, image prev/next arrows — all live on `:8090`, committed + tagged `booth-v0.1.5/0.1.6/0.1.7`); root-caused the **worldtree herald re-nudge bug** → forseti shipped **althing-core v2.1.2** (`d5d33df`, deployed on nh3-dev); attempted the **fleet-CI-resilience flip** and **PARKED** it on a runner-auth blocker (→ Recent decisions, my deferred research task) with gitea **`REQUIRE_SIGNIN_VIEW=false`** now a standing change. The 08-03 b169 watch is CLOSED (worldtree-dev confirmed both instances clean). The prior-arc headlines below are historical context (see the 2026-08-02/03 Recent-decisions entries):_ - **mimir-inbox DEPLOYED + verified on corviduo-dev `10.250.50.152:8091`** (#377 browser-facing half; write+read proven end to end). Live commit `8ece117` (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory `reference_mimir_inbox_deploy`. - **#377 read path fully working** — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs **#380** (wing-blind index → concepts in wrong collection; fixed b164 + one-shot `--reindex `), **#381** (stale Chroma client → **restart `worldtree-personal-worldtree-api-1` after any ingest/re-index** until their fix), **#382** (intermittent Mimir grounding / silent training-substitution; fixed b166 = `_index.md` per wing job-dir + a prompt rule; **verified 3/3** by ratatoskr-dev). DCC re-file SETTLED = **no** (prompt rule grounds it; `_index.md` rides next re-ingest). - **muninn-gate → muninn-dispatch 0.1.5** (rebuilt off `vh/muninn-gate` `bc04c4c`, image 0.0.14; serves `concept_schema`/`concept_schema_source`). BuildKit gitea secret; recreate w/ `compose up -d` not bare restart. @@ -122,18 +122,24 @@ _As of 2026-08-03 — **session at a natural close; nothing infra-ops-side block **Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle, **expanded same day with onyx-58 clips → 52.0s multi-clip ref**); add a voice = drop `.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`). -**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (` completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched). **ACTIVE WATCH — b169 personal deploy:** worldtree-dev re-running the staging/v1.0.0b169 personal deploy after a pull-fail I diagnosed as a transient shared-containerd concurrent-pull race (NOT disk); all-clear given = just re-run, **do NOT prune** (`6e34a87` is in-use by the running demo instance — pruning would down demo; see Tried-and-abandoned). worldtree-dev verifies health + filed #388 for a deploy concurrency-lock. **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (503 on b146 until a staging promotion — operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry. +**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (` completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched). **b169 watch CLOSED** (2026-08-05 — worldtree-dev confirmed both instances clean after the re-run; was a shared-containerd pull-race, NOT disk — see Tried-and-abandoned). **Current standing item — worldtree #389 config delta (2026-08-05, parity-only, NO infra-ops action):** commit `f66559c` (unpushed) changes `agents/mimir/{system.md, config.yaml}`; `agents/` is **IN-IMAGE** (Dockerfile:54 `COPY agents/`), not among the 5 bind-mounted config files, so it ships to demo/personal via worldtree-dev's **CI/CD image build** — infra-ops just stands by, no deploy-wt-config pre-sync (source-verified with worldtree-dev; `matrix.yaml` carries only per-agent display identity, no companion; pinned excluded); b174 pushed 2026-08-05 (`9b7f1fd`), demo deploy running — no action. **+ CARRIED TASK (do early next session):** publish **muninn-dispatch 0.1.6** to the Gitea PyPI registry (worldtree-dev asked; source on Worldtree `origin/main` `packages/muninn-dispatch`, comment-only bump; self-serve via the vh-token per `reference_infra_ops_vh_gitea_token_and_sdk_publish`), then ping worldtree-dev to announce — muninn-dev holds its 0.1.5 pin awaiting it. See the /tmp handoff. **STANDING CONFIG CHANGE:** gitea `REQUIRE_SIGNIN_VIEW=false` on ana-docker (from the CI-flip attempt, kept per operator — internal WG net). **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry. **Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev). -**eshpfi has UNPUSHED local commits** — `main` is **~21 ahead of `origin/main`** (mostly 2026-08-02→03 memory commits from the worldtree arc + this snapshot; plus the earlier mimir-inbox/muninn-gate/Audio8/#383 work). **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore, don't stage). +**eshpfi push state:** `main` WAS pushed to origin this session (synced through the booth verbatim-wrap + memory commits); a handful of 2026-08-05 commits are unpushed since — booth doc-viewer `315faac`, arrows `c37a425`, triton memory `b5ae936`, + this snapshot. **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns every commit, ignore, don't stage). **PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made. -**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); **Audio8 → TTS zoo** (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); **worldtree config-fold PENDING** (next routine config sync): add `reference_knowledge.tier3_wings: ["fiction"]` to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose). +**Carried standing (non-blocking):** **INFRA (non-)WATCH (2026-08-05, worldtree-dev — CORRECTED, was over-stated):** the R42 empties were **NOT an infra fault** — Brokkr's full-args capture found mimir **hallucinating a non-empty `facet_filter`** that legitimately matches nothing (agent-side, worldtree's). **NO evidence of an embed/Chroma silent-empty seam fault as of 2026-08-05** (my empty-log grep was the finding, but the cause was agent, not infra). The tripwire now in the instance logs (b175/b176 guards): **"filters … emptied a subject-bearing search"** = agent hallucination (worldtree's, expected occasional noise, NOT infra) · **"silent zero-row"** = a genuine embed/Chroma seam fault (MINE) → dig at the embed/Chroma layer **only if these show in volume.** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); **Audio8 → TTS zoo** (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); **worldtree config-fold PENDING** (next routine config sync): add `reference_knowledge.tier3_wings: ["fiction"]` to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose). ## Recent decisions +- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md` + +- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b//view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`). + +- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`. + - `[2026-08-03]` **worldtree b168/#384/#385 arc COMPLETE** — providers.yaml boot-gate pre-sync → b168 deploy → DCC+P&P re-ingest (705+667 concepts, 0 truncations, #385 budget fix validated vs April's 785 control) → #381 restart → operator-approved production dedup sweep (785 April orphans deleted from `main`, 4009→3224). Fiction wing 166→1,372 concepts; consumer verify 0/5→5/5→saturated. Three of MY foot-guns hardened into fleet runbook rules (`mv -t`, `docker exec -u 1000`, shared-containerd pull-race — see Tried-and-abandoned). Full runbooks (deploy-wt-config, Chroma-verify, config-delta pre-sync rule, #381, sweep) → `persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md` - `[2026-08-02]` **mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice.** mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live `8ece117`); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and **verified 3/3** by ratatoskr-dev; muninn-gate → dispatch 0.1.5; **donut** voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md` @@ -200,28 +206,11 @@ _As of 2026-08-03 — **session at a natural close; nothing infra-ops-side block - `[2026-07-17]` **Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock.** Config captured (14a0004, unpushed); build = cu128 base + `uv sync` vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → `persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md` -- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. See Current state for the full layout + backups. - -- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). - -- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`. - - `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md` - `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md` -- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) - -- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`. - -- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now. - -- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator. - -- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90). - - -_167 older entries archived to archival-memory.md._ +_175 older entries archived to archival-memory.md._ ## Tried and abandoned